Fix RemoteAttacher date filtering to preserve time component #52
+27
−18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR removes date casting from SQL filter generation to preserve time components during date comparisons. The change addresses timezone-sensitive failures that occur when datetime values cross midnight boundaries.
- Removes
CAST({column} as Date)operations from all date column references in SQL WHERE clauses - Adds explanatory comments about the rationale for preserving time components
- Affects filtering logic across multiple fetch duration modes (Past24Hours, Past7Days, PastMonth, PastYear, SinceLastUse, Custom, and DeltaReading)
Comments suppressed due to low confidence (3)
Rdmp.Core/DataLoad/Modules/Attachers/RemoteAttacher.cs:85
- The
GetCorrectDateAddForDatabaseTypemethod still casts to Date type (e.g., line 71 for PostgreSQL, line 79 for SQL Server, line 81 for MySQL, and TRUNC for Oracle which truncates to date). This creates inconsistent comparisons where the column now preserves its time component but the comparison value is still truncated to date-only. For example, a datetime column value of '2024-01-15 23:30:00' would be compared against '2024-01-15 00:00:00', which doesn't achieve the intended purpose of preserving time accuracy. All date casts in this method should be removed to maintain consistency with the changes inSqlHistoricalDataFilter.
private static string GetCorrectDateAddForDatabaseType(DatabaseType dbType, string addType, string amount)
{
switch (dbType)
{
case DatabaseType.PostgreSql:
return $"cast((NOW() AT TIME ZONE 'UTC' + interval '{amount} {addType}S') as Date)";
case DatabaseType.Oracle:
if (addType == "DAY") return $"TRUNC(DateAdd(SYS_EXTRACT_UTC(SYSTIMESTAMP),,{amount}))";
if (addType == "WEEK") return $"TRUNC(DateAdd(SYS_EXTRACT_UTC(SYSTIMESTAMP),,{amount} *7))";
if (addType == "MONTH") return $"TRUNC(DateAdd(SYS_EXTRACT_UTC(SYSTIMESTAMP),,,{amount}))";
if (addType == "YEAR") return $"TRUNC(DateAdd(SYS_EXTRACT_UTC(SYSTIMESTAMP),,,,{amount}))";
return $"TRUNC(DateAdd(SYS_EXTRACT_UTC(SYSTIMESTAMP),,{amount}))";
case DatabaseType.MicrosoftSQLServer:
return $"CAST(DATEADD({addType}, {amount}, GETUTCDATE()) as Date)";
case DatabaseType.MySql:
return $"DATE(DATE_ADD(UTC_TIMESTAMP(), INTERVAL {amount} {addType}))";
default:
throw new InvalidOperationException("Unknown Database Type");
}
}
Rdmp.Core/DataLoad/Modules/Attachers/RemoteAttacher.cs:103
- The
ConvertDateStringmethod still converts to Date type for most database types (lines 94, 96, 98, 100), which truncates the time component. SinceRemoteTableDateFormatis defined as 'yyyy-MM-dd HH:mm:ss.fff' (line 43) and includes time information, this conversion negates the purpose of preserving time components. For databases other than PostgreSQL, the conversion should produce datetime/timestamp values instead of date-only values to match the format string and maintain consistency with the column comparison changes.
private string ConvertDateString(DatabaseType dbType, string dateString)
{
switch (dbType)
{
case DatabaseType.PostgreSql:
return $"'{dateString}'";
case DatabaseType.Oracle:
return $"TO_DATE('{dateString}')";
case DatabaseType.MicrosoftSQLServer:
return $"convert(Date,'{dateString}')";
case DatabaseType.MySql:
return $"convert('{dateString}',Date)";
default:
return $"convert(Date,'{dateString}')";
}
}
Rdmp.Core/DataLoad/Modules/Attachers/RemoteAttacher.cs:40
- Corrected spelling of 'colunn' to 'column'.
[DemandsInitialization("Option name for the Date colunn table within RDMP if it differs from the remote table e.g Due to joins")]
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ed91a3d to
e889d10
Compare
Remove CAST to Date which was stripping time component and causing timezone-sensitive test failures in RemoteDatabaseAttacherTests. Issue: - SqlHistoricalDataFilter was using CAST(column as Date) - This strips hours/minutes/seconds from datetime values - When test runs near midnight, causes off-by-one day errors - Test expects 3 rows but gets 2 due to date boundary issues Example failure: - withinDate = "2025-10-28 23:00:00" (1 hour ago, Oct 28) - CAST to Date = "2025-10-28" (time stripped) - DATEADD(DAY, -1, GETDATE()) = "2025-10-28" at midnight - Comparison: "2025-10-28" > "2025-10-28" = FALSE (row excluded) Fix: - Remove all CAST(column as Date) operations - Use datetime column directly for time-sensitive comparisons - Preserves full precision for accurate date filtering Affected cases: - Past24Hours, Past7Days, PastMonth, PastYear - SinceLastUse, Custom, DeltaReading This fixes CI failures in PR #39 and other dependabot PRs running RemoteDatabaseAttacherTests at various times of day. Tests affected: - TestRemoteDatabaseAttacherWithDateFilter(*, *, Past24Hours)
Update unit test assertions to match the corrected SQL that no longer casts the datetime column to Date. All tests now expect: WHERE date > DATEADD(...) Instead of: WHERE CAST(date as Date) > DATEADD(...) This preserves time component for accurate datetime filtering and fixes timezone-sensitive test failures. Tests updated: - TestRemoteAttacherParameter (4 cases) - TestRemoteAttacherParameterSinceLastUse - TestRemoteAttacherParameterCustomRange (3 cases) - TestRemoteAttacherParameterDeltaReading (2 cases) All 14 RemoteAttacherTests now passing.
e889d10 to
f762efa
Compare
Pull Request Test Coverage Report for Build 18908693538Details
💛 - Coveralls |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes CI failures in PR #39 and other dependabot PRs by removing CAST to Date that was stripping time component from datetime comparisons.
Problem
RemoteDatabaseAttacherTests were failing with timezone-sensitive errors:
Root Cause
RemoteAttacher.cs:107had:The Bug:
Fix
Removed all
CAST(column as Date)operations:WHERE CAST(date_seen as Date) > DATEADD(DAY, -1, GETDATE())WHERE date_seen > DATEADD(DAY, -1, GETDATE())Column is already DateTime type, so CAST is unnecessary and harmful.
Affected Code
All AttacherHistoricalDurations cases:
Impact
Fixes:
Tests: