update ingestion timestamp metadata when all rows filtered out#17758
update ingestion timestamp metadata when all rows filtered out#17758jadami10 wants to merge 3 commits intoapache:masterfrom
Conversation
| } | ||
|
|
||
| @Test | ||
| public void testIngestionTimestampUpdatedWhenAllEventsFiltered() |
There was a problem hiding this comment.
This test was claude generated. I and claude could not figure out how to run the integration tests, so i'm relying on CI here to do it. If this doesn't work, I will just remove it.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #17758 +/- ##
============================================
+ Coverage 63.17% 63.20% +0.02%
Complexity 1454 1454
============================================
Files 3176 3179 +3
Lines 191025 191321 +296
Branches 29206 29266 +60
============================================
+ Hits 120688 120924 +236
- Misses 60920 60956 +36
- Partials 9417 9441 +24
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
cc @noob-se7en got 1 more freshness checker fix for you |
| } | ||
| } | ||
| return new Result(rows, skippedRowCount, incompleteRowCount, sanitizedRowCount); | ||
| return new Result(rows, incompleteRowCount, skippedRowCount, sanitizedRowCount); |
There was a problem hiding this comment.
whoops - this was meant for a different PR
|
cc @Jackie-Jiang - good to merge this as well? |
This is a small bug fix for an edge case where freshness never catches up on server restart. The series of events are:
minimumIngestionLagnever drops below the initial indexed eventInstead, we now detect when we've conusmed events but they've all been filtered, and we correctly set latest ingestion and min freshness based on current time.
I've added 1 unit test and 1 integration test for this. We also deployed this to the main production cluster that was having issues. Previously we would see
After this deployment: