Improve progress bar by tracking record metrics #184
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #184 +/- ##
==========================================
+ Coverage 83.07% 83.35% +0.28%
==========================================
Files 34 35 +1
Lines 3609 3718 +109
==========================================
+ Hits 2998 3099 +101
- Misses 611 619 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
This PR improves the progress bar to show other metrics that were normally not available due to the limitation of The updating of the progress bar now also happens via a thread that periodically reads the It shows how many records have been processed/read and how many have matched the selector, and how many have been excluded by the selector. I'm not so sure about the naming of |
There was a problem hiding this comment.
Pull Request Overview
This PR adds an AppContext for tracking record reader metrics throughout the flow record processing pipeline. The context tracks records read, matched, and excluded counts, enabling better progress reporting and statistics.
Key Changes
- Introduces
AppContextclass for tracking record processing metrics - Updates all record adapters to utilize the context for metric tracking
- Enhances rdump tool's progress bar to display detailed metrics
Reviewed Changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
flow/record/context.py |
New context module with AppContext class and helper functions |
flow/record/stream.py |
Updates RecordReader to track metrics in app context |
flow/record/tools/rdump.py |
Enhanced progress bar with detailed metrics display |
flow/record/adapter/*.py |
Updates all adapters to track metrics via app context |
tests/conftest.py |
Test fixture to reset app context between tests |
tests/record/test_context.py |
Comprehensive tests for app context functionality |
tests/tools/test_rdump.py |
Updated tests for new progress bar format |
tests/test_regressions.py |
Test updates to accommodate new context behavior |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Also still contemplating if I should change |
8c06419 to
7edcb8d
Compare
All record reader adapters now utilize an `AppContext` to track metrics such as the number of records read, matched, and excluded. This context is accessible via `flow.record.context.get_app_context()`. The progress bar in `rdump` has been updated to display these metrics.
This helper function matches the record against given selector, and updates metrics in the context.
- The `ctx.read += 1` now happens in the helper function - Renamed ctx.excluded to ctx.unmatched
1956df3 to
2f96e95
Compare
Co-authored-by: Erik Schamper <[email protected]>
Co-authored-by: Erik Schamper <[email protected]>
All record reader adapters now utilize an
AppContextto track metrics such as the number of records read, matched, and excluded.This context is accessible via
flow.record.context.get_app_context().The progress bar in
rdumphas been updated to display these metrics.