feat(histogram): OTEL Cummulative enhanced rate + quantile calculation with max-min values#2119
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements support for OTEL cumulative histogram rate calculation with min/max value tracking and counter correction. The implementation follows the composition pattern used elsewhere in the codebase, reusing existing functions for clean and maintainable code.
Changes:
- Added
CumulativeHistRateAndMinMaxFunctionclass that composes histogram rate calculation with min/max aggregation for cumulative histograms - Updated test data generation to use constant min/max values (200.0 and 1.0) to better simulate realistic OTEL-style observed values
- Added comprehensive test coverage including basic rate calculation, counter reset handling, composition verification, and edge cases
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| query/src/main/scala/filodb/query/exec/rangefn/AggrOverTimeFunctions.scala | Implements CumulativeHistRateAndMinMaxFunction with counter correction support and min/max aggregation |
| core/src/test/scala/filodb.core/TestData.scala | Adds cumulative histogram dataset definition and helper functions, updates histMaxMin to use constant values for OTEL-style metrics |
| query/src/test/scala/filodb/query/exec/rangefn/AggrOverTimeFunctionsSpec.scala | Adds four comprehensive test cases covering basic functionality, counter resets, composition, and edge cases |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
query/src/main/scala/filodb/query/exec/rangefn/AggrOverTimeFunctions.scala
Outdated
Show resolved
Hide resolved
query/src/test/scala/filodb/query/exec/rangefn/AggrOverTimeFunctionsSpec.scala
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| val startRowNum = tsReader.binarySearch(tsVectorAcc, tsVector, startTime) & 0x7fffffff | ||
| val endRowNum = Math.min(tsReader.ceilingIndex(tsVectorAcc, tsVector, endTime), info.numRows - 1) |
There was a problem hiding this comment.
The binary search is performed redundantly here. The parent class CounterChunkedRangeFunction.addChunks already performs the same binary search at lines 135-136 before calling this method. These row numbers could be passed down from the parent or stored, avoiding the duplicate computation. While this doesn't cause incorrect behavior, it adds unnecessary overhead for every chunk processed.
There was a problem hiding this comment.
looked in the feasibility of it to pass those values, but that would break the interface. Checking further. Currently leaning towards adding a separate PR to address this
|
Seeing the expected values with the new rate function. |
…cala Co-authored-by: Copilot <[email protected]>
|
Also seeing max/min values being passed to quantile correctly. bucketEnd is also being clamped to max accordingly. |
sherali42
left a comment
There was a problem hiding this comment.
👍 thanks Sandeep for the PR
vishramachandran
left a comment
There was a problem hiding this comment.
Thank you for the PR and the extensive unit tests. If you took the pain to write the tests yourself, great work. If you used genAI - kudos for the efficiency! :)
If there is one suggestion I can give it would be to add another test case for counter corrected histogram, which is relevant for cumulative metrics.
|
@vishramachandran Thank you ! we have added the test case for counter correction scenario |
* feat (core): writing part keys to downsample keyspace directly from the raw FiloDB (#2102) * writing part keys to downsample keyspace directly from the raw FiloDB clusters * removing accidental prints --------- Co-authored-by: Kier Petrov <[email protected]> * fixing clippy linting errors (#2104) Co-authored-by: Kier Petrov <[email protected]> * fixing nondeterministic test (#2105) Co-authored-by: Kier Petrov <[email protected]> * adding more debugging for the spec execution (#2106) Co-authored-by: Kier Petrov <[email protected]> * fix(core): adding C* configs for TimeSeriesMemStoreDownsampleIndexSpec (#2107) * adding cassandra host/port config * moving the config out of the spec file to the config file --------- Co-authored-by: Kier Petrov <[email protected]> * use absent(sum(last()) to to implement absent_over_time. (#2108) * use absent(sum(last()) to to implement absent_over_time. --------- Co-authored-by: Yu Zhang <[email protected]> * fix(gateway): Simplify TestTimeSeriesProducer options and fix query link & command (#2111) * Revert "feat(query): make unless operator timestamp-aware and correct test ut…" (#2114) This reverts commit 58411e8. * fix(misc): rename last-delta schema to v2 (#2101) * fix(query) Fix the java.util.NoSuchElementException: head of empty list issue when the inner rvs of a scalar is empty (#2116) * fix(query): Wrong rate function used on cumulative min-max-histograms (#2117) * feat(histogram): OTEL Cummulative enhanced rate + quantile calculation with max-min values (#2119) * feat(histogram): OTEL Cummulative enhanced rate + quantile calculation with max-min * Adding additional unit tests * build(deps): bump oneshot from 0.1.8 to 0.1.13 in /core/src/rust (#2124) Bumps [oneshot](https://github.com/faern/oneshot) from 0.1.8 to 0.1.13. - [Changelog](https://github.com/faern/oneshot/blob/v0.1.13/CHANGELOG.md) - [Commits](faern/oneshot@v0.1.8...v0.1.13) --- updated-dependencies: - dependency-name: oneshot dependency-version: 0.1.13 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(query) Handle cases where non pipe regex are passed gracefully with a BadQueryException instead of default scala.MatchError (#2122) * misc(cluster): Adding helper function for ShardMapperV2 (#2123) * misc(cluster): Adding helper function for ShardMapperV2 * misc(cluster): Adding an async method to get the ShardMapperV2 response (#2127) * misc(cluster): Adding an async method to get the ShardMapperV2 response * fix(query): Handle empty histograms in ScalarOperationMapper to prevent ArrayIndexOutOfBoundsException (#2126) * maint(all): Make changes for code to be compatible to scala 2.13 (#2112) * perf(core): Optimize FilodbMetrics allocation in hot path (#2129) Reduce OTel attribute allocations when additional tags are not involved. This appeared in the allocation hot path where chunks scanned counters are incremented too often resulting in lot of allocations. Perf tests after the change showed that this is not in the hot path anymore. Also fixed a bug where we are relying on a mutable `AttributesBuilder` to create new time series. Fixed it to immutable `Attributes`. * Add preagg delta histogram v2 schema (#2132) * Add preagg delta histogram v2 schema * Fix typo --------- Co-authored-by: Srividhya Anantharamakrishnan <[email protected]> * updating main version --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: kvpetrov <[email protected]> Co-authored-by: Kier Petrov <[email protected]> Co-authored-by: yu-shipit <[email protected]> Co-authored-by: Yu Zhang <[email protected]> Co-authored-by: Vish Ramachandran <[email protected]> Co-authored-by: Amol Nayak <[email protected]> Co-authored-by: Shaik Sher Ali <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: srivik <[email protected]> Co-authored-by: Srividhya Anantharamakrishnan <[email protected]>
Pull Request checklist
Current behavior : Rate on OTEL-Cumulative histograms was not emitting max and min values for more accurate quantile calculation.
New behavior : Added
CumulativeHistRateAndMinMaxFunctionfor passing the captured max and min observations from the users and passing the values to the histogram'squantilefunction.