[dbsp] TopN and BottomN GC. #5460

ryzhyk · 2026-01-19T05:01:00Z

Partially addresses #1975

We implement two new GC strategies:

Top-N: Keep N largest value below the waterline. Unlike Last-N, this strategy doesn's assume that values are sorted by timestamp. Can be used to GC the Max aggregate and the top-k group transformer.
Bottom-N: Keep N smallest values below the waterline. Doesn't assume that values are sorted by timestamp. Can be used to implement bottom-k and Min. Can be also used to implement MinSome, but this requires setting N to 2 to account for the potential None value.

Requires compiler changes.

mihaibudiu · 2026-01-19T22:15:14Z

crates/dbsp/src/trace/cursor.rs

crates/dbsp/src/operator/accumulate_trace.rs

mihaibudiu · 2026-01-21T23:48:34Z

I have implemented MIN, MAX, ARG_MIN, and ARG_MAX for now, I am thinking to leave the others for a separate PR.

mihaibudiu · 2026-01-21T23:51:55Z

Other features to build also tracked in #1850

Copilot

Pull request overview

This PR implements two new garbage collection (GC) strategies for DBSP: Top-N and Bottom-N. These strategies enable efficient memory management for aggregate operations like MAX, MIN, and top-k/bottom-k transformers by retaining only the N largest or smallest values below the waterline, without assuming that values are sorted by timestamp.

Changes:

Renamed DBSPIntegrateTraceRetainValuesLastNOperator to DBSPIntegrateTraceRetainNValuesOperator to support multiple retention strategies (LastN, TopN, BottomN)
Added new GroupFilter variants (TopN and BottomN) with corresponding cursor implementations
Integrated new GC strategies into the monotonicity analysis pipeline to automatically apply them for MIN/MAX aggregates
Added comprehensive test coverage including property-based tests

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/tools/CompilerCircuitStream.java`	Added overloaded `step` method accepting `Change` objects directly
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/tools/Change.java`	Updated comment to clarify shuffle parameter
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/tools/BaseSQLTests.java`	Changed visibility of `currentTestInformation` to protected
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/streaming/StreamingTests.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/streaming/LatenessTests.java`	Added comprehensive test suite for MIN/MAX with lateness using new GC strategies
`sql-to-dbsp-compiler/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/simple/IncrementalRegressionTests.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/ir/expression/DBSPZSetExpression.java`	Added `@CheckReturnValue` annotations and improved multi-line formatting
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/ir/expression/DBSPExpression.java`	Added `@CheckReturnValue` annotation to `add` method
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/recursive/ValidateRecursiveOperators.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/InsertLimiters.java`	Added logic to determine which GC strategy to use based on aggregate type and integrated TopN/BottomN for MIN/MAX
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/CheckRetain.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/StrayGC.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/LowerAsof.java`	Updated references and comments for renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/FindDeadCode.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/CircuitVisitor.java`	Updated method signatures for renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/CircuitRewriter.java`	Updated method signatures and field references for renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/CircuitCloneVisitor.java`	Updated method signatures for renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/visitors/monotone/PartiallyMonotoneTuple.java`	Changed `mayBeNull` field visibility to public
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/frontend/calciteCompiler/optimizer/CalciteOptimizer.java`	Added Calcite optimization rules for single values
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/backend/rust/ToRustVisitor.java`	Updated references to renamed operator class
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/backend/dot/ToDotNodesVisitor.java`	Simplified color logic for retain operators
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/compiler/backend/ToJsonOuterVisitor.java`	Updated serialization to use `which` enum instead of boolean flag
`sql-to-dbsp-compiler/SQL-compiler/src/main/java/org/dbsp/sqlCompiler/circuit/operator/DBSPIntegrateTraceRetainValuesLastNOperator.java`	Renamed class to `DBSPIntegrateTraceRetainNValuesOperator` and added `WhichN` enum to support multiple strategies
`crates/dbsp/src/trace/test/test_batch.rs`	Added test implementations for TopN and BottomN filters
`crates/dbsp/src/trace/filter.rs`	Added `TopN` and `BottomN` variants to `GroupFilter` enum
`crates/dbsp/src/trace/cursor.rs`	Implemented cursor logic for TopN and BottomN filters
`crates/dbsp/src/operator/trace.rs`	Added public API for top-n trace retention
`crates/dbsp/src/operator/dynamic/trace.rs`	Added dynamic API and tests for top-n trace retention
`crates/dbsp/src/operator/dynamic/aggregate.rs`	Added property-based tests for MAX with retention and fixed typo
`crates/dbsp/src/operator/dynamic/accumulate_trace.rs`	Added dynamic APIs for top-n and bottom-n accumulate trace retention
`crates/dbsp/src/operator/accumulate_trace.rs`	Added public APIs for top-n and bottom-n accumulate trace retention

.../src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/InsertLimiters.java

crates/dbsp/src/trace/cursor.rs

...er/SQL-compiler/src/test/java/org/dbsp/sqlCompiler/compiler/sql/streaming/LatenessTests.java

.../src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/InsertLimiters.java

ryzhyk · 2026-01-22T00:07:34Z

.../src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/InsertLimiters.java

+                            }
+                            case ArgMinSome -> DBSPIntegrateTraceRetainNValuesOperator.WhichN.BottomN;
+                            case MinSome1 -> {
+                                limit = 2;


A comment explaining why this needs to be 2 may be helpful.

I thought there was some explanation somewhere in DBSP, but I could not find it.

We implement two new GC strategies: - Top-N: Keep N largest value below the waterline. Unlike Last-N, this strategy doesn's assume that values are sorted by timestamp. Can be used to GC the Max aggregate and the top-k group transformer. - Bottom-N: Keep N smallest values below the waterline. Doesn't assume that values are sorted by timestamp. Can be used to implement bottom-k and Min. Can be also used to implement MinSome, but this requires setting N to 2 to account for the potential None value. Signed-off-by: Leonid Ryzhyk <[email protected]>

Signed-off-by: Mihai Budiu <[email protected]>

ryzhyk added the DBSP core Related to the core DBSP library label Jan 19, 2026

ryzhyk force-pushed the top-k-gc branch from 2424756 to ab72268 Compare January 19, 2026 05:08

mihaibudiu approved these changes Jan 19, 2026

View reviewed changes

crates/dbsp/src/trace/cursor.rs Show resolved Hide resolved

crates/dbsp/src/operator/accumulate_trace.rs Show resolved Hide resolved

gz added the marketing Relevant for marketing content label Jan 20, 2026

mihaibudiu force-pushed the top-k-gc branch from ab72268 to 7203804 Compare January 21, 2026 23:45

mihaibudiu added the SQL compiler Related to the SQL compiler label Jan 21, 2026

mihaibudiu marked this pull request as ready for review January 21, 2026 23:48

Copilot AI review requested due to automatic review settings January 21, 2026 23:48

Copilot AI reviewed Jan 21, 2026

View reviewed changes

.../src/main/java/org/dbsp/sqlCompiler/compiler/visitors/outer/monotonicity/InsertLimiters.java Outdated Show resolved Hide resolved

crates/dbsp/src/trace/cursor.rs Outdated Show resolved Hide resolved

ryzhyk commented Jan 22, 2026

View reviewed changes

mihaibudiu force-pushed the top-k-gc branch from 7203804 to f891f61 Compare January 22, 2026 01:24

mihaibudiu added this pull request to the merge queue Jan 22, 2026

mihaibudiu removed this pull request from the merge queue due to a manual request Jan 22, 2026

mihaibudiu force-pushed the top-k-gc branch 2 times, most recently from 8eb1e53 to 5d4ec58 Compare January 22, 2026 05:33

mihaibudiu enabled auto-merge January 22, 2026 05:33

mihaibudiu added this pull request to the merge queue Jan 22, 2026

mihaibudiu removed this pull request from the merge queue due to a manual request Jan 22, 2026

mihaibudiu force-pushed the top-k-gc branch from 5d4ec58 to a633b0a Compare January 22, 2026 06:17

mihaibudiu enabled auto-merge January 22, 2026 06:17

mihaibudiu added this pull request to the merge queue Jan 22, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 22, 2026

mihaibudiu force-pushed the top-k-gc branch from a633b0a to 0ff0ab9 Compare January 22, 2026 17:12

ryzhyk and others added 2 commits January 22, 2026 09:12

[SQL] Compiler support for GC for TopK operators

f31c2fa

Signed-off-by: Mihai Budiu <[email protected]>

mihaibudiu force-pushed the top-k-gc branch from 0ff0ab9 to f31c2fa Compare January 22, 2026 17:13

mihaibudiu enabled auto-merge January 22, 2026 17:13

mihaibudiu added this pull request to the merge queue Jan 22, 2026

Merged via the queue into main with commit 164f3af Jan 22, 2026
1 check passed

mihaibudiu deleted the top-k-gc branch January 22, 2026 18:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dbsp] TopN and BottomN GC. #5460

[dbsp] TopN and BottomN GC. #5460

Uh oh!

ryzhyk commented Jan 19, 2026

Uh oh!

mihaibudiu commented Jan 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

mihaibudiu commented Jan 21, 2026 •

edited

Loading

Uh oh!

mihaibudiu commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryzhyk Jan 22, 2026

Uh oh!

mihaibudiu Jan 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[dbsp] TopN and BottomN GC. #5460

[dbsp] TopN and BottomN GC. #5460

Uh oh!

Conversation

ryzhyk commented Jan 19, 2026

Uh oh!

mihaibudiu commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mihaibudiu commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mihaibudiu commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryzhyk Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mihaibudiu commented Jan 19, 2026 •

edited

Loading

mihaibudiu commented Jan 21, 2026 •

edited

Loading