dbsp: tracking bloom filter #5500

snkas · 2026-01-23T14:58:55Z

Tracks for each Bloom filter the number of hits and misses. They are included in the statistics that can be retrieved by the spine. The spine sums up over all the Bloom filter statistics it receives from its batches, which loses insight into individual batch statistics. The spine reports three new metadata fields:

"Bloom filter hits": usize integer
"Bloom filter misses": usize integer
"Bloom filter hit rate": percentage

The Web Console profiler is updated to display the new metadata fields.

PR information:

Documentation: not updated
Changelog: not updated
Backward incompatible changes: only new metadata fields are added, and the Web Console profiler has been set to display it. I've checked that it is able to load profiles from before the new metadata fields are added.
Fixes: [dbsp] Report Bloom filter hit rate #5439

snkas · 2026-01-23T15:17:11Z

Using the following SQL:

CREATE TABLE t1 (
    id BIGINT NOT NULL PRIMARY KEY
) WITH (
    'connectors' = '[{
        "name": "gen1",
        "transport": {
            "name": "datagen",
            "config": {
                "plan": [{
                    "limit": 100000000000, 
                    "rate": 1000000,
                    "fields": {
                        "id": { "strategy": "uniform", "range": [0, 10000] }
                    }
                }]
            } 
        }
    }]'
);

... varying runtime configuration min_storage_bytes gives:

min_storage_bytes=0: some ~100%, some ~65%, some ~20%, some ~0%
min_storage_bytes=1000: some ~100%, some ~63%, some 7%, some 0%
min_storage_bytes=10000: some ~100%, several ~60%, one 25%

(The above vary per run)
The most interesting/consistent one to debug/investigate is min_storage_bytes=10000.

Possible alterations to the program is increasing the tuple size by adding:

# Add column:
data VARCHAR NOT NULL

# Add to datagen:
"data": { "strategy": "paragraphs", "range": [0, 100] }

blp · 2026-01-23T17:23:58Z

I was picturing this being reported as a metric on spines, in AsyncMerger::metadata, rather than through the log. If it's reported in the per-spine metadata then we'll get it in profiles "for free" without having to look through the log.

blp · 2026-01-23T17:26:36Z

I think I'd use a pair of AtomicU64s instead of Mutexes. They should be cheaper.

Copilot

Pull request overview

This PR introduces tracking capabilities for Bloom filters used in DBSP batch operations, capturing hit/miss statistics for performance analysis. The changes enable monitoring of Bloom filter effectiveness across the storage layer and make these metrics visible in the Web Console profiler.

Changes:

Adds TrackingBloomFilter wrapper around BloomFilter to count hits and misses using atomic counters
Refactors filter_size() method to filter_stats() throughout the codebase to return comprehensive statistics
Updates the profiler to display three new metadata fields: hits, misses, and hit rate

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
js-packages/profiler-lib/src/profile.ts	Adds new Bloom filter metrics to measurement categories
crates/dbsp/src/storage/tracking_bloom_filter.rs	Implements new `TrackingBloomFilter` with hit/miss tracking
crates/dbsp/src/storage/file/writer.rs	Updates writer to use `TrackingBloomFilter`
crates/dbsp/src/storage/file/reader.rs	Updates reader to use `TrackingBloomFilter` and expose stats
crates/dbsp/src/storage/file/format.rs	Updates format conversions for `TrackingBloomFilter`
crates/dbsp/src/trace.rs	Changes `BatchReader` trait method from `filter_size()` to `filter_stats()`
crates/dbsp/src/trace/spine_async.rs	Aggregates filter stats across batches and adds new metadata fields
crates/dbsp/src/trace/spine_async/snapshot.rs	Updates snapshot to use `filter_stats()`
crates/dbsp/src/trace/test/test_batch.rs	Updates test batch to return default stats
crates/dbsp/src/trace/ord/vec/*.rs	Updates vec-based batches to return default stats
crates/dbsp/src/trace/ord/file/*.rs	Updates file-based batches to delegate to file reader
crates/dbsp/src/trace/ord/fallback/*.rs	Updates fallback batches to delegate to inner implementation
crates/dbsp/src/circuit/metadata.rs	Adds `Float` variant to `MetaItem` enum
crates/dbsp/src/storage.rs	Exports new `tracking_bloom_filter` module

crates/dbsp/src/trace/spine_async.rs

crates/dbsp/src/storage/tracking_bloom_filter.rs

snkas · 2026-01-28T17:58:36Z

Screenshot of profiler:

snkas · 2026-01-28T17:59:23Z

Ready for review!

crates/dbsp/src/trace/spine_async.rs

mihaibudiu · 2026-01-28T18:06:21Z

js-packages/profiler-lib/src/profile.ts

        "bounds",
        "Bloom filter size",
-        "Bloom filter bits/key"]);
+        "Bloom filter bits/key",


these will need to change once we merge #5514, but let's see which PR lands first

Let's see, this one is current queued, so if it (hopefully) passes CI, the other PR will land after.

mihaibudiu · 2026-01-28T18:17:17Z

js-packages/profiler-lib/src/profile.ts

+        "Bloom filter bits/key",
+        "Bloom filter hits",
+        "Bloom filter misses",
+        "Bloom filter hit rate"]);


hit rate should move below, at the PercentValue case.

Thanks! I think you meant line 489 needs to be moved upward, line 380 is the category mapping.

blp

Well done, thank you

Tracks for each Bloom filter the number of hits and misses. They are included in the statistics that can be retrieved by the spine. The spine sums up over all the Bloom filter statistics it receives from its batches, which loses insight into individual batch statistics. The spine reports three new metadata fields: - "Bloom filter hits": usize integer - "Bloom filter misses": usize integer - "Bloom filter hit rate": percentage The Web Console profiler is updated to display the new metadata fields. Signed-off-by: Simon Kassing <[email protected]>

snkas · 2026-01-28T18:33:43Z

Updated screenshot of profiler:

snkas mentioned this pull request Jan 23, 2026

[dbsp] Report Bloom filter hit rate #5439

Closed

snkas force-pushed the bloom-filter-hit-rate branch from f28c4d6 to 14c154a Compare January 28, 2026 17:18

snkas marked this pull request as ready for review January 28, 2026 17:26

Copilot AI review requested due to automatic review settings January 28, 2026 17:26

Copilot AI reviewed Jan 28, 2026

View reviewed changes

crates/dbsp/src/trace/spine_async.rs Outdated Show resolved Hide resolved

crates/dbsp/src/storage/tracking_bloom_filter.rs Show resolved Hide resolved

snkas temporarily deployed to ci January 28, 2026 17:53 — with GitHub Actions Inactive

snkas requested review from blp, gz and ryzhyk January 28, 2026 17:55

mihaibudiu approved these changes Jan 28, 2026

View reviewed changes

snkas force-pushed the bloom-filter-hit-rate branch from 14c154a to 9e31ddf Compare January 28, 2026 18:14

mihaibudiu reviewed Jan 28, 2026

View reviewed changes

snkas force-pushed the bloom-filter-hit-rate branch from 9e31ddf to ba8cca4 Compare January 28, 2026 18:17

blp approved these changes Jan 28, 2026

View reviewed changes

snkas force-pushed the bloom-filter-hit-rate branch from ba8cca4 to 13d975d Compare January 28, 2026 18:32

snkas added this pull request to the merge queue Jan 28, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 28, 2026

snkas added this pull request to the merge queue Jan 29, 2026

Merged via the queue into main with commit 247b89e Jan 29, 2026
1 check passed

snkas deleted the bloom-filter-hit-rate branch January 29, 2026 12:59

dbsp: tracking bloom filter #5500

dbsp: tracking bloom filter #5500

Uh oh!

Conversation

snkas commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snkas commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blp commented Jan 23, 2026

Uh oh!

blp commented Jan 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

snkas commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snkas commented Jan 28, 2026

Uh oh!

Uh oh!

mihaibudiu Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

snkas Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

mihaibudiu Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

snkas Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

blp left a comment

Choose a reason for hiding this comment

Uh oh!

snkas commented Jan 28, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

snkas commented Jan 23, 2026 •

edited

Loading

snkas commented Jan 23, 2026 •

edited

Loading

snkas commented Jan 28, 2026 •

edited

Loading