Use trie abstractions for batch implementations #616
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the bespoke trie logic in ord_neu.rs into reusable “layer” abstractions (Vals, Upds, and the UpdsBuilder), and updates both the value‐batch and key‐batch implementations (and the columnar.rs example) to use these new layers instead of manual offset handling and singleton optimizations.
- Introduce
layersmodule withValsandUpdscontainers and anUpdsBuilderhelper - Replace manual offset arrays and singleton‐optimization code in
OrdValStorage,OrdKeyStorage, and their builders - Update the
columnar.rsexample to use the new abstractions
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| differential-dataflow/src/trace/implementations/ord_neu.rs | Extracted trie layers into layers::{Vals, Upds} and rewrote val_batch/key_batch to use them |
| differential-dataflow/examples/columnar.rs | Updated example builders to leverage Vals, Upds, and UpdsBuilder |
Comments suppressed due to low confidence (2)
differential-dataflow/src/trace/implementations/ord_neu.rs:161
- The doc comment refers to
self.vals, but this method actually usesself.offsto compute offsets intotimes/diffs. Update the comment to accurately describe what is being bounded (e.g., "offsets intotimesanddiffs").
/// Lower and upper bounds in `self.vals` of the indexed list.
differential-dataflow/src/trace/implementations/ord_neu.rs:143
- [nitpick] The abbreviation
Updsmay be unclear to new readers. Consider renaming to a more descriptive name (e.g.,Updates) to align withValsand improve readability.
pub struct Upds<O, T, D> {
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our
ord_neu.rsbatch/trace implementations use a manual implementation of a short trie. Rather than have the work happen in bespoke methods we need to copy/paste, extract the logic out into trie "layers" which can be composed. For example, the "singleton optimization" for updates lived in four locations, with a fifth inrhh.rs. This change moves that to be one location, in the update trie layer, used by all four.This is the first step in trying to make these types more "trie-forward", revealing their layered structure rather than living behind abstractions that conceal the structure. The goal for the moment is to get a sense for what the code looks like when you compartmentalize and modularize the logic and data. So far, pretty good!
Historically we had something similar, though it was more complicated than it needed to be. The reason seems to be that we previously had as trie layers pairs
(Vec<T>, Vec<usize>)to indicate a list of keys and their offsets in the next layer. It turns out that(Vec<usize>, Vec<T>)is a better representation, with fewer cross-layer dependencies.