Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Demonstrate Columnar batch builder #602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 24, 2025
Merged

Demonstrate Columnar batch builder #602

merged 4 commits into from
Jun 24, 2025

Conversation

frankmcsherry
Copy link
Member

The columnar.rs example is extended to use columnar containers for the merge batcher, allowing most of the merge batcher state to be in serialized form. The runtimes of the example are so

     Running `target/release/examples/columnar 1000000 1000`
3.344818375s    loading complete
7.658054084s    queries complete
7.662470667s    shut down

versus the current spines.rs example

     Running `target/release/examples/spines 1000000 1000 new`
Running ["new"] arrangement
2.845316875s    loading complete
6.4664785s      queries complete
6.469992416s    shut down

@frankmcsherry frankmcsherry force-pushed the columnar_builder branch 2 times, most recently from 5bb9784 to cecae53 Compare June 20, 2025 23:30
antiguru
antiguru approved these changes Jun 24, 2025
View reviewed changes
Copy link
Member

@antiguru antiguru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@antiguru antiguru marked this pull request as ready for review June 24, 2025 11:41
@antiguru
Copy link
Member

The last commit restores performance by producing a key batch instead of a val batch:

➜  differential-dataflow git:(columnar_builder) cargo run --release --example spines -- 1000000 1000 new
   Compiling differential-dataflow v0.15.3 (/home/moritz/dev/repos/differential-dataflow/differential-dataflow)
    Finished `release` profile [optimized + debuginfo] target(s) in 38.86s
     Running `target/release/examples/spines 1000000 1000 new`
Running ["new"] arrangement
4.410642495s    loading complete
10.006482626s   queries complete
10.012127944s   shut down
➜  differential-dataflow git:(columnar_builder) cargo run --release --example columnar -- 1000000 1000
    Finished `release` profile [optimized + debuginfo] target(s) in 0.03s
     Running `target/release/examples/columnar 1000000 1000`
4.089337934s    loading complete
9.359655481s    queries complete
9.365747091s    shut down

@frankmcsherry frankmcsherry merged commit 3418974 into TimelyDataflow:master Jun 24, 2025
5 checks passed
@github-actions github-actions bot mentioned this pull request Jun 24, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@antiguru antiguru antiguru approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@frankmcsherry @antiguru