-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
du: Add parallel traversal for performance #9195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
du: Add parallel traversal for performance #9195
Conversation
|
please provide benchmark with hyperfine gnu |
|
GNU testsuite comparison: |
Introduces parallel directory traversal using Rayon to significantly
improve performance on large directory trees.
The implementation is conservative, only activating on directories with
more than 4 subdirectories to avoid threading overhead on small workloads.
Benchmark on a 219 GB directory (172k subdirs, 1.17M files):
Sequential: 8.880s
Parallel: 4.687s
Speedup: 1.90x
d15d7bc to
16bb63b
Compare
CodSpeed Performance ReportMerging #9195 will degrade performances by 3.25%Comparing Summary
Benchmarks breakdown
Footnotes
|
|
GNU testsuite comparison: |
|
[founder@quantum-encoding-ltd coreutils]$ unset LD_PRELOAD === BUILD === === BENCHMARK 1: random stuff (219 GB, 172K dirs, 1.17M files) === Benchmark 2: ./target/release/du -sh /home/founder/storage_transfer Summary === BENCHMARK 2: /usr === Benchmark 2: sudo ./target/release/du -sh /usr Warning: The first benchmarking run for this command was significantly slower than the rest (2.361 s). This could be caused by (filesystem) caches that were not filled until after the first run. You are already using the '--warmup' option which helps to fill these caches before the actual benchmark. You can either try to increase the warmup count further or re-run this benchmark on a quiet system in case it was a random outlier. Alternatively, consider using the '--prepare' option to clear the caches before each timing run. Summary === BENCHMARK 3: /usr/lib === Benchmark 2: sudo ./target/release/du -sh /usr/lib Summary === BENCHMARK 4: /usr/share === Benchmark 2: sudo ./target/release/du -sh /usr/share Summary === BENCHMARK 5: /var === Benchmark 2: sudo ./target/release/du -sh /var Summary [founder@quantum-encoding-ltd coreutils]$ ./benchmark_multi_dirs.sh [1/3] Benchmarking /usr/lib Benchmark 2: sudo ./target/release/du -sh /usr/lib Summary [2/3] Benchmarking /usr/share Benchmark 2: sudo ./target/release/du -sh /usr/share Summary [3/3] Benchmarking /var Benchmark 2: sudo ./target/release/du -sh /var Summary |
Introduces parallel directory traversal using Rayon to significantly improve performance on large directory trees.
The implementation is conservative, only activating on directories with more than 4 subdirectories to avoid threading overhead on small workloads.
Benchmark on a 219 GB directory (172k subdirs, 1.17M files):