Codestin Search App

kosiew · 2026-02-26T08:43:36Z

Which issue does this PR close?

Part of Speedup execution of sqllogictests with more parallelization #20524.

Rationale for this change

The sqllogictest runner executes files in parallel, but it was hard to pinpoint which test files dominate wall-clock time. This change adds deterministic per-file elapsed timing observability so we can identify long-tail files and prioritize follow-up optimization work, while keeping default output usable for both local development (TTY) and CI (non-TTY).

What changes are included in this PR?

Collect per-file elapsed durations in the sqllogictest runner and aggregate them at end-of-run.
Print a deterministic timing summary (stable sort: elapsed desc, path asc; stable formatting) via MultiProgress to avoid interleaved progress-bar noise.
Add CLI flags and environment variables to control output:
- --timing-summary auto|off|top|full (also SLT_TIMING_SUMMARY)
- --timing-top-n <N> (also SLT_TIMING_TOP_N, must be >= 1)
Default behavior:
- auto maps to off for local TTY runs and top for CI/non-TTY runs.
Add optional debug logging for slow files (over 30s) behind SLT_TIMING_DEBUG_SLOW_FILES=1.
Update datafusion/sqllogictest/README.md with usage examples.

Are these changes tested?

Covered by existing sqllogictests integration test execution; no new unit tests were added.
Manual validation plan (ran locally / in CI as applicable):
- cargo test --test sqllogictests -- push_down_filter_ --test-threads 16
- cargo test --test sqllogictests -- --test-threads 16
- cargo test --test sqllogictests -- --timing-summary top --timing-top-n 10
- cargo test --test sqllogictests -- --timing-summary full
Verified output properties:
- Summary ordering is deterministic across repeated runs (elapsed desc, path asc).
- auto mode is quiet on TTY but prints a top-N summary on non-TTY/CI.
- Pass/fail behavior and error reporting are unchanged.

Are there any user-facing changes?

Yes (test-runner UX only):

New optional timing summary output for sqllogictests.
New CLI flags / env vars documented in datafusion/sqllogictest/README.md:
- --timing-summary auto|off|top|full / SLT_TIMING_SUMMARY
- --timing-top-n <N> / SLT_TIMING_TOP_N
- SLT_TIMING_DEBUG_SLOW_FILES=1 (optional debug logging for slow files >30s)

No public DataFusion APIs are changed.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.

Capture elapsed time once in spawned per-file task and reuse after join. Remove redundant post-join measurement while maintaining existing error behavior. Implement safe fallback to Duration::ZERO for join-level panics or errors where elapsed time is not available.

Ensure --timing-top-n accepts only values >= 1 by using clap's value parser with a defined range. Update help text to reflect this new requirement and clarify in README.md to avoid silent runtime coercion.

Clarify default behavior for timing summaries in TTY and non-TTY/CI runs. Maintain conciseness within the existing timing-summary section.

Replace Clap parser call in sqllogictests.rs:949 with a custom parser function. Add validation to ensure usize values are >= 1 in lines 433-443, providing a clear error message for any input of 0.

martin-g · 2026-02-26T09:52:46Z

datafusion/sqllogictest/bin/sqllogictests.rs

+
+    let top_n = options.timing_top_n;
+    let count = match mode {
+        TimingSummaryMode::Off => 0,


nit: This is already handled at line 391.

Good catch. I will remove the redundant TimingSummaryMode::Off handling from the count calculation since Off already returns early.

martin-g · 2026-02-26T09:55:19Z

datafusion/sqllogictest/bin/sqllogictests.rs

+    let top_n = options.timing_top_n;
+    let count = match mode {
+        TimingSummaryMode::Off => 0,
+        TimingSummaryMode::Auto | TimingSummaryMode::Top => top_n,


nit: mode cannot be TimingSummaryMode::Auto because Options::timing_summary_mode() does not return it. But it is not doing any harm either.

Agreed. timing_summary_mode() normalizes Auto to Top/Off before this point.
I will update the branch logic to only rely on Top vs Full and add a debug_assert! to document/enforce that invariant in debug builds.

…n mode

kosiew · 2026-02-26T11:33:09Z

Thanks @martin-g for the quick review.

alamb

Thanks @kosiew and @martin-g

This feature will be super helpful to try and schedule the tests more carefully if we go with #20576

One quick thought I had while skimming this PR was I wonder if we really need all the different modes.

It seems the key thing that we can't do without changes to sqllogictests itself is get the per-file timing. However, everything else we could do with post run scripts,.

For example, rather than adding a special flag --timings-top-n 10 maybe we could follow the unix philosophy and pipe the output to head -n 10

cargo test --test sqllogictests -- --timing-summary | head -n 10

Just a thought to keep the code a bit simpler

alamb · 2026-02-26T14:05:04Z

datafusion/sqllogictest/bin/sqllogictests.rs

            ColorChoice::Auto => {
                // CARGO_TERM_COLOR takes precedence over auto-detection
-                let cargo_term_color = ColorChoice::from_str(
+                let cargo_term_color = <ColorChoice as FromStr>::from_str(


is this needed?

It is needed because clap::ValueEnum is in scope too.
https://docs.rs/clap/latest/clap/trait.ValueEnum.html#method.from_str

Streamline timing summary to a single switch, enabling full deterministic per-file timings sorted slowest-first. Eliminate all mode and top-N options in sqllogictests.rs, including the removal of TimingSummaryMode and related auto branching for summary output. Update README.md to recommend Unix post-processing with `| head -n 10`.

kosiew · 2026-02-27T09:14:25Z

For example, rather than adding a special flag --timings-top-n 10 maybe we could follow the unix philosophy and pipe the output to head -n 10 ...a thought to keep the code a bit simpler

Agreed and simplified.

❯ cargo test --test sqllogictests -- --timing-summary
...
Running with 10 test threads (available parallelism: 10)
Per-file elapsed summary (deterministic):                                                 
1.   18.405s  push_down_filter.slt                                                      
2.    6.874s  joins.slt                                                                 
3.    6.713s  aggregate.slt 
...
408.    0.001s  avro.slt                                                                
Completed 408 test files in 19 seconds

kosiew · 2026-02-27T09:28:11Z

hmmm....

❯ cargo test --test sqllogictests -- --timing-summary 2>&1| head -n 10
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running bin/sqllogictests.rs (target/debug/deps/sqllogictests-47b2fcb888654300)
Running with 10 test threads (available parallelism: 10)
Progress: 50/408 files completed (12%)
Progress: 100/408 files completed (25%)
Progress: 150/408 files completed (37%)
Progress: 200/408 files completed (49%)
Progress: 250/408 files completed (61%)
Progress: 300/408 files completed (74%)
Progress: 350/408 files completed (86%)

It's not as straightforward as I thought.
I'll merge before the simplification and work on simplifying it as a follow up.

This reverts commit 2eb94d4.

alamb · 2026-02-27T12:00:16Z

hmmm....

❯ cargo test --test sqllogictests -- --timing-summary 2>&1| head -n 10
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.66s
     Running bin/sqllogictests.rs (target/debug/deps/sqllogictests-47b2fcb888654300)
Running with 10 test threads (available parallelism: 10)
Progress: 50/408 files completed (12%)
Progress: 100/408 files completed (25%)
Progress: 150/408 files completed (37%)
Progress: 200/408 files completed (49%)
Progress: 250/408 files completed (61%)
Progress: 300/408 files completed (74%)
Progress: 350/408 files completed (86%)

It's not as straightforward as I thought. I'll merge before the simplification and work on simplifying it as a follow up.

looks good -- thanks!

If possibility is to avoid printing progress when in "timing mode" and only print out the overall runtime

kosiew added 7 commits February 26, 2026 15:50

Add timing summary feature for SQL logic tests

9815ee4

Add per-file timing summary feature to README for sqllogictests

76c7979

Refactor color choice parsing to use associated type for clarity

5b81cfe

Add optional debug logging for slow test files in sqllogictests

6f18799

Enforce parse-time validation for --timing-top-n

080e53a

Ensure --timing-top-n accepts only values >= 1 by using clap's value parser with a defined range. Update help text to reflect this new requirement and clarify in README.md to avoid silent runtime coercion.

Update README.md for timing summary behavior

7f1ee85

Clarify default behavior for timing summaries in TTY and non-TTY/CI runs. Maintain conciseness within the existing timing-summary section.

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Feb 26, 2026

kosiew added 2 commits February 26, 2026 16:48

Fix parser for usize in sqllogictests.rs

57c472e

Replace Clap parser call in sqllogictests.rs:949 with a custom parser function. Add validation to ensure usize values are >= 1 in lines 433-443, providing a clear error message for any input of 0.

clippy fix

88714e9

kosiew mentioned this pull request Feb 26, 2026

Split push_down_filter.slt into standalone sqllogictest files to reduce long-tail runtime #20566

Merged

kosiew marked this pull request as ready for review February 26, 2026 09:24

martin-g approved these changes Feb 26, 2026

View reviewed changes

Refactor timing summary logic to simplify count determination based o…

789e64e

…n mode

alamb reviewed Feb 26, 2026

View reviewed changes

alamb added the development-process Related to development process of DataFusion label Feb 26, 2026

github-actions bot removed the development-process Related to development process of DataFusion label Feb 27, 2026

Revert "Simplify timing UX and remove complexity"

b133317

This reverts commit 2eb94d4.

kosiew added this pull request to the merge queue Feb 27, 2026

Merged via the queue into apache:main with commit e583fe9 Feb 27, 2026
28 checks passed

kosiew mentioned this pull request Feb 27, 2026

Simplify sqllogictest timing summary to boolean flag and remove top-N modes #20598

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add deterministic per-file timing summary to sqllogictest runner#20569

Add deterministic per-file timing summary to sqllogictest runner#20569
kosiew merged 12 commits intoapache:mainfrom
kosiew:sqllogictest-runtime-observability-20524a

kosiew commented Feb 26, 2026 •

edited

Loading

Uh oh!

martin-g Feb 26, 2026

Uh oh!

kosiew Feb 26, 2026

Uh oh!

martin-g Feb 26, 2026

Uh oh!

kosiew Feb 26, 2026

Uh oh!

kosiew commented Feb 26, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Feb 26, 2026

Uh oh!

martin-g Feb 26, 2026

Uh oh!

kosiew commented Feb 27, 2026

Uh oh!

kosiew commented Feb 27, 2026

Uh oh!

Uh oh!

alamb commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kosiew commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

LLM-generated code disclosure

Uh oh!

martin-g Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew commented Feb 26, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

martin-g Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew commented Feb 27, 2026

Uh oh!

kosiew commented Feb 27, 2026

Uh oh!

Uh oh!

alamb commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kosiew commented Feb 26, 2026 •

edited

Loading