Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Skip errored runs when computing repetition statistics#2199

Merged
LebedevRI merged 7 commits into
google:mainfrom
Haihan-Jiang:codex/benchmark-skip-error-stats
May 26, 2026
Merged

Skip errored runs when computing repetition statistics#2199
LebedevRI merged 7 commits into
google:mainfrom
Haihan-Jiang:codex/benchmark-skip-error-stats

Conversation

@Haihan-Jiang

Copy link
Copy Markdown
Contributor

Fixes #1406.

Summary

  • compute repetition aggregates from non-errored runs only
  • use the first successful run as the metadata/statistics reference instead of reports[0]
  • ignore skipped runs when building aggregate counter inputs and labels
  • add a regression test for a skipped first run followed by successful repetitions

Validation

  • cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_TESTING=ON -DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON -DCMAKE_CXX_FLAGS="-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1"
  • cmake --build build --target statistics_gtest skip_with_error_test -- -j4
  • ctest --test-dir build -R statistics_gtest|skip_with_error_test --output-on-failure

Note: the explicit CMAKE_CXX_FLAGS include path is only needed on my local macOS CommandLineTools install because its default C++ header search path does not find SDK libc++ headers.

@Haihan-Jiang Haihan-Jiang marked this pull request as ready for review May 24, 2026 18:54
Comment thread src/statistics.cc Outdated
Comment thread test/statistics_gtest.cc Outdated

@LebedevRI LebedevRI left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clang-format the diff

@Haihan-Jiang

Copy link
Copy Markdown
Contributor Author

Applied clang-format 18 to the touched test file and pushed c0a4bda. Local validation:

  • /Users/haihan/Library/Python/3.9/bin/clang-format --dry-run --Werror src/statistics.cc test/repetitions_test.cc
  • cmake --build build --target statistics_gtest skip_with_error_test -- -j4
  • ctest --test-dir build -R 'statistics_gtest|skip_with_error_test' --output-on-failure

Comment thread src/statistics.cc Outdated
Comment on lines 132 to 133
@@ -129,14 +134,17 @@ std::vector<BenchmarkReporter::Run> ComputeStats(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should probably also use successful_count?

Comment thread src/statistics.cc Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

@Haihan-Jiang

Copy link
Copy Markdown
Contributor Author

I do not see any active threads left now, and the clang-format check is passing after c0a4bda. Happy to make another pass if I missed something.

@LebedevRI

Copy link
Copy Markdown
Collaborator

I do not see any active threads left now, and the clang-format check is passing after c0a4bda. Happy to make another pass if I missed something.

There are last two inline comments.

@Haihan-Jiang

Copy link
Copy Markdown
Contributor Author

Thanks, I missed those two. Updated in e5409c3: the time accumulators and per-counter buffers now reserve the successful run count as well. I also added the explicit size_type cast so the warning flags stay clean.

Locally checked with clang-format --dry-run, the statistics/skip-with-error build targets, and the matching ctest filter.

Comment thread src/statistics.cc Outdated
std::find_if(reports.begin(), reports.end(), is_successful);
const auto successful_count =
std::count_if(reports.begin(), reports.end(), is_successful);
const auto successful_count = static_cast<std::vector<Run>::size_type>(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Why, C++, why?!
  2. Let's just hardcode size_t here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7caecf6. I switched the cast to size_t and reran the same local checks: clang-format dry run, the statistics/skip-with-error build targets, and the matching ctest filter.

@LebedevRI LebedevRI left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@LebedevRI LebedevRI merged commit d6303b2 into google:main May 26, 2026
88 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Uninitialized pointer access when using --benchmark_repetitions in some scenarios

2 participants