Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Benchmarking][CI] Benchmarking buildkite runs fail to build Arrow #45939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
raulcd opened this issue Mar 26, 2025 · 5 comments
Closed

[Benchmarking][CI] Benchmarking buildkite runs fail to build Arrow #45939

raulcd opened this issue Mar 26, 2025 · 5 comments
Assignees
Milestone

Comments

@raulcd
Copy link
Member

raulcd commented Mar 26, 2025

Describe the bug, including details regarding any error messages, version, and platform.

As noted by @zanmato1984 on this PR:

Benchmarks have stopped running. The latest benchmarked commit PR merged is:
#45929
The next one merged doesn't seem to have benchmarks run:
#45907
Which matches with the runs here:
https://conbench.ursa.dev/
It does seem that buildkite runs are failing with the following error:

[445/640] Building CXX object src/arrow/c/CMakeFiles/arrow-bridge-benchmark.dir/bridge_benchmark.cc.o
FAILED: src/arrow/c/CMakeFiles/arrow-bridge-benchmark.dir/bridge_benchmark.cc.o
/var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/ccache /var/lib/buildkite-agent/.conda/envs/arrow-commit/bin/x86_64-conda-linux-gnu-c++ -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_WITH_BENCHMARKS_REFERENCE -DARROW_WITH_TIMING_TESTS -I/tmp/5909f767-7a62-48fc-a33c-79508cc22d8b/cpp/src -I/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src -I/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/generated -isystem /var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/thirdparty/flatbuffers/include -isystem /tmp/5909f767-7a62-48fc-a33c-79508cc22d8b/cpp/_deps/googletest-src/googletest/include -isystem /tmp/5909f767-7a62-48fc-a33c-79508cc22d8b/cpp/_deps/googletest-src/googletest -isystem /tmp/5909f767-7a62-48fc-a33c-79508cc22d8b/cpp/_deps/googletest-src/googlemock/include -isystem /tmp/5909f767-7a62-48fc-a33c-79508cc22d8b/cpp/_deps/googletest-src/googlemock -Wredundant-move -Wno-noexcept-type -Wno-self-move -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /var/lib/buildkite-agent/.conda/envs/arrow-commit/include -fdiagnostics-color=always  -Wall -Wno-conversion -Wno-sign-conversion -Wdate-time -Wimplicit-fallthrough -Wunused-result -fno-semantic-interposition -msse4.2  -O3 -DNDEBUG -O2 -ftree-vectorize  -std=c++17 -fPIE -pthread -MD -MT src/arrow/c/CMakeFiles/arrow-bridge-benchmark.dir/bridge_benchmark.cc.o -MF src/arrow/c/CMakeFiles/arrow-bridge-benchmark.dir/bridge_benchmark.cc.o.d -o src/arrow/c/CMakeFiles/arrow-bridge-benchmark.dir/bridge_benchmark.cc.o -c /var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc
In file included from /var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:20:
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:149:1: error: no matching function for call to 'make_unique<benchmark::internal::FunctionBenchmark>(const char [11], <unresolved overloaded function type>)'
  149 | BENCHMARK(ExportType);
      | ^~~~~~~~~
In file included from /var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/memory:78,
                 from /var/lib/buildkite-agent/.conda/envs/arrow-commit/include/benchmark/benchmark.h:177:
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note: candidate: 'template<class _Tp, class ... _Args> std::__detail::__invalid_make_unique_t<_Tp> std::make_unique(_Args&& ...)' (deleted)
 1094 |     make_unique(_Args&&...) = delete;
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note:   template argument deduction/substitution failed:
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h: In substitution of 'template<class _Tp> using std::__detail::__invalid_make_unique_t = typename std::__detail::_MakeUniq::__invalid_type [with _Tp = benchmark::internal::FunctionBenchmark]':
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5:   required by substitution of 'template<class _Tp, class ... _Args> std::__detail::__invalid_make_unique_t<_Tp> std::make_unique(_Args&& ...) [with _Tp = benchmark::internal::FunctionBenchmark; _Args = <missing>]'
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:149:1:   required from here
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1055:11: error: no type named '__invalid_type' in 'struct std::__detail::_MakeUniq<benchmark::internal::FunctionBenchmark>'
 1055 |     using __invalid_make_unique_t = typename _MakeUniq<_Tp>::__invalid_type;
      |           ^~~~~~~~~~~~~~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note: candidate: 'template<class _Tp> std::__detail::__unique_ptr_array_t<_Tp> std::make_unique(size_t)'
 1084 |     make_unique(size_t __num)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note:   candidate expects 1 argument, 2 provided
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note: candidate: 'std::__detail::__unique_ptr_t<_Tp> std::make_unique(_Args&& ...) [with _Tp = benchmark::internal::FunctionBenchmark; _Args = {}; __detail::__unique_ptr_t<_Tp> = __detail::__unique_ptr_t<benchmark::internal::FunctionBenchmark>]'
 1069 |     make_unique(_Args&&... __args)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note:   candidate expects 0 arguments, 2 provided
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:150:1: error: no matching function for call to 'make_unique<benchmark::internal::FunctionBenchmark>(const char [13], <unresolved overloaded function type>)'
  150 | BENCHMARK(ExportSchema);
      | ^~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note: candidate: 'template<class _Tp, class ... _Args> std::__detail::__invalid_make_unique_t<_Tp> std::make_unique(_Args&& ...)' (deleted)
 1094 |     make_unique(_Args&&...) = delete;
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note:   template argument deduction/substitution failed:
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note: candidate: 'template<class _Tp> std::__detail::__unique_ptr_array_t<_Tp> std::make_unique(size_t)'
 1084 |     make_unique(size_t __num)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note:   candidate expects 1 argument, 2 provided
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note: candidate: 'std::__detail::__unique_ptr_t<_Tp> std::make_unique(_Args&& ...) [with _Tp = benchmark::internal::FunctionBenchmark; _Args = {}; __detail::__unique_ptr_t<_Tp> = __detail::__unique_ptr_t<benchmark::internal::FunctionBenchmark>]'
 1069 |     make_unique(_Args&&... __args)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note:   candidate expects 0 arguments, 2 provided
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:151:1: error: no matching function for call to 'make_unique<benchmark::internal::FunctionBenchmark>(const char [12], <unresolved overloaded function type>)'
  151 | BENCHMARK(ExportArray);
      | ^~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note: candidate: 'template<class _Tp, class ... _Args> std::__detail::__invalid_make_unique_t<_Tp> std::make_unique(_Args&& ...)' (deleted)
 1094 |     make_unique(_Args&&...) = delete;
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note:   template argument deduction/substitution failed:
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note: candidate: 'template<class _Tp> std::__detail::__unique_ptr_array_t<_Tp> std::make_unique(size_t)'
 1084 |     make_unique(size_t __num)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note:   candidate expects 1 argument, 2 provided
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note: candidate: 'std::__detail::__unique_ptr_t<_Tp> std::make_unique(_Args&& ...) [with _Tp = benchmark::internal::FunctionBenchmark; _Args = {}; __detail::__unique_ptr_t<_Tp> = __detail::__unique_ptr_t<benchmark::internal::FunctionBenchmark>]'
 1069 |     make_unique(_Args&&... __args)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note:   candidate expects 0 arguments, 2 provided
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:152:1: error: no matching function for call to 'make_unique<benchmark::internal::FunctionBenchmark>(const char [18], <unresolved overloaded function type>)'
  152 | BENCHMARK(ExportRecordBatch);
      | ^~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note: candidate: 'template<class _Tp, class ... _Args> std::__detail::__invalid_make_unique_t<_Tp> std::make_unique(_Args&& ...)' (deleted)
 1094 |     make_unique(_Args&&...) = delete;
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1094:5: note:   template argument deduction/substitution failed:
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note: candidate: 'template<class _Tp> std::__detail::__unique_ptr_array_t<_Tp> std::make_unique(size_t)'
 1084 |     make_unique(size_t __num)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1084:5: note:   candidate expects 1 argument, 2 provided
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note: candidate: 'std::__detail::__unique_ptr_t<_Tp> std::make_unique(_Args&& ...) [with _Tp = benchmark::internal::FunctionBenchmark; _Args = {}; __detail::__unique_ptr_t<_Tp> = __detail::__unique_ptr_t<benchmark::internal::FunctionBenchmark>]'
 1069 |     make_unique(_Args&&... __args)
      |     ^~~~~~~~~~~
/var/lib/buildkite-agent/.conda/envs/arrow-commit/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:1069:5: note:   candidate expects 0 arguments, 2 provided
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:91:13: warning: 'void arrow::ExportRecordBatch(benchmark::State&)' defined but not used [-Wunused-function]
   91 | static void ExportRecordBatch(benchmark::State& state) {  // NOLINT non-const reference
      |             ^~~~~~~~~~~~~~~~~
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:80:13: warning: 'void arrow::ExportArray(benchmark::State&)' defined but not used [-Wunused-function]
   80 | static void ExportArray(benchmark::State& state) {  // NOLINT non-const reference
      |             ^~~~~~~~~~~
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:69:13: warning: 'void arrow::ExportSchema(benchmark::State&)' defined but not used [-Wunused-function]
   69 | static void ExportSchema(benchmark::State& state) {  // NOLINT non-const reference
      |             ^~~~~~~~~~~~
/var/lib/buildkite-agent/builds/amd64-c6a-4xlarge-linux-i-0e8310a07745abccb-1/apache-arrow/arrow-bci-benchmark-on-amd64-c6a-4xlarge-linux/arrow/cpp/src/arrow/c/bridge_benchmark.cc:58:13: warning: 'void arrow::ExportType(benchmark::State&)' defined but not used [-Wunused-function]
   58 | static void ExportType(benchmark::State& state) {  // NOLINT non-const reference
      |             ^~~~~~~~~~
[461/640] Building CXX object src/arrow/dataset/CMakeFiles/arrow_dataset_objlib.dir/scanner.cc.o
ninja: build stopped: subcommand failed.

Component(s)

Benchmarking

@raulcd
Copy link
Member Author

raulcd commented Mar 26, 2025

There was a new release of Google Benchmark yesterday (1.9.2) which is the one used on the run and checks with the problem:
https://github.com/google/benchmark/releases/tag/v1.9.2

@raulcd
Copy link
Member Author

raulcd commented Mar 26, 2025

It does seem related to this change:

But I am unsure on the fix, @pitrou any idea?

@pitrou
Copy link
Member

pitrou commented Mar 26, 2025

I think this might be because the benchmark function names (ExportType, etc.) clash with the public API functions they benchmark, so std::make_unique is not able to resolve a single signature. I'll try a fix.

pitrou added a commit to pitrou/arrow that referenced this issue Mar 26, 2025
Benchmarks fail compiling with Google Benchmark 1.9.2.
This is because the benchmark function names conflict with Arrow API function names.

We don't want to break continuous benchmarking history by changing the benchmark names, so instead move the benchmarks to a dedicated namespace.
pitrou added a commit to pitrou/arrow that referenced this issue Mar 26, 2025
Benchmarks fail compiling with Google Benchmark 1.9.2.
This is because the benchmark function names conflict with Arrow API function names.

We don't want to break continuous benchmarking history by changing the benchmark names, so instead move the benchmarks to a dedicated namespace.
pitrou added a commit to pitrou/arrow that referenced this issue Mar 26, 2025
Benchmarks fail compiling with Google Benchmark 1.9.2.
This is because the benchmark function names conflict with Arrow API function names.

We don't want to break continuous benchmarking history by changing the benchmark names, so instead move the benchmarks to a dedicated namespace.
pitrou added a commit to pitrou/arrow that referenced this issue Mar 26, 2025
Benchmarks fail compiling with Google Benchmark 1.9.2.
This is because the benchmark function names conflict with Arrow API function names.

We don't want to break continuous benchmarking history by changing the benchmark names, so instead move the benchmarks to a dedicated namespace.
pitrou added a commit to pitrou/arrow that referenced this issue Mar 26, 2025
Benchmarks fail compiling with Google Benchmark 1.9.2.
This is because the benchmark function names conflict with Arrow API function names.

We don't want to break continuous benchmarking history by changing the benchmark names, so instead move the benchmarks to a dedicated namespace.
pitrou added a commit that referenced this issue Mar 26, 2025
### Rationale for this change

Benchmarks fail compiling with Google Benchmark 1.9.2. The errors seem triggered by this change upstream: google/benchmark#1948

### What changes are included in this PR?

* Move some benchmark functions to a dedicated namespace so that there is no ambiguity with other functions
* Reduce reliance on templating in benchmark functions to avoid resolution ambiguities
* Other required fixes

Note that we don't rename the benchmark functions, to avoid breaking continuous benchmarking history.

### Are these changes tested?

By continuous benchmarking.

### Are there any user-facing changes?

No.
* GitHub Issue: #45939

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
@pitrou
Copy link
Member

pitrou commented Mar 26, 2025

Issue resolved by pull request 45942
#45942

@pitrou pitrou closed this as completed Mar 26, 2025
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Apr 15, 2025
…45942)

### Rationale for this change

Benchmarks fail compiling with Google Benchmark 1.9.2. The errors seem triggered by this change upstream: google/benchmark#1948

### What changes are included in this PR?

* Move some benchmark functions to a dedicated namespace so that there is no ambiguity with other functions
* Reduce reliance on templating in benchmark functions to avoid resolution ambiguities
* Other required fixes

Note that we don't rename the benchmark functions, to avoid breaking continuous benchmarking history.

### Are these changes tested?

By continuous benchmarking.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#45939

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Apr 15, 2025
…45942)

### Rationale for this change

Benchmarks fail compiling with Google Benchmark 1.9.2. The errors seem triggered by this change upstream: google/benchmark#1948

### What changes are included in this PR?

* Move some benchmark functions to a dedicated namespace so that there is no ambiguity with other functions
* Reduce reliance on templating in benchmark functions to avoid resolution ambiguities
* Other required fixes

Note that we don't rename the benchmark functions, to avoid breaking continuous benchmarking history.

### Are these changes tested?

By continuous benchmarking.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#45939

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants