Add test with bazel #969

Spacetown · 2024-09-19T22:02:16Z

Add test with bezel build system to verify #966.

noxfile.py

tests/bazel/Makefile

Spacetown · 2024-09-19T22:24:21Z

@oliverlee Do you have an idea why in the pipeline following error is showing?

bazel build --force_pic --collect_code_coverage=True //main:testcase
/usr/local/bin/bazel: 1: Syntax error: Bad function name

oliverlee · 2024-09-19T22:30:26Z

@oliverlee Do you have an idea why in the pipeline following error is showing?
bazel build --force_pic --collect_code_coverage=True //main:testcase
/usr/local/bin/bazel: 1: Syntax error: Bad function name

I'm not sure, it builds for me locally:

gcovr/tests/bazel on  add_test_with_bazel via  6GiB/8GiB | 2GiB/4GiB took 1s
127 ✗  bazel build --force_pic --collect_code_coverage=True //main:testcase
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=261
INFO: Reading rc options for 'build' from /Users/oliver/.bazelrc:
  'build' options: --announce_rc
INFO: Analyzed target //main:testcase (87 packages loaded, 441 targets configured).
INFO: Found 1 target...
Target //main:testcase up-to-date:
  bazel-bin/main/testcase
INFO: Elapsed time: 8.462s, Critical Path: 0.28s
INFO: 11 processes: 9 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 11 total actions

oliverlee · 2024-09-19T22:47:24Z

tests/bazel/MODULE.bazel

you may want to set USE_BAZEL_VERSION given use of bazelisk
https://github.com/bazelbuild/bazelisk?tab=readme-ov-file#bazeliskrc-configuration-file

otherwise, bazel version updates may break ci

I had this at the beginning.

oliverlee · 2024-09-20T01:00:24Z

I was able to generate coverage with bazel and clang with this command:

gcovr/tests/bazel on  add_test_with_bazel [!?] via  6GiB/8GiB | 3GiB/4GiB
❯  bazel coverage //main:testcase --instrumentation_filter=//main:testcase --instrument_test_targets=true --config=brew-clang

...

INFO: No lcov file found.
Sep 20, 2024 12:48:24 AM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Sep 20, 2024 12:48:24 AM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Sep 20, 2024 12:48:24 AM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: Found one .profdata file.
================================================================================
INFO: Found 1 test target...
Target //main:testcase up-to-date:
  bazel-bin/main/testcase
INFO: Elapsed time: 1.765s, Critical Path: 0.80s
INFO: 6 processes: 2 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 6 total actions
//main:testcase                                                          PASSED in 0.4s
  /private/var/tmp/_bazel_oliver/a46f8234038dded0774b19face33441e/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/main/testcase/coverage.dat

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

I was not able to generate coverage with bazel and gcc:

gcovr/tests/bazel on  add_test_with_bazel [!?] via  6GiB/8GiB | 3GiB/4GiB took 1s
❯  bazel coverage //main:testcase --instrumentation_filter=//main:testcase --instrument_test_targets=true --config=brew-gcc


...


INFO: No lcov file found.
Sep 20, 2024 12:50:21 AM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Sep 20, 2024 12:50:21 AM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Sep 20, 2024 12:50:21 AM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Sep 20, 2024 12:50:21 AM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
================================================================================
INFO: Found 1 test target...
Target //main:testcase up-to-date:
  bazel-bin/main/testcase
INFO: Elapsed time: 1.942s, Critical Path: 0.74s
INFO: 6 processes: 2 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 6 total actions
//main:testcase                                                          PASSED in 0.4s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

config settings are defined by this bazelrc:

# show log output
test --test_output=all
test --test_env=VERBOSE_COVERAGE=1

# brew clang toolchain
common:brew-clang --action_env=CC=/opt/homebrew/opt/llvm/bin/clang
coverage:brew-clang --copt=-ffile-compilation-dir=.
coverage:brew-clang --test_env=COVERAGE_GCOV_PATH=/opt/homebrew/opt/llvm/bin/llvm-profdata

# brew gcc toolchain
common:brew-gcc --action_env=CC=/opt/homebrew/bin/gcc-14
build:brew-gcc --features=-coverage
build:brew-gcc --features=-macos_default_link_flags
build:brew-gcc --copt=--coverage
build:brew-gcc --linkopt=--coverage
build:brew-gcc --linkopt=-no-canonical-prefixes
coverage:brew-gcc --test_env=COVERAGE_GCOV_PATH=/opt/homebrew/bin/gcov-14
#coverage:brew-gcc --test_env=GCOV_PREFIX_STRIP=9999

# https://github.com/bazelbuild/bazel/issues/14970#issuecomment-1894565761
# https://github.com/bazelbuild/bazel/issues/14970#issuecomment-1894565761
# https://github.com/bazelbuild/bazel/issues/23312#issuecomment-2291751671
# https://github.com/bazelbuild/bazel/blob/7384d10f8ccab0180f93a0adba8aa6cd9b53c57b/tools/cpp/unix_cc_toolchain_config.bzl#L1619
# https://gcc.gnu.org/onlinedocs/gcc/Cross-profiling.html

I think the issue is related to the values of GCOV_PREFIX and GCOV_PREFIX_STRIP, which are set here:
https://github.com/bazelbuild/bazel/blob/7384d10f8ccab0180f93a0adba8aa6cd9b53c57b/tools/test/collect_coverage.sh#L90-L94

as the gcda file is created if I manually invoke the test target:

gcovr/tests/bazel on  add_test_with_bazel [!?] via  6GiB/8GiB | 3GiB/4GiB
❯  [ ! -f main.gcda ] && GCOV_PREFIX=/ GCOV_PREFIX_STRIP=9999 ./bazel-bin/main/testcase && ls main.gcda
main.gcda

I'm not sure if this is macOS specific behavior as I haven't had issues in the past with gcc 12 and x86-64-linux.

Spacetown · 2024-09-22T19:55:06Z

I don't get it why the test is working locally with gcc and clang but on GH runners I get /usr/local/bin/bazel: Syntax error: Bad function name.

oliverlee · 2024-09-22T20:14:31Z

I don't get it why the test is working locally with gcc and clang but on GH runners I get /usr/local/bin/bazel: Syntax error: Bad function name.

I'm not sure. Is this only running on linux-arm64?

gh runners provide bazelisk but I guess you can't or don't want to use that?

Spacetown · 2024-09-22T20:28:42Z

In the Runner itself it‘s working but your right, on GH docker image I need to install another bazelisk than on my Mac.

codacy-production · 2024-09-23T18:56:53Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ 100.00% (target: 95.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`97e51f1`)	4176	4085	97.82%
Head commit (`96a9970`)	4176 (+0)	4085 (+0)	97.82% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#969)	1	1	100.00%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Ignore bazel files (added by symlink)

Spacetown · 2024-09-28T19:39:43Z

@oliverlee Please can you check the example? I was able to add a example for Ubuntu but with clang the coverage command from Bazel will result in an error

Add the .nox directories to the ignore globings.

oliverlee · 2024-09-30T15:46:08Z

I should have some time later today.

Spacetown · 2024-09-30T21:05:02Z

The coverage command of Bazel is only working on Ubuntu. What's always working was running the executable on my own and then collecting the gcov files by gcovr.

Spacetown · 2024-09-30T21:05:59Z

To run the test locally you can use:

nox -rs tests -- -k bazel

oliverlee · 2024-09-30T21:07:48Z

To run the test locally you can use:
nox -rs tests -- -k bazel

~~Can I just install nox from apt? My linux machine is ubuntu 20.04~~

Okay it seems to pass locally, although I'm not sure how I would run this outside of nox

❯ bazel run //tools:nox -- -f$(pwd)/../../noxfile.py -rs tests -- -k bazel
INFO: Reading 'startup' options from /home/oliver/.bazelrc: --host_jvm_args=-Xmx4g, --host_jvm_args=-XX:+UseG1GC, --host_jvm_args=-XX:ConcGCThreads=2, --host_jvm_args=-XX:InitiatingHeapOccupancyPercent=45, --host_jvm_args=-XX:MaxGCPauseMillis=100
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=221
INFO: Reading rc options for 'run' from /home/oliver/.bazelrc:
  Inherited 'build' options: --announce_rc --enable_bzlmod=false
INFO: Analyzed target //tools:nox (1 packages loaded, 1 target configured).
INFO: Found 1 target...
Target //tools:nox up-to-date:
  bazel-bin/tools/nox
INFO: Elapsed time: 0.111s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tools/nox -f/home/oliver/repos/gcovr/tests/bazel/../../noxfile.py -rs tests -- -k bazel
nox > Running session tests
nox > Re-using existing virtual environment at .nox/tests.
nox > python -m pip install jinja2 lxml pygments==2.13.0 pytest pytest-timeout cmake yaxmldiff
nox > python -m pip install -e .
nox > Print tool versions
nox > python --version
Python 3.12.4
nox > /home/linuxbrew/.linuxbrew/bin/gcc-14 --version
gcc-14 (Homebrew GCC 14.2.0) 14.2.0
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nox > /home/linuxbrew/.linuxbrew/bin/g++-14 --version
g++-14 (Homebrew GCC 14.2.0) 14.2.0
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nox > /home/linuxbrew/.linuxbrew/bin/gcov-14 --version
gcov (Homebrew GCC 14.2.0) 14.2.0
JSON format version: 2
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nox > Using reference data for gcc-14
nox > cd tests
nox > make --silent clean
nox > python -m pytest -k bazel -- doc gcovr tests
==================================================================================================== test session starts ====================================================================================================
platform linux -- Python 3.12.4, pytest-8.3.3, pluggy-1.5.0 -- /home/oliver/repos/gcovr/.nox/tests/bin/python
cachedir: .pytest_cache
rootdir: /home/oliver/repos/gcovr
configfile: setup.cfg
plugins: timeout-2.3.1
timeout: 90.0s
timeout method: signal
timeout func_only: False
collected 631 items / 630 deselected / 1 selected

tests/test_gcovr.py::test_build[bazel-json] SKIPPED (Bazel test not working with JSON intermediate format, on Windows or on MacOs.)                                                                                   [100%]

============================================================================================ 1 skipped, 630 deselected in 0.67s =============================================================================================
nox > Session tests was successful.

Spacetown · 2024-10-01T04:53:31Z

Nox is a python module which sets up the environment to run the tests. You can run it without nox but then you need to do the setup yourself.

codacy-production · 2024-10-01T17:41:39Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ 100.00% (target: 95.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`67478e8`)	4184	4093	97.83%
Head commit (`cd3b42a`)	4184 (+0)	4093 (+0)	97.83% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#969)	1	1	100.00%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Add a test case which is using bazel to build the SW and shows how this can be used together with gcovr.

Spacetown added the QA related to testing, build infrastructure, etc label Sep 19, 2024

Spacetown added this to the Upcoming release milestone Sep 19, 2024

Spacetown force-pushed the add_test_with_bazel branch 2 times, most recently from 0725e49 to f83f957 Compare September 19, 2024 22:18

oliverlee reviewed Sep 19, 2024

View reviewed changes

noxfile.py Outdated Show resolved Hide resolved

oliverlee reviewed Sep 19, 2024

View reviewed changes

tests/bazel/Makefile Outdated Show resolved Hide resolved

oliverlee reviewed Sep 19, 2024

View reviewed changes

Spacetown force-pushed the add_test_with_bazel branch from 2094530 to b6c0bb4 Compare September 23, 2024 18:49

Spacetown added 5 commits September 26, 2024 21:24

Install bazel

9d5aa96

Add test with bazel

4d363a1

Ignore bazel files (added by symlink)

Add reference data

a315195

Update documentation

4be816d

Print stderr of gcov in verbose mode

9f5bae5

Spacetown force-pushed the add_test_with_bazel branch from b6c0bb4 to 9f5bae5 Compare September 26, 2024 20:56

Deactivate test for Windows and MacOS

f2c1cb6

gcovr deleted a comment from codacy-production bot Sep 28, 2024

Spacetown marked this pull request as ready for review September 28, 2024 19:36

Spacetown added 3 commits September 30, 2024 07:16

Merge branch 'main' into add_test_with_bazel

e1b266c

Move ignore globing to setup.cfg

9444083

Add the .nox directories to the ignore globings.

Update reference data

ac8c27d

Merge remote-tracking branch 'upstream/main' into add_test_with_bazel

96a9970

Spacetown added 2 commits October 1, 2024 19:16

Add suggested verbose flags

b76174f

Merge remote-tracking branch 'upstream/main' into add_test_with_bazel

6663377

Spacetown added 2 commits October 6, 2024 20:56

Add gcc-14 to bazel test

1765573

Add clang to bazel test

34be080

Spacetown force-pushed the add_test_with_bazel branch from 178c65f to 5b11500 Compare October 6, 2024 20:06

Spacetown marked this pull request as draft October 6, 2024 20:19

Spacetown force-pushed the add_test_with_bazel branch 12 times, most recently from e65af3c to eb678c1 Compare October 7, 2024 17:59

Add again MacOs for test with bazel

cd3b42a

Spacetown force-pushed the add_test_with_bazel branch from eb678c1 to cd3b42a Compare October 7, 2024 18:26

Spacetown marked this pull request as ready for review October 7, 2024 19:03

Spacetown merged commit 95a75aa into gcovr:main Oct 7, 2024
34 checks passed

Spacetown deleted the add_test_with_bazel branch October 7, 2024 19:05

Spacetown removed this from the Upcoming release milestone Oct 8, 2024

D-Walther pushed a commit to D-Walther/gcovr that referenced this pull request Jun 16, 2025

Add test with bazel (gcovr#969)

1959149

Add a test case which is using bazel to build the SW and shows how this can be used together with gcovr.

Add test with bazel #969

Add test with bazel #969

Uh oh!

Conversation

Spacetown commented Sep 19, 2024

Uh oh!

Uh oh!

Uh oh!

Spacetown commented Sep 19, 2024

Uh oh!

oliverlee commented Sep 19, 2024

Uh oh!

oliverlee Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

Spacetown Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

oliverlee commented Sep 20, 2024

Uh oh!

Spacetown commented Sep 22, 2024

Uh oh!

oliverlee commented Sep 22, 2024

Uh oh!

Spacetown commented Sep 22, 2024

Uh oh!

codacy-production bot commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Uh oh!

Spacetown commented Sep 28, 2024

Uh oh!

oliverlee commented Sep 30, 2024

Uh oh!

Spacetown commented Sep 30, 2024

Uh oh!

Spacetown commented Sep 30, 2024

Uh oh!

oliverlee commented Sep 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Spacetown commented Oct 1, 2024

Uh oh!

codacy-production bot commented Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codacy-production bot commented Sep 23, 2024 •

edited

Loading

oliverlee commented Sep 30, 2024 •

edited

Loading

codacy-production bot commented Oct 1, 2024 •

edited

Loading