Don't include pyc files as part of the hermetic toolchain #713

tomgr · 2022-05-25T07:06:49Z

PR Checklist

Please check if your PR fulfills the following requirements:

Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

I'm not sure how to test this.

PR Type

What kind of change does this PR introduce?

Bugfix
Feature (please, look at the "Scope of the project" section in the README.md file)
Code style update (formatting, local variables)
Refactoring (no functional changes, no api changes)
Build related changes
CI related changes
Documentation content changes
Other... Please describe:

What is the current behavior?

Using rules_python with the hermetic toolchain via python_register_toolchains results in cache misses when using bazel with a remote cache. This appears to be because pyc files that are generated on the build host are included as part of one of the globs, but since pyc files don't appear to be generated deterministically. For example, from one run we saw one action that used the hermetic toolchain had the following file as an input:

        {
          "path": "external/python3_9_x86_64-pc-windows-msvc/lib/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "acc87949c12c0a73eace38bed1a4d9e311ff283fe29a7e28bdfb339ff267ed27"
        },

but from another run we saw:

        {
          "path": "external/python3_9_x86_64-pc-windows-msvc/lib/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "afb43608c750e3f7d4dcb0c968aec6d84f678cebc506666325b43c24ce964e5a"
        },

The hashes above are taken from the bazel execution log as per https://bazel.build/docs/remote-execution-caching-debug.
Note that although the above are with the windows toolchain, we saw the same behaviour on all MacOS, Linux, and Windows.

What is the new behavior?

pyc files are excluded from the glob. We're no longer seeing cache misses when using a cache.

Does this PR introduce a breaking change?

Yes
No

pyc files don't appear to be generated deterministically, so including them as part of the :files filegroup means that actions that use the hermetic toolchain do not have deterministic hashes, and so bazel needlessly re-executes them. This can be seen when using bazel together with a remote cache: builds on separate machines will generate pyc files with different hashes, leading to cache misses.

aignas

LGTM, Thank you!

tomgr · 2022-05-31T06:44:58Z

No problem - thanks for merging!

f0rmiga

I first would like to understand why this happens before simply excluding .pyc files. To clarify, we download the python toolchain deterministically, then make the directory tree read-only. So, excluding the .pyc sounds like burying the real problem.

f0rmiga · 2022-05-31T06:51:58Z

@tomgr You said you see this on all platforms, right? The hash you posted is from the Windows toolchain, which we don't make the tree read-only.

f0rmiga · 2022-05-31T07:11:34Z

@tomgr Could you check if 2485990 solves the problem under Windows?

f0rmiga · 2022-05-31T07:19:55Z

Sorry, 43bdb71 should be the one...

tomgr · 2022-05-31T09:06:06Z

I've just tried using 43bdb71 and it still has the same problem. And just to confirm: we saw this on all three platforms, and this patch appears to have completely solved the issue. We've been using it for all of our CI runs for the last week, and our builds now complete entirely from the cache if there are no changes. Previously, whenever we had a fresh VM bazel would rebuild all of the targets that used python.

As an example, with your latest patch, I again saw the hash of _markupbase.cpython-39.pyc change. For example, on the linux builds on one run we had:

        {
          "path": "external/python3_9_aarch64-unknown-linux-gnu/lib/python3.9/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "3e3c8aaadc6a61e09acbe7cacc9082871fc017209b961c777eeb95f3e59a4507"
        },

and on a second run:

        {
          "path": "external/python3_9_aarch64-unknown-linux-gnu/lib/python3.9/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "ae9d198fb89c74012b2613c07057af98fa8d0cd8f70c1001e0c4e039fba07307"
        },

The same was also true on windows with the first run containing:

        {
          "path": "external/python3_9_x86_64-pc-windows-msvc/lib/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "ee128cd267464d5e68f3b1dd1c4a57d422b7490538d517bc1f5d37b4de178ca5"
        },

and the second run:

        {
          "path": "external/python3_9_x86_64-pc-windows-msvc/lib/__pycache__/_markupbase.cpython-39.pyc",
          "digest": "0b57d0d519df845a69ec5d460a8682e4641c7b78dc70ab3737ce23b055e3c98b"
        },

It seems that not all python modules suffer from the problem. Looking at the logs I see the issue with: gettext, markupbase, typing, pathlib, socket, site, zipfile, hashlib and message, but not any others, despite the fact we do use more modules than that. There's no obvious pattern from this.

I presume that the read-only permissions must be being overridden. We do run our Linux builds inside a docker container with the repository cache mounted as a volume. However our Windows and Mac builds run directly on a VM and we still see the problem on there: each different VM creates different pyc files.

We do run bazel using a specific --repository_cache, otherwise I don't think we use any bazel flags that could affect this.

f0rmiga · 2022-06-01T00:05:32Z

@tomgr When you say "first run" and then "second run", is there any bazel clean involved between them, or different machines? Can you ever get into this state from the same machine without cleaning the Bazel cache?

f0rmiga · 2022-06-01T00:13:21Z

It throws me off that somehow read-only is not taking any effects.
Nevertheless, we could be hitting https://docs.python.org/3/reference/import.html#pyc-invalidation. I have to check if the metadata of those files changes between extractions.

tomgr · 2022-06-01T07:44:40Z

@tomgr When you say "first run" and then "second run", is there any bazel clean involved between them, or different machines? Can you ever get into this state from the same machine without cleaning the Bazel cache?

There's no bazel clean, but I was deliberately running them on separate machines to reproduce the issue. I've never noticed it happening from the same machine: it seems that once the pyc files have been modified, they are then stable. Although we run our Linux builds inside Docker containers, we do share the repository cache between the containers. I suspect that removing the shared repository cache would reproduce the same issue.

Is there any extra data that I can gather to help debug why the read-only setting is not working?

mattoberle · 2022-06-06T18:20:27Z

@tomgr any chance your builds are running as root inside the Docker containers?
When running as a user with CAP_DAC_OVERRIDE the read-only mode on the hermetic toolchain directory is ignored.

alexgartner-bc · 2022-07-01T17:26:57Z

@tomgr any chance your builds are running as root inside the Docker containers? When running as a user with CAP_DAC_OVERRIDE the read-only mode on the hermetic toolchain directory is ignored.

Interesting. I'm seeing similar issues but only on our stateless build servers (which are running builds as root) vs our stateful build servers which run builds as uid 1000.

This only started happening after we configured python_register_toolchains. I noticed it because our rules_docker python images started permadiffing:

container-diff diff

$ container-diff diff --type=history --type=file us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:45de6b34dddce7c24935a5d2f3706973750d06a9f3d27ad2f8f23a9894036ba2 us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:ee7655603252efbf1d852f7d065622c6d3017c2bb01d8e2ae516183c5de68b09

-----File-----

These entries have been added to us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:45de6b34dddce7c24935a5d2f3706973750d06a9f3d27ad2f8f23a9894036ba2: None

These entries have been deleted from us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:45de6b34dddce7c24935a5d2f3706973750d06a9f3d27ad2f8f23a9894036ba2: None

These entries have been changed between us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:45de6b34dddce7c24935a5d2f3706973750d06a9f3d27ad2f8f23a9894036ba2 and us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:ee7655603252efbf1d852f7d065622c6d3017c2bb01d8e2ae516183c5de68b09:
FILE                                                                                                                                                                 SIZE1        SIZE2
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/__pycache__/zipfile.cpython-38.pyc                    57.3K        57.3K
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/asyncio/__pycache__/base_events.cpython-38.pyc        49.9K        49.9K
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/__pycache__/pathlib.cpython-38.pyc                    43.3K        43.3K
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/json/__pycache__/decoder.cpython-38.pyc               9.7K         9.7K
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/__pycache__/_markupbase.cpython-38.pyc                7.7K         7.7K
/app/src/python/sense/services/mock_isv/image.binary.runfiles/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/__pycache__/hashlib.cpython-38.pyc                    6.7K         6.7K


-----History-----

Docker history lines found only in us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:45de6b34dddce7c24935a5d2f3706973750d06a9f3d27ad2f8f23a9894036ba2: None

Docker history lines found only in us-west2-docker.pkg.dev/REDACTED/oci/mock_isv@sha256:ee7655603252efbf1d852f7d065622c6d3017c2bb01d8e2ae516183c5de68b09: None

Sounds like an interim solution might be capsh --drop=CAP_DAC_OVERRIDE before the build?

f0rmiga · 2022-07-01T17:31:17Z

A better idea is not to run as root!

alexgartner-bc · 2022-07-01T17:38:51Z

A better ida is not to run as root!

I generally agree but we are currently using google cloud build which does not allow you to run as non-root (afaik). Just fyi for anyone else browsing this thread later.

Update: actually there is a way

phlax · 2022-08-15T10:28:13Z

i use a temporary container locally for working on a project using rules_python which currently uses root inside

i get this is not "best-practice" but im well aware of the risks and in this case there arent really any

it seems a bit arbitrary to enforce that you cant use rules_python at all as root

Cartman75 · 2022-09-09T02:29:08Z

BTw, its seems impossible to run this in gitlab because it forces root. Did someone figure out how to get around that?

I even tried:

      sudo -H -i -u citests bash -e -x << EOS
      whoami
      cd \$CI_BUILDS_DIR
      ./bazelisk --output_base bazel_cache test //...
      EOS

and claimed it was it running as root and it was a no-go.

keith · 2022-09-23T17:42:06Z

FYI there is a setting to opt out of this root warning, https://github.com/bazelbuild/rules_python//commit/e67e7dd719d34d5a13c15b24d0234c1ac753b52d

ph03 · 2024-01-12T12:44:30Z

@ph03 were you able to find any solution to run this as non-root user via gitlab?

I don't recall the details anymore, but we're happily running gitlab CI without root user by using an image that setup a non-root user like this in the related Dockerfile

# Run as developer
RUN \
    useradd --create-home --groups sudo,docker developer && \
    echo "developer:developer" | sudo chpasswd
USER developer

debkanchan · 2024-01-17T16:32:25Z

This is causing GCP cloud build to fail

See bazel-contrib/rules_python#713 and bazel-contrib/rules_python#907.

layus · 2024-06-25T21:36:39Z

Did anyone ever consider and/or test chattr +i to make files and folder immutable, even for root ?

This is a requirement for one dependency. Otherwise, we get the error: "The current user is root, please run as non-root when using the hermetic Python interpreter. See bazel-contrib/rules_python#713."

loeffel-io · 2024-11-19T11:21:08Z

Hey, for clarification

I do need to add this

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = False,
    ignore_root_user_error = True,
    python_version = "3.11",
)

while one of my deps is using rules_python for my builds since i do not "use" rules_python?

it works btw but this does not feel good

Thanks

Our CI fails to build Bazel, producing this error: "The current user is root, please run as non-root when using the hermetic Python interpreter. See bazel-contrib/rules_python#713." The workaround was suggested at bazel-contrib/rules_python#1169 (comment)

Our CI fails to build Bazel, producing this error: "The current user is root, please run as non-root when using the hermetic Python interpreter. See bazel-contrib/rules_python#713." The workaround was suggested at bazel-contrib/rules_python#1169 (comment) Closes #24549. PiperOrigin-RevId: 702362811 Change-Id: Ib6d4da1e09fd2b7dfd68af02bbb0eb997e8f427c

Our CI fails to build Bazel, producing this error: "The current user is root, please run as non-root when using the hermetic Python interpreter. See bazel-contrib/rules_python#713." The workaround was suggested at bazel-contrib/rules_python#1169 (comment) Closes #24549. PiperOrigin-RevId: 702362811 Change-Id: Ib6d4da1e09fd2b7dfd68af02bbb0eb997e8f427c Co-authored-by: Florian Weikert <[email protected]>

Currently, by default, rules_python immediately fails when Bazel is run as root. The reasoning behind this involves .pyc files being generated for hermetic toolchains when they're first used, causing cache misses; to work around this, rules_python opts to make the toolchain installation directory read-only, but running Bazel as root would circumvent this. So rules_python actively detects if the current user is root, and hard fails. This check can be disabled by the root module by setting `python.override(ignore_root_user_error=True)`. (See more context in the linked issues/PRs.) This causes a reverberating effect across the Bazel ecosystem, as rules_python is essentially a dependency of every single Bazel project through protobuf. Effectively, any Bazel project wishing to run as root need to add the override tag above, even if they don't have anything to do with Python at all. This PR changes the default value of the `ignore_root_user_error` to True instead. Besides, it now unconditionally tries to make the toolchain installation directory read-only, and only outputs a warning if it's detected that the current user is root. See previous discussions at #713, #749, #907, #1008, #1169, etc. Fixes #1169.

Currently, by default, rules_python immediately fails when Bazel is run as root. The reasoning behind this involves .pyc files being generated for hermetic toolchains when they're first used, causing cache misses; to work around this, rules_python opts to make the toolchain installation directory read-only, but running Bazel as root would circumvent this. So rules_python actively detects if the current user is root, and hard fails. This check can be disabled by the root module by setting `python.override(ignore_root_user_error=True)`. (See more context in the linked issues/PRs.) This causes a reverberating effect across the Bazel ecosystem, as rules_python is essentially a dependency of every single Bazel project through protobuf. Effectively, any Bazel project wishing to run as root need to add the override tag above, even if they don't have anything to do with Python at all. This PR changes the default value of the `ignore_root_user_error` to True instead. Besides, it now unconditionally tries to make the toolchain installation directory read-only, and only outputs a warning if it's detected that the current user is root. See previous discussions at #713, #749, #907, #1008, #1169, etc. Fixes #1169. --------- Co-authored-by: Richard Levasseur <[email protected]>

…-contrib#2636) Currently, by default, rules_python immediately fails when Bazel is run as root. The reasoning behind this involves .pyc files being generated for hermetic toolchains when they're first used, causing cache misses; to work around this, rules_python opts to make the toolchain installation directory read-only, but running Bazel as root would circumvent this. So rules_python actively detects if the current user is root, and hard fails. This check can be disabled by the root module by setting `python.override(ignore_root_user_error=True)`. (See more context in the linked issues/PRs.) This causes a reverberating effect across the Bazel ecosystem, as rules_python is essentially a dependency of every single Bazel project through protobuf. Effectively, any Bazel project wishing to run as root need to add the override tag above, even if they don't have anything to do with Python at all. This PR changes the default value of the `ignore_root_user_error` to True instead. Besides, it now unconditionally tries to make the toolchain installation directory read-only, and only outputs a warning if it's detected that the current user is root. See previous discussions at bazel-contrib#713, bazel-contrib#749, bazel-contrib#907, bazel-contrib#1008, bazel-contrib#1169, etc. Fixes bazel-contrib#1169. --------- Co-authored-by: Richard Levasseur <[email protected]>

This leads to errors, see bazel-contrib/rules_python#713 (comment)

tomgr requested review from brandjon and lberki as code owners May 25, 2022 07:06

aignas approved these changes May 30, 2022

View reviewed changes

Merge branch 'main' into improve-remote-caching

e324945

thundergolfer requested review from f0rmiga and removed request for brandjon and lberki May 31, 2022 05:52

thundergolfer approved these changes May 31, 2022

View reviewed changes

f0rmiga suggested changes May 31, 2022

View reviewed changes

thundergolfer added the need: discussion On the agenda for next team meeting label Jun 21, 2022

f0rmiga closed this Jul 1, 2022

This was referenced Jul 12, 2022

Fix remote cache miss #748

Closed

fix: fail if the user is root #749

Merged

keith mentioned this pull request Sep 12, 2022

Unable to build/test as root (was: bazel build -c opt envoy not working on mian branch) envoyproxy/envoy#23065

Closed

phst added a commit to phst/rules_elisp that referenced this pull request Jan 18, 2024

Attempt to work around nondeterminism on Windows.

4c97dd8

See bazel-contrib/rules_python#713 and bazel-contrib/rules_python#907.

phst mentioned this pull request Jan 18, 2024

Attempt to work around nondeterminism on Windows. phst/rules_elisp#350

Merged

phst added a commit to phst/rules_elisp that referenced this pull request Jan 18, 2024

Attempt to work around nondeterminism on Windows.

69a9350

See bazel-contrib/rules_python#713 and bazel-contrib/rules_python#907.

phst added a commit to phst/rules_elisp that referenced this pull request Jan 18, 2024

Attempt to work around nondeterminism on Windows.

af7e4d5

See bazel-contrib/rules_python#713 and bazel-contrib/rules_python#907.

phst added a commit to phst/rules_elisp that referenced this pull request Jan 19, 2024

Attempt to work around nondeterminism on Windows.

eb60756

See bazel-contrib/rules_python#713 and bazel-contrib/rules_python#907.

dneto0 mentioned this pull request Apr 25, 2024

Use bzlmod google/effcee#64

Merged

surenderky mentioned this pull request May 31, 2024

Boringssl and rules python fix for s390x envoyproxy/envoy-openssl#166

Merged

fweikert mentioned this pull request Dec 3, 2024

Enable ignore_root_user_error for Python toolchain bazelbuild/bazel#24549

Closed

keertk mentioned this pull request Dec 4, 2024

[8.0.0] Enable ignore_root_user_error for Python toolchain bazelbuild/bazel#24567

Merged

deadlywing mentioned this pull request Dec 31, 2024

generate clangd index secretflow/yacl#456

Closed

grefab mentioned this pull request Feb 23, 2025

Bazel: Migrate to bzlmod emscripten-core/emsdk#1530

Merged

Wyverald mentioned this pull request Feb 28, 2025

fix: Downgrade "running as root" error to a warning by default #2636

Merged

cburchert added a commit to cburchert/emsdk that referenced this pull request Mar 5, 2025

Don't run Linux tests as root in the Docker container.

c8e64d5

This leads to errors, see bazel-contrib/rules_python#713 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't include pyc files as part of the hermetic toolchain #713

Don't include pyc files as part of the hermetic toolchain #713

tomgr commented May 25, 2022

aignas left a comment

tomgr commented May 31, 2022

f0rmiga left a comment

f0rmiga commented May 31, 2022

f0rmiga commented May 31, 2022

f0rmiga commented May 31, 2022

tomgr commented May 31, 2022

f0rmiga commented Jun 1, 2022

f0rmiga commented Jun 1, 2022 •

edited

Loading

tomgr commented Jun 1, 2022

mattoberle commented Jun 6, 2022

alexgartner-bc commented Jul 1, 2022

f0rmiga commented Jul 1, 2022 •

edited

Loading

alexgartner-bc commented Jul 1, 2022 •

edited

Loading

phlax commented Aug 15, 2022

Cartman75 commented Sep 9, 2022

keith commented Sep 23, 2022

ph03 commented Jan 12, 2024

debkanchan commented Jan 17, 2024

layus commented Jun 25, 2024

loeffel-io commented Nov 19, 2024

Don't include pyc files as part of the hermetic toolchain #713

Don't include pyc files as part of the hermetic toolchain #713

Conversation

tomgr commented May 25, 2022

PR Checklist

PR Type

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

aignas left a comment

Choose a reason for hiding this comment

tomgr commented May 31, 2022

f0rmiga left a comment

Choose a reason for hiding this comment

f0rmiga commented May 31, 2022

f0rmiga commented May 31, 2022

f0rmiga commented May 31, 2022

tomgr commented May 31, 2022

f0rmiga commented Jun 1, 2022

f0rmiga commented Jun 1, 2022 • edited Loading

tomgr commented Jun 1, 2022

mattoberle commented Jun 6, 2022

alexgartner-bc commented Jul 1, 2022

f0rmiga commented Jul 1, 2022 • edited Loading

alexgartner-bc commented Jul 1, 2022 • edited Loading

phlax commented Aug 15, 2022

Cartman75 commented Sep 9, 2022

keith commented Sep 23, 2022

ph03 commented Jan 12, 2024

debkanchan commented Jan 17, 2024

layus commented Jun 25, 2024

loeffel-io commented Nov 19, 2024

f0rmiga commented Jun 1, 2022 •

edited

Loading

f0rmiga commented Jul 1, 2022 •

edited

Loading

alexgartner-bc commented Jul 1, 2022 •

edited

Loading