bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

fmeum · 2024-12-16T14:57:27Z

Without this change, language runtimes still result in a build of //:protoc even with a prebuilt proto_toolchain registered or --proto_compiler set to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build of java_proto_library targets without a C++ toolchain assuming a prebuilt protoc.

Work towards #19558

Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc.

fmeum · 2024-12-16T15:25:35Z

CC @comius @alexeagle

fmeum · 2024-12-16T15:26:09Z

@shaod2 Would you be available to review this?

mkruskal-google · 2024-12-19T19:21:03Z

Hey Fabian,

So IIUC, this PR is trying to make the protoc binary used by proto_library customizable. While that might be a possible long-term solution, it opens the door to a lot of potential misuse. Notably, mixing alternate implementations or different versions of protoc is typically unsupported and can have unexpected results. We have poison pills to avoid some of these situations outside of Bazel, but they haven't been added to all our languages yet. A more conservative change would be to just have some kind of flag that toggles between build-from-source and the prebuilt releases we publish to github, without allowing arbitrary binaries to be injected.

Additionally, this PR is touching protobuf.bzl, which just contains our legacy internal macros/rules (which we want to delete). These have been replaced by the rules in //bazel, which were recently moved from other Bazel repos.

fmeum · 2024-12-30T09:34:46Z

Additionally, this PR is touching protobuf.bzl, which just contains our legacy internal macros/rules (which we want to delete). These have been replaced by the rules in //bazel, which were recently moved from other Bazel repos.

These macros are still used to bootstrap the Java well-known protos for the default Java proto_lang_toolchain, which is why protoc ends up being compiled from source even when using a precompiled protoc binary.

So IIUC, this PR is trying to make the protoc binary used by proto_library customizable. While that might be a possible long-term solution, it opens the door to a lot of potential misuse. Notably, mixing alternate implementations or different versions of protoc is typically unsupported and can have unexpected results.

Isn't the protoc binary used by proto_library already configurable via --proto_compiler or toolchain registration (the latter with --incompatible_enable_proto_toolchain_resolution)? The aim of this PR is specifically to ensure that if users use custom protoc binary, it's the only protoc used in the entire build. Right now, when configuring a non-default protoc binary in any of the existing ways, the //:protoc from this repo still ends up being used to bootstrap protos.

@crt-31

Keeps the `README` guidance in sync with what we're actually using in `WORKSPACE` for consistency's sake. @crt-31 and I found that the Windows build failure for bazel-contrib#1710 mentioned in the earlier commit is related to the Windows/MSVC file path length limit. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. - protocolbuffers/protobuf#12947 Furthermore, the Protobuf team currently plans to just drop MSVC support: - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 I plan to experiment again with "Protobuf Toolchainization", which I'd tried in October when beginning the Bzlmod experiment. Here are some interesting background resources before I dig in on that: - bazelbuild/rules_proto#213 - bazelbuild/rules_proto#179 - https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 - https://github.com/aspect-build/toolchains_protoc/ - protocolbuffers/protobuf#20182 - protocolbuffers/protobuf#19679 - protocolbuffers/protobuf#19558

Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazel-contrib#1482 and bazel-contrib#1652. Stops `protoc` recompilation, and fixes the build breakage in bazel-contrib#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazel-contrib#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0

This is an experiment to see if applying protocolbuffers/protobuf#19679 via a patch and registering `@rules_scala_toolchains//protoc/...:all` will fix Windows MSVC builds. Local experiments suggest that this does indeed enable the toolchainized `protoc` to take precedence and avoid compiling protobuf.

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - Scalafmt: 3.9.2 => 3.9.3 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.10.0 - `protobuf`: 21.7 => 30.0 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains(//protoc:all)` toolchains is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Bumps Scalafmt to 3.9.3 out of convenience. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - Scalafmt: 3.9.2 => 3.9.3 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.10.0 - `protobuf`: 21.7 => 30.0 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains(//protoc:all)` toolchains is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Bumps Scalafmt to 3.9.3 out of convenience. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

comius · 2025-03-19T13:09:34Z

If I understand correctly protobuf team would like to eventually remove --protocol_compiler flag and also isn't completely decided to go in the direction of --incompatible_enable_toolchain_resolution. That's fine.

The PR is in my opinion an improvement, because it improves the consistency of protobuf code (protoc is always obtained in the same way), even if only of the legacy and to be removed code. @mkruskal would you mind accepting it as such? The (Google's) standard for code reviews is that the PR is an improvement of the code.

It doesn't get us much closer to precompiled protoc binaries, but it also doesn't hurt us, because the legacy code may still be removed as well as --protocol_compiler and --incompatible_enable_toolchain_resolution flags.

…mes (#19679) Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build of `java_proto_library` targets without a C++ toolchain assuming a prebuilt protoc. Work towards #19558 Closes #19679 COPYBARA_INTEGRATE_REVIEW=#19679 from fmeum:19558-dont-hardcode-protoc 3bd939d FUTURE_COPYBARA_INTEGRATE_REVIEW=#19679 from fmeum:19558-dont-hardcode-protoc 3bd939d PiperOrigin-RevId: 739382145

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: bazel-contrib#1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

mkruskal-google · 2025-04-30T23:24:23Z

O yea msys2 is definitely the problem there. You're getting extra characters from tools/msys64/home, which is going to push you over the limit for sure. We're very close to the limit and I think C:\tmp is the extent of what Bazel+MSVC can handle today (it's what our CI covers).

Resolves the Windows symlink breakage from bazelbuild/bazel-central-registry#5490 by replacing examples/crossbuild/protobuf.patch with a full copy. Bumps rules_python to 1.5.2 for both Bzlmod and legacy WORKSPACE builds. --- This is hopefully a temporary measure to unblock the next `rules_scala` release. This will probably require creating a v7.1.1 tag to trigger a new .bcr/presubmit.yml workflow. See bazelbuild/bazel-central-registry#5506 for details about the breakage. If and when bazelbuild/continuous-integration#2350 lands and makes its way to the BCR presubmit build, we could restore the symlink. Better yet, if protocolbuffers/protobuf#19679 ever lands, we could remove this symlink and all the others. Bumped `rules_python` to 1.5.2 for legacy `WORKSPACE` builds because it contains bazel-contrib/rules_python#3134, which fixes bazel-contrib/rules_python#3119.

github-actions · 2025-09-12T10:07:07Z

We triage inactive PRs and issues in order to make it easier to find active work. If this PR should remain active, please add a comment.

This PR is labeled inactive because the last activity was over 90 days ago. This PR will be closed and archived after 14 additional days without activity.

alexeagle · 2025-09-12T18:10:01Z

@mkruskal-google I think we've confirmed that the toolchain flag isn't being removed in the near future, could we consider merging this one? Will be needed to get pre-built protoc working as expected.

Updates the following dependencies: - Go: 1.25.0 => 1.25.1 - `golang.org/x/tools`: 0.36.0 => 0.37.0 - `google-common-protos`: 2.61.0 => 2.61.1 - `jline`: 3.30.5 => 3.30.6 - `protobuf`: 32.0 => 32.1 - `protobuf-java`: 4.32.0 => 4.32.1 - `rules_cc`: 0.2.3 => 0.2.8 - `rules_python`: 1.6.0 => 1.6.1 - `rules_shell`: 0.6.0 => 0.6.1 - `util-interface`: 1.11.5 => 1.11.6 Also changes all the `bazel_dep` declarations for `protobuf` to remove the curently unnecessary `version` parameter from: ```py bazel_dep( name = "protobuf", version = "32.0", repo_name = "com_google_protobuf", ) ``` to: ```py bazel_dep(name = "protobuf", repo_name = "com_google_protobuf") ``` If protocolbuffers/protobuf#19679 ever lands, we'll remove the `single_version_override` statements and restore `version` to these `bazel_dep` declarations. --- This is mostly a routine version bump before a new release. The changes to the `bazel_dep` statements for `protobuf` aim to make the diffs smaller when increasing the `protobuf` version in the future.

Updates the following dependencies: - Go: 1.25.0 => 1.25.1 - `bazel-contrib/.github:release_ruleset.yaml`: 7.2.2 => 7.2.3 - `bazel-contrib/publish-to-bcr:publish.yaml`: 0.1.0 => 1.0.0 - `golang.org/x/tools`: 0.36.0 => 0.37.0 - `google-common-protos`: 2.61.0 => 2.61.1 - `jline`: 3.30.5 => 3.30.6 - `protobuf-java`: 4.32.0 => 4.32.1 - `protobuf`: 32.0 => 32.1 - `rules_cc`: 0.2.3 => 0.2.8 - `rules_python`: 1.6.0 => 1.6.1 - `rules_shell`: 0.6.0 => 0.6.1 - `util-interface`: 1.11.5 => 1.11.6 Also changes all the `bazel_dep` declarations for `protobuf` to remove the curently unnecessary `version` parameter from: ```py bazel_dep( name = "protobuf", version = "32.0", repo_name = "com_google_protobuf", ) ``` to: ```py bazel_dep(name = "protobuf", repo_name = "com_google_protobuf") ``` If protocolbuffers/protobuf#19679 ever lands, we'll remove the `single_version_override` statements and restore `version` to these `bazel_dep` declarations. --- This is mostly a routine version bump before a new release. The changes to the `bazel_dep` statements for `protobuf` aim to make the diffs smaller when increasing the `protobuf` version in the future.

Updates the following dependencies: - Go: 1.25.0 => 1.25.1 - `bazel-contrib/.github:release_ruleset.yaml`: 7.2.2 => 7.2.3 - `bazel-contrib/publish-to-bcr:publish.yaml`: 0.1.0 => 1.0.0 - `golang.org/x/tools`: 0.36.0 => 0.37.0 - `google-common-protos`: 2.61.0 => 2.61.1 - `jline`: 3.30.5 => 3.30.6 - `protobuf-java`: 4.32.0 => 4.32.1 - `protobuf`: 32.0 => 32.1 - `rules_cc`: 0.2.3 => 0.2.8 - `rules_python`: 1.6.0 => 1.6.1 - `rules_shell`: 0.6.0 => 0.6.1 - `util-interface`: 1.11.5 => 1.11.6 Also changes all the `bazel_dep` declarations for `protobuf` to remove the curently unnecessary `version` parameter from: ```py bazel_dep( name = "protobuf", version = "32.0", repo_name = "com_google_protobuf", ) ``` to: ```py bazel_dep(name = "protobuf", repo_name = "com_google_protobuf") ``` If protocolbuffers/protobuf#19679 ever lands, we'll remove the `bazel_dep` declarations altogether and rely on `deps/latest/MODULE.bazel`. --- This is mostly a routine version bump before a new release. The changes to the `bazel_dep` statements for `protobuf` aim to make the diffs smaller when increasing the `protobuf` version in the future.

vibbix · 2025-10-10T17:16:57Z

Whats the current status on this patch?

Updates the following Bazel dependency versions: - `.bazelversion`: 7.6.1 => 7.6.2 - Go: 1.25.1 => 1.25.3 - `bazel_skylib`: 1.8.1 => 1.8.2 - `golang.org/x/tools`: 0.37.0 => 0.38.0 - `protobuf`: v32.1 => v33.0 - Still missing protocolbuffers/protobuf#19679, so we keep patching. - `rules_cc`: 0.2.8 => 0.2.10 - `rules_go`: 0.57.0 => bazel-contrib/rules_go@74199c92 - This is a temporary workaround for bazel-contrib/rules_go#4480. - `rules_java`: 8.15.2 => 8.16.1 - Updates all legacy `WORKSPACE` files to invoke `bazel_features_deps` _before_ `rules_java_dependencies`, required since `rules_java` 8.16.0. - Adds `bazel_features` 1.37.0 to `rules_scala_dependencies` from `latest_deps.bzl` as an explicit dependency to enable the dependency setup macro reordering. - `rules_python`: 1.6.1 => 1.6.3 Updates the following Maven artifact versions, with `scripts/create_repository.py` having generated the changes to `third_party/repositories/scala_*.bzl`: - `com.google.api.grpc:proto-google-common-protos`: 2.61.1 => 2.62.0 - `com.google.guava:guava`: 33.4.8-jre => 33.5.0-jre - `com.google.protobuf:protobuf-java`: 4.32.1 => 4.33.0 - `com.thesamet.scalapb:scalapb-runtime_*`: 1.0.0-alpha.2 => 1.0.0-alpha.3 - `io.grpc:grpc-api`: 1.75.0 => 1.76.0 - `org.scala-lang.modules:scala-parser-combinators_*`: 1.1.2 => 2.4.0 - Adds logic to `scripts/create_repository.py` to keep the Scala 2.11 version of `scala-parser-combinators` at version 1.1.2 - `org.scala-sbt:compiler-interface`: 1.10.1 => 1.11.0 - `org.scala-sbt:util-interface`: 1.11.6 => 1.11.7 - `org.scalameta:scalafmt-*`: 3.9.9 => 3.10.1 - Updates `version` in all the `**.scalafmt*.conf` files to match. - `org.typelevel:kind-projector_*`: 0.13.3 => 0.13.4 --- This is in preparation for releasing v7.1.3, which will also contain the Scala 3.3.7 update from bazel-contrib#1777.

Updates the following Bazel dependency versions: - `.bazelversion`: 7.6.1 => 7.6.2 - Go: 1.25.1 => 1.25.3 - `bazel_skylib`: 1.8.1 => 1.8.2 - `golang.org/x/tools`: 0.37.0 => 0.38.0 - `protobuf`: v32.1 => v33.0 - Still missing protocolbuffers/protobuf#19679, so we keep patching. - `rules_cc`: 0.2.8 => 0.2.10 - `rules_go`: 0.57.0 => bazel-contrib/rules_go@74199c92 - This is a temporary workaround for bazel-contrib/rules_go#4480. - `rules_java`: 8.15.2 => 8.16.1 - Updates all legacy `WORKSPACE` files to invoke `bazel_features_deps` _before_ `rules_java_dependencies`, required since `rules_java` 8.16.0. - Adds `bazel_features` 1.37.0 to `rules_scala_dependencies` from `latest_deps.bzl` as an explicit dependency to enable the dependency setup macro reordering. - `rules_python`: 1.6.1 => 1.6.3 Updates the following Maven artifact versions, with `scripts/create_repository.py` having generated the changes to `third_party/repositories/scala_*.bzl`: - `com.google.api.grpc:proto-google-common-protos`: 2.61.1 => 2.62.0 - `com.google.guava:guava`: 33.4.8-jre => 33.5.0-jre - `com.google.protobuf:protobuf-java`: 4.32.1 => 4.33.0 - `com.thesamet.scalapb:scalapb-runtime_*`: 1.0.0-alpha.2 => 1.0.0-alpha.3 - `io.grpc:grpc-api`: 1.75.0 => 1.76.0 - `org.scala-lang.modules:scala-parser-combinators_*`: 1.1.2 => 2.4.0 - Adds logic to `scripts/create_repository.py` to keep the Scala 2.11 version of `scala-parser-combinators` at version 1.1.2 - `org.scala-sbt:compiler-interface`: 1.10.1 => 1.11.0 - `org.scala-sbt:util-interface`: 1.11.6 => 1.11.7 - `org.scalameta:scalafmt-*`: 3.9.9 => 3.10.1 - Updates `version` in all the `**.scalafmt*.conf` files to match. - `org.typelevel:kind-projector_*`: 0.13.3 => 0.13.4 --- This is in preparation for releasing v7.1.3, which will also contain the Scala 3.3.7 update from #1777.

…mes (#19679) Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build of `java_proto_library` targets without a C++ toolchain assuming a prebuilt protoc. Work towards #19558 Closes #19679 COPYBARA_INTEGRATE_REVIEW=#19679 from fmeum:19558-dont-hardcode-protoc 2d67e82 PiperOrigin-RevId: 828014596

Replaces the `rules_scala` prebuilt `protoc` toolchain everywhere except in a few `test_dependency_versions.sh` test cases. Updates `README.md` to recommend using the `protobuf` prebuilt `protoc` toolchain, and refactors and adds test cases to `test_dependency_versions.sh`. Note that legacy `WORKSPACE` builds continue to use the `rules_scala` prebuilt `protoc` toolchain, since the `protobuf` implementation only supports Bzlmod. After removing `protobuf.patch` files, updating `MODULE.bazel{,.lock}` files, and updating Maven dependencies, the most significant changes for supporting `protobuf` v33.4 are: - `protoc/private/protoc_toolchains.bzl`, `scala/toolchains_repo.bzl`: Moves `proto_lang_toolchain` from `@rules_scala_protoc_toolchains` to `@rules_scala_toolchains//scala_proto`. This can break users using `rules_proto` < 7.0.0 and `protobuf` < v28.2, but `README.md` contains a patch for such users. - `protoc/private/toolchain_impl.bzl`: Applies changes from the final state of protocolbuffers/protobuf#19679 and updates the prebuilt `protoc` toolchain resolution check and error message. - `test_dependency_versions.sh`, `deps/test/MODULE.bazel.template`, `deps/test/protoc_toolchains.template`: Adds Bazel 9 and `protobuf` prebuilt `protoc` toolchain test cases. Updates `do_build_and_test` to support both toolchains. Extracts the `protoc_toolchains.template` to include it only in test cases using the `rules_scala` toolchain. Other key components of this change include: - `protoc/BUILD`: Aliases `@protobuf//bazel/toolchains:prefer_prebuilt_protoc` as `@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelrc`, `tools/bazel.rc.buildkite`: Adds `common --@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelci/presubmit.yml`, `.bcr/presubmit.yml`: Adds `9.x` to the `bcr_bazel` matrix . Also applies whitespace and quoting format changes to `.bazelci/presubmit.yml`. Also contains these small, but important changes to fix continuous integration breakages and `last_green` Bazel compatibility: - `.github/workflows/release_prep.sh`: Excludes `.bazelversion` symlinks, since per bazelbuild/bazel-worker-api#21, these break Bazel as of bazelbuild/bazel@f942a70. - `MODULE.bazel`, `scala/private/macros/workspace_compat.bzl`: Bumps `bazelbuild/bazel-worker-api` to v0.0.9 (though we need v0.0.10 to fix `last_green` builds per bazelbuild/bazel-worker-api#22). - `test/shell/test_runner.sh`, `test/shell/test_bzlmod_macros.sh`: Replaces `relative_path_to_parent` with `convert_msys2_path` for bazelbuild/continuous-integration#2445, bazelbuild/bazel#28406. Other minor changes include: - `MODULE.bazel`: Removes the `version` parameter from the `module` declaration. The Publish to BCR GitHub Actions workflow patches this parameter anyway. See: - https://bazel.build/external/faq#module-versioning-best-practices - https://github.com/bazel-contrib/publish-to-bcr/blob/v1.1.0/src/domain/create-entry.ts#L176-L216 - https://bazelbuild.slack.com/archives/C014RARENH0/p1767892286188219 - Replaces `precompiled` with `prebuilt` in every reference to the prebuilt `protoc` toolchain. - `scala/toolchains_repo.bzl`: Replaces `@@{rules_scala_repo}` references with plain `@rules_scala` references, allowing repo mapping to do its thing. - `test_dependency_version.sh`: Skips non-prebuilt `protoc` prebuilt toolchain test cases from for macOS on continuous integration. This test suite was taking a disproportionately long time to run before adding the new test cases, which Linux builds will continue to cover. A summary of these changes excluding `README.md`, `protobuf.patch`, `MODULE.bazel{,.lock}` files, and Maven dependency updates: ```txt $ git diff --stat HEAD^ ':!README.md' ':!**MODULE.bazel*' \ ':!**protobuf.patch' ':!third_party/repositories/scala_*' .bazelci/presubmit.yml | 66 ++++++++++++++++++++... .bazelrc | 9 +++++-- .bcr/presubmit.yml | 2 +- .github/workflows/release_prep.sh | 7 +++++- WORKSPACE | 15 ++++++----- deps/test/protoc_toolchains.template | 12 +++++++++ protoc/BUILD | 8 ++++++ protoc/private/protoc_integrity.bzl | 10 ++++++++ protoc/private/protoc_toolchains.bzl | 19 +++----------- protoc/private/toolchain_impl.bzl | 30 ++++++++++++++-------- protoc/toolchains.bzl | 2 +- scala/extensions/protoc.bzl | 8 +++--- scala/latest_deps.bzl | 8 +++--- scala/private/macros/workspace_compat.bzl | 6 ++--- scala/toolchains_repo.bzl | 37 ++++++++++++++++----... scala_proto/scala_proto_toolchain.bzl | 2 ++ scripts/create_repository.py | 2 +- scripts/update_protoc_integrity.py | 6 +++-- test/shell/test_bzlmod_macros.sh | 3 ++- test/shell/test_runner.sh | 28 ++++++++++----------- test_dependency_versions.sh | 151 ++++++++++++++++++++... test_version.sh | 2 -- tools/bazel.rc.buildkite | 9 +++++-- 23 files changed, 288 insertions(+), 154 deletions(-) ``` --- The following Bazel Slack thread pointed out the apparent incompatibility with the new Bazel 9.0.0 release: - https://bazelbuild.slack.com/archives/CDCKJ2KFZ/p1768383285232129 - https://github.com/bazelbuild/bazel/releases/tag/9.0.0 This happened because Bazel 9 now sets `--incompatible_enable_proto_toolchain_resolution` by default: - https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution Since the `scala_proto` implementation depends on this flag, this broke projects that didn't configure the `rules_scala` prebuilt `protoc` toolchain. This change fixes that problem while ensuring seamless compatibility with the `protobuf` v33.4 prebuilt protoc toolchain. We will likely release v7.2.0 as soon as this change lands. After that, we can bump other dependency versions in a patch level release, including `bazel-worker-api` as soon as v0.0.10 appears.

@WojciechMazur

Replaces the `rules_scala` prebuilt `protoc` toolchain everywhere except in a few `test_dependency_versions.sh` test cases. Updates `README.md` to recommend using the `protobuf` prebuilt `protoc` toolchain, and refactors and adds test cases to `test_dependency_versions.sh`. Note that legacy `WORKSPACE` builds continue to use the `rules_scala` prebuilt `protoc` toolchain, since the `protobuf` implementation only supports Bzlmod. After removing `protobuf.patch` files, updating `MODULE.bazel{,.lock}` files, and updating Maven dependencies, the most significant changes for supporting `protobuf` v33.4 are: - `protoc/BUILD`, `protoc/private/protoc_toolchains.bzl`: Moves `proto_lang_toolchain` from `@rules_scala_protoc_toolchains` to `//protoc:scala_protoc_toolchain`. This can break users using `rules_proto` < 7.0.0 and `protobuf` < v28.2, but `README.md` contains a patch for such users. - `protoc/private/toolchain_impl.bzl`: Applies changes from the final state of protocolbuffers/protobuf#19679 and updates the prebuilt `protoc` toolchain resolution check and error message. - `test_dependency_versions.sh`, `deps/test/MODULE.bazel.template`, `deps/test/protoc_toolchains.template`: Adds Bazel 9 and `protobuf` prebuilt `protoc` toolchain test cases. Updates `do_build_and_test` to support both toolchains. Extracts the `protoc_toolchains.template` to include it only in test cases using the `rules_scala` toolchain. Other key components of this change include: - `protoc/BUILD`: Aliases `@protobuf//bazel/toolchains:prefer_prebuilt_protoc` as `@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelrc`, `tools/bazel.rc.buildkite`: Adds `common --@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelci/presubmit.yml`, `.bcr/presubmit.yml`: Adds `9.x` to the `bcr_bazel` matrix . Also applies whitespace and quoting format changes to `.bazelci/presubmit.yml`. Also contains these small, but important changes to fix continuous integration breakages and `last_green` Bazel compatibility: - `.github/workflows/release_prep.sh`: Excludes `.bazelversion` symlinks, since per bazelbuild/bazel-worker-api#21, these break Bazel as of bazelbuild/bazel@f942a70. - `MODULE.bazel`, `scala/private/macros/workspace_compat.bzl`: Bumps `bazelbuild/bazel-worker-api` to v0.0.10 to fix `last_green` builds per bazelbuild/bazel-worker-api#22. - `test/shell/test_runner.sh`, `test/shell/test_bzlmod_macros.sh`: Replaces `relative_path_to_parent` with `convert_msys2_path` for bazelbuild/continuous-integration#2445, bazelbuild/bazel#28406. It turns out @WojciechMazur patched this in bazel-contrib#1794, but this change supersedes the change from that pull request. Other minor changes include: - `MODULE.bazel`: Removes the `version` parameter from the `module` declaration. The Publish to BCR GitHub Actions workflow patches this parameter anyway. See: - https://bazel.build/external/faq#module-versioning-best-practices - https://github.com/bazel-contrib/publish-to-bcr/blob/v1.1.0/src/domain/create-entry.ts#L176-L216 - https://bazelbuild.slack.com/archives/C014RARENH0/p1767892286188219 - Replaces `precompiled` with `prebuilt` in every reference to the prebuilt `protoc` toolchain. - `scala/toolchains_repo.bzl`: Replaces `@@{rules_scala_repo}` references with plain `@rules_scala` references, allowing repo mapping to do its thing. - `test_dependency_version.sh`: Skips non-prebuilt `protoc` prebuilt toolchain test cases from for macOS on continuous integration. This test suite was taking a disproportionately long time to run before adding the new test cases, which Linux builds will continue to cover. A summary of these changes excluding `README.md`, `protobuf.patch`, `MODULE.bazel{,.lock}` files, and Maven dependency updates: ```txt $ git diff --stat HEAD^ ':!README.md' ':!**MODULE.bazel*' \ ':!**protobuf.patch' ':!third_party/repositories/scala_*' .bazelci/presubmit.yml | 66 ++++++... .bazelrc | 9 ++++-- .bcr/presubmit.yml | 2 +- .github/workflows/release_prep.sh | 7 ++++- WORKSPACE | 15 ++++++... deps/test/protoc_toolchains.template | 12 ++++++++ protoc/BUILD | 20 ++++++... protoc/private/protoc_integrity.bzl | 10 +++++++ protoc/private/protoc_toolchains.bzl | 19 +++---... protoc/private/toolchain_impl.bzl | 30 ++++++... protoc/toolchains.bzl | 2 +- scala/extensions/protoc.bzl | 8 +++--- scala/latest_deps.bzl | 8 ++---- scala/private/macros/test/MODULE.bzlmod_test | 8 +++++- scala/private/macros/test/MODULE.bzlmod_test_root_module | 8 +++++- scala/private/macros/workspace_compat.bzl | 6 ++-- scala/toolchains.bzl | 5 +++- scala/toolchains_repo.bzl | 37 ++++++... scala_proto/scala_proto_toolchain.bzl | 2 ++ scripts/create_repository.py | 2 +- scripts/update_protoc_integrity.py | 6 ++-- test/shell/test_bzlmod_macros.sh | 23 ++++--... test/shell/test_runner.sh | 34 ++++++... test_dependency_versions.sh | 151 ++++++... test_version.sh | 2 -- tools/bazel.rc.buildkite | 9 ++++-- 26 files changed, 328 insertions(+), 173 deletions(-) ``` --- In the following Bazel Slack thread, @rafikk pointed out the apparent incompatibility with the new Bazel 9.0.0 release: - https://bazelbuild.slack.com/archives/CDCKJ2KFZ/p1768383285232129 - https://github.com/bazelbuild/bazel/releases/tag/9.0.0 This happened because Bazel 9 now sets `--incompatible_enable_proto_toolchain_resolution` by default: - https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution Since the `scala_proto` implementation depends on this flag, this broke projects that didn't configure the `rules_scala` prebuilt `protoc` toolchain. This change fixes that problem while ensuring seamless compatibility with the `protobuf` v33.4 prebuilt protoc toolchain. We will likely release v7.2.0 as soon as this change lands. After that, we can bump other dependency versions in a patch level release.

@WojciechMazur

Replaces the `rules_scala` prebuilt `protoc` toolchain everywhere except in a few `test_dependency_versions.sh` test cases. Updates `README.md` to recommend using the `protobuf` prebuilt `protoc` toolchain, and refactors and adds test cases to `test_dependency_versions.sh`. Note that legacy `WORKSPACE` builds continue to use the `rules_scala` prebuilt `protoc` toolchain, since the `protobuf` implementation only supports Bzlmod. After removing `protobuf.patch` files, updating `MODULE.bazel{,.lock}` files, and updating Maven dependencies, the most significant changes for supporting `protobuf` v33.4 are: - `protoc/BUILD`, `protoc/private/protoc_toolchains.bzl`: Moves `proto_lang_toolchain` from `@rules_scala_protoc_toolchains` to `//protoc:scala_protoc_toolchain`. This can break users using `rules_proto` < 7.0.0 and `protobuf` < v28.2, but `README.md` contains a patch for such users. - `protoc/private/toolchain_impl.bzl`: Applies changes from the final state of protocolbuffers/protobuf#19679 and updates the prebuilt `protoc` toolchain resolution check and error message. - `test_dependency_versions.sh`, `deps/test/MODULE.bazel.template`, `deps/test/protoc_toolchains.template`: Adds Bazel 9 and `protobuf` prebuilt `protoc` toolchain test cases. Updates `do_build_and_test` to support both toolchains. Extracts the `protoc_toolchains.template` to include it only in test cases using the `rules_scala` toolchain. Other key components of this change include: - `protoc/BUILD`: Aliases `@protobuf//bazel/toolchains:prefer_prebuilt_protoc` as `@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelrc`, `tools/bazel.rc.buildkite`: Adds `common --@rules_scala//protoc:prefer_prebuilt_protoc`. - `.bazelci/presubmit.yml`, `.bcr/presubmit.yml`: Adds `9.x` to the `bcr_bazel` matrix . Also applies whitespace and quoting format changes to `.bazelci/presubmit.yml`. Also contains these small, but important changes to fix continuous integration breakages and `last_green` Bazel compatibility: - `.github/workflows/release_prep.sh`: Excludes `.bazelversion` symlinks, since per bazelbuild/bazel-worker-api#21, these break Bazel as of bazelbuild/bazel@f942a70. - `MODULE.bazel`, `scala/private/macros/workspace_compat.bzl`: Bumps `bazelbuild/bazel-worker-api` to v0.0.10 to fix `last_green` builds per bazelbuild/bazel-worker-api#22. - `test/shell/test_runner.sh`, `test/shell/test_bzlmod_macros.sh`: Replaces `relative_path_to_parent` with `convert_msys2_path` for bazelbuild/continuous-integration#2445, bazelbuild/bazel#28406. It turns out @WojciechMazur patched this in bazel-contrib#1794, but this change supersedes the change from that pull request. Other minor changes include: - `MODULE.bazel`: Removes the `version` parameter from the `module` declaration. The Publish to BCR GitHub Actions workflow patches this parameter anyway. See: - https://bazel.build/external/faq#module-versioning-best-practices - https://github.com/bazel-contrib/publish-to-bcr/blob/v1.1.0/src/domain/create-entry.ts#L176-L216 - https://bazelbuild.slack.com/archives/C014RARENH0/p1767892286188219 - Replaces `precompiled` with `prebuilt` in every reference to the prebuilt `protoc` toolchain. - `scala/toolchains_repo.bzl`: Replaces `@@{rules_scala_repo}` references with plain `@rules_scala` references, allowing repo mapping to do its thing. - `test_dependency_version.sh`: Skips non-prebuilt `protoc` prebuilt toolchain test cases from for macOS on continuous integration. This test suite was taking a disproportionately long time to run before adding the new test cases, which Linux builds will continue to cover. A summary of these changes excluding `README.md`, `protobuf.patch`, `MODULE.bazel{,.lock}` files, and Maven dependency updates: ```txt $ git diff --stat HEAD^ ':!README.md' ':!**MODULE.bazel*' \ ':!**protobuf.patch' ':!third_party/repositories/scala_*' .bazelci/presubmit.yml | 66 ++++++... .bazelrc | 9 ++++-- .bcr/presubmit.yml | 2 +- .github/workflows/release_prep.sh | 7 ++++- WORKSPACE | 15 ++++++... deps/test/protoc_toolchains.template | 12 ++++++++ protoc/BUILD | 20 ++++++... protoc/private/protoc_integrity.bzl | 10 +++++++ protoc/private/protoc_toolchains.bzl | 19 +++---... protoc/private/toolchain_impl.bzl | 30 ++++++... protoc/toolchains.bzl | 2 +- scala/extensions/protoc.bzl | 8 +++--- scala/latest_deps.bzl | 8 ++---- scala/private/macros/test/MODULE.bzlmod_test | 8 +++++- scala/private/macros/test/MODULE.bzlmod_test_root_module | 8 +++++- scala/private/macros/workspace_compat.bzl | 6 ++-- scala/toolchains.bzl | 5 +++- scala/toolchains_repo.bzl | 26 ++++++... scala_proto/scala_proto_toolchain.bzl | 2 ++ scripts/create_repository.py | 2 +- scripts/update_protoc_integrity.py | 6 ++-- test/shell/test_bzlmod_macros.sh | 23 ++++--... test/shell/test_runner.sh | 34 ++++++... test_dependency_versions.sh | 151 ++++++... test_version.sh | 2 -- tools/bazel.rc.buildkite | 9 ++++-- 26 files changed, 317 insertions(+), 173 deletions(-) ``` --- In the following Bazel Slack thread, @rafikk pointed out the apparent incompatibility with the new Bazel 9.0.0 release: - https://bazelbuild.slack.com/archives/CDCKJ2KFZ/p1768383285232129 - https://github.com/bazelbuild/bazel/releases/tag/9.0.0 This happened because Bazel 9 now sets `--incompatible_enable_proto_toolchain_resolution` by default: - https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution Since the `scala_proto` implementation depends on this flag, this broke projects that didn't configure the `rules_scala` prebuilt `protoc` toolchain. This change fixes that problem while ensuring seamless compatibility with the `protobuf` v33.4 prebuilt protoc toolchain. We will likely release v7.2.0 as soon as this change lands. After that, we can bump other dependency versions in a patch level release. --- Also, Bazel 10.0.0-pre.20251208.3 selects the prebuilt `protoc` toolchain, and 10.0.0-pre.20251217.3 doesn't. Investigating.

bazel: Remove hardcoded dependency on //:protoc from language runtimes

bc1a0e3

Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc.

fmeum marked this pull request as ready for review December 16, 2024 15:25

Switch to toolchains

3bd939d

fmeum force-pushed the 19558-dont-hardcode-protoc branch from d92eae2 to 3bd939d Compare December 16, 2024 21:20

zhangskz added the bazel label Dec 18, 2024

zhangskz requested a review from mkruskal-google December 18, 2024 16:57

mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Dec 18, 2024

github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Dec 18, 2024

JasonLunn added the wait for user action label Dec 23, 2024

fmeum mentioned this pull request Jan 15, 2025

chore: add Bazel 8 testing aspect-build/toolchains_protoc#39

Merged

alexeagle mentioned this pull request Jan 16, 2025

feat: re-enable Java example of using pre-built protoc aspect-build/toolchains_protoc#41

Closed

tonyliaoss removed the wait for user action label Feb 19, 2025

mbland mentioned this pull request Mar 1, 2025

Bazel 8 + rules_java 8 updates, protoc toolchain bazel-contrib/rules_scala#1710

Merged

mbland mentioned this pull request Mar 3, 2025

Support Bzlmod and add rules_scala to bazel-central-registry bazel-contrib/rules_scala#1482

Closed

copybara-service bot mentioned this pull request Mar 22, 2025

bazel: Remove hardcoded dependency on //:protoc from language runtimes (#19679) #20794

Closed

Merge remote-tracking branch 'upstream' into 19558-dont-hardcode-protoc

2d67e82

mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Mar 22, 2025

github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Mar 22, 2025

mbland mentioned this pull request May 9, 2025

Publish [email protected] bazelbuild/bazel-central-registry#4552

Merged

nickbreen mentioned this pull request Jul 12, 2025

mitigate protocolbuffers/protobuf#19558 nickbreen/rules_unused_deps#9

Closed

mbland mentioned this pull request Aug 11, 2025

[Bug]: Windows builds do not support symlinks bazelbuild/bazel-central-registry#5506

Closed

mbland mentioned this pull request Aug 11, 2025

Fix BCR presubmit, update rules_python to 1.5.2 bazel-contrib/rules_scala#1760

Merged

github-actions bot added the inactive Denotes the issue/PR has not seen activity in the last 90 days. label Sep 12, 2025

github-actions bot added untriaged auto added to all issues by default when created. and removed inactive Denotes the issue/PR has not seen activity in the last 90 days. labels Sep 13, 2025

mbland mentioned this pull request Sep 15, 2025

Bump dependency versions for rules_scala v7.1.2 bazel-contrib/rules_scala#1772

Merged

mbland mentioned this pull request Oct 19, 2025

Update dependencies in preparation for v7.1.3 bazel-contrib/rules_scala#1778

Merged

mkruskal-google approved these changes Nov 3, 2025

View reviewed changes

copybara-service bot closed this in 4986a77 Nov 4, 2025

fmeum deleted the 19558-dont-hardcode-protoc branch January 10, 2026 05:44

mbland mentioned this pull request Jan 26, 2026

Support Protobuf 33.4 w/ prebuilt protoc toolchain bazel-contrib/rules_scala#1803

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

Uh oh!

fmeum commented Dec 16, 2024 •

edited

Loading

Uh oh!

fmeum commented Dec 16, 2024

Uh oh!

fmeum commented Dec 16, 2024

Uh oh!

mkruskal-google commented Dec 19, 2024

Uh oh!

fmeum commented Dec 30, 2024

Uh oh!

comius commented Mar 19, 2025

Uh oh!

mkruskal-google commented Apr 30, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 12, 2025

Uh oh!

alexeagle commented Sep 12, 2025

Uh oh!

vibbix commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679

bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679

Uh oh!

Conversation

fmeum commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fmeum commented Dec 16, 2024

Uh oh!

fmeum commented Dec 16, 2024

Uh oh!

mkruskal-google commented Dec 19, 2024

Uh oh!

fmeum commented Dec 30, 2024

Uh oh!

comius commented Mar 19, 2025

Uh oh!

mkruskal-google commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 12, 2025

Uh oh!

alexeagle commented Sep 12, 2025

Uh oh!

vibbix commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

fmeum commented Dec 16, 2024 •

edited

Loading

mkruskal-google commented Apr 30, 2025 •

edited

Loading