feat: mirror protoc releases to a Starlark constant #205
Conversation
This allows rules_proto to supply a pre-built binary toolchain for protoc without needing any Bazel module supplied by the protocolbuffers maintainers. Obviates the need for protobuf to be released to the BCR as in protocolbuffers/protobuf#16014 Allows the protocolbuffers team to stop caring about Bazel builds for external users, and to close associated bugs. Next step in rules_proto supplying a toolchain that doesn't require users build protoc from sources.
It seems that Googlers aren't checking the same buildifier warnings in google3 that GitHub enforces, so the main branch was already red. Fixing to green up my PR.
…or code users access
comius
left a comment
There was a problem hiding this comment.
Not to diverge rules_proto, I think it would be good to accept this PR. But eventually when the rules are in protobuf, the toolchain registration should happen there as well.-
| # Workaround: protobuf doesn't publish their integrity hashes to e.g. checksums.txt | ||
| # Create a file that looks like a checksums.txt from a shasum command, i.e. | ||
| # sha384-RVFu8PJJCOSXwYTqH7FyWRSgsP1AAjcEa+VViddVTgtd9wYvZjQoQ8jmlFxwfFw+ protobuf-26.0-rc3.tar.gz | ||
| # sha384-JYSXGTSBfwUU6UzqazUTkT3lTZDzx10YdaNQYjojrT7X1Ro1fA+T4tjJw0e8UISV protobuf-26.0-rc3.zip | ||
| CHECKSUMS=$(mktemp) | ||
| for url in $(jq --arg version $VERSION --raw-output "$DOWNLOAD_URLS_FILTER" <$RELEASES); do | ||
| sha=$(curl -sSL $url | shasum -b -a 384 | awk "{ print \$1 }" | xxd -r -p | base64) | ||
| echo "sha384-${sha} $(basename $url)" | ||
| done | tee "$CHECKSUMS" |
There was a problem hiding this comment.
Downloading an computing hashes doesn't feel very secure / the right thing to do.
There was a problem hiding this comment.
It's https://en.wikipedia.org/wiki/Trust_on_first_use - we assume that within 24 hours of a protobuf release, the files they are serving are still uncompromised.
I can leave a TODO here to use the checksums.txt file if the protobuf project starts publishing one. WDYT?
| - uses: actions/checkout@v4 | ||
| - run: | | ||
| ./proto/private/protoc/mirror_protoc_releases.sh | ||
| npx @bazel/buildifier proto/private/protoc/versions.bzl |
There was a problem hiding this comment.
Does this need to be a separate step or can it be part of the script?
There was a problem hiding this comment.
It could be part of the script. The choice of npx here is technically tied to the software we know to be installed on the GHA runner machine, so if someone ran mirror_protoc_releases.sh locally, it might not be installed. However I don't expect anyone to run it locally, so I'm happy to relocate it if you prefer.
|
Note, |
* chore: fix e2e * chore: fix * chore: copy code review improvements from upstream bazelbuild/rules_proto#205
This allows rules_proto to supply a pre-built binary toolchain for protoc without needing any Bazel module supplied by the protocolbuffers maintainers. Obviates the need for protobuf to be released to the BCR as in protocolbuffers/protobuf#16014 Allows the protocolbuffers team to stop caring about Bazel builds for external users, and to close associated bugs.
Next step in rules_proto supplying a toolchain that doesn't require users build protoc from sources.