Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: allow Protobuf 6.x #1445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Mar 18, 2025
Merged

fix: allow Protobuf 6.x #1445

merged 27 commits into from
Mar 18, 2025

Conversation

parthea
Copy link
Contributor

@parthea parthea commented Mar 5, 2025

See https://pypi.org/project/protobuf/6.30.0/
See googleapis/gapic-generator-python#2347 for removing dev in setup.py

Wait for googleapis/gapic-generator-python#2352, googleapis/python-api-core#804, googleapis/proto-plus-python#536

BEGIN_COMMIT_OVERRIDE
fix: add trove classifier for Python 3.13
fix: resolve issue where pre-release versions of dependencies are installed
fix(deps): require protobuf >= 3.20.2, < 7.0.0
fix(deps): require google-crc32c >= 1.1.3
fix(deps): require requests >= 2.22.0
tests: add tests for Python 3.13
tests: add pre-release testing via prerelease_deps nox session
tests(samples): add configuration for python 3.13
tests: add constraints file to test the minimum supported version of a dependency
END_COMMIT_OVERRIDE

@parthea parthea added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 5, 2025
@product-auto-label product-auto-label bot added size: s Pull request size is small. api: storage Issues related to the googleapis/python-storage API. labels Mar 5, 2025
@parthea
Copy link
Contributor Author

parthea commented Mar 12, 2025

Wait for cl/736096543

@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Mar 12, 2025
@parthea parthea removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 12, 2025
@parthea
Copy link
Contributor Author

parthea commented Mar 12, 2025

Wait for cl/736277031

@parthea
Copy link
Contributor Author

parthea commented Mar 13, 2025

Python 3.13 google-crc32 related tests are failing:

nox > Running session unit-3.13
nox > Creating virtual environment (virtualenv) using python3.13 in .nox/unit-3-13
nox > python -m pip install mock pytest pytest-cov brotli -c /tmpfs/src/github/python-storage/testing/constraints-3.13.txt
nox > python -m pip install opentelemetry-api opentelemetry-sdk
nox > python -m pip install -e . -c /tmpfs/src/github/python-storage/testing/constraints-3.13.txt
nox > pip uninstall -y google-resumable-media
Found existing installation: google-resumable-media 2.7.2
Uninstalling google-resumable-media-2.7.2:
  Successfully uninstalled google-resumable-media-2.7.2
nox > py.test --quiet --junitxml=unit_3.13_sponge_log.xml --cov=google.cloud.storage --cov=google.cloud --cov=tests.unit --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=0 tests/unit tests/resumable_media/unit
........................................................................ [  4%]
........................................................................ [  9%]
........................................................................ [ 14%]
........................................................................ [ 19%]
........................................................................ [ 24%]
........................................................................ [ 29%]
........................................................................ [ 34%]
......................Exception ignored in: <google.cloud.storage.fileio.BlobWriter object at 0x7f8d7cf21e10>
Traceback (most recent call last):
  File "/tmpfs/src/github/python-storage/google/cloud/storage/fileio.py", line 419, in close
    self._upload_chunks_from_buffer(1)
  File "/tmpfs/src/github/python-storage/google/cloud/storage/fileio.py", line 387, in _upload_chunks_from_buffer
    self._initiate_upload()
  File "/tmpfs/src/github/python-storage/google/cloud/storage/fileio.py", line 372, in _initiate_upload
    self._upload_and_transport = self._blob._initiate_resumable_upload(
  File "/tmpfs/src/github/python-storage/google/cloud/storage/blob.py", line 2141, in _initiate_resumable_upload
    info = self._get_upload_arguments(client, content_type, command=command)
  File "/tmpfs/src/github/python-storage/google/cloud/storage/blob.py", line 1833, in _get_upload_arguments
    **client._extra_headers,
  File "/tmpfs/src/github/python-storage/.nox/unit-3-13/lib/python3.13/site-packages/mock/mock.py", line 699, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute '_extra_headers'
.................................................. [ 39%]
........................................................................ [ 44%]
........................................................................ [ 49%]
........................................................................ [ 54%]
........................................................................ [ 59%]
........................................................................ [ 64%]
........................................................................ [ 69%]
........................................................................ [ 74%]
........................................................................ [ 79%]
........................................................................ [ 84%]
........................................................................ [ 89%]
.......................F.....................................F.F........ [ 94%]
.F....................................................................F. [ 99%]
..                                                                       [100%]
=================================== FAILURES ===================================
______________________ test__is_crc32c_available_and_fast ______________________

    def test__is_crc32c_available_and_fast():
        import sys
    
        import google_crc32c
    
>       assert google_crc32c.implementation == "c"
E       AssertionError: assert 'python' == 'c'
E         
E         - c
E         + python

tests/resumable_media/unit/test__helpers.py:183: AssertionError
________________ TestMultipartUpload.test_constructor_defaults _________________

self = <tests.resumable_media.unit.test__upload.TestMultipartUpload object at 0x7f8d7c737610>

    def test_constructor_defaults(self):
        upload = _upload.MultipartUpload(MULTIPART_URL)
        assert upload.upload_url == MULTIPART_URL
        assert upload._headers == {}
>       assert upload._checksum_type == "crc32c"  # converted from "auto"
E       AssertionError: assert 'md5' == 'crc32c'
E         
E         - crc32c
E         + md5

tests/resumable_media/unit/test__upload.py:194: AssertionError
______________ TestMultipartUpload.test_constructor_explicit_auto ______________

self = <tests.resumable_media.unit.test__upload.TestMultipartUpload object at 0x7f8d7c3b49d0>

    def test_constructor_explicit_auto(self):
        headers = {"spin": "doctors"}
        upload = _upload.MultipartUpload(
            MULTIPART_URL, headers=headers, checksum="auto"
        )
        assert upload.upload_url == MULTIPART_URL
        assert upload._headers is headers
>       assert upload._checksum_type == "crc32c"
E       AssertionError: assert 'md5' == 'crc32c'
E         
E         - crc32c
E         + md5

tests/resumable_media/unit/test__upload.py:214: AssertionError
_____________________ TestResumableUpload.test_constructor _____________________

self = <tests.resumable_media.unit.test__upload.TestResumableUpload object at 0x7f8d7c737750>

    def test_constructor(self):
        chunk_size = ONE_MB
        upload = _upload.ResumableUpload(RESUMABLE_URL, chunk_size)
        assert upload.upload_url == RESUMABLE_URL
        assert upload._headers == {}
        assert not upload._finished
        _check_retry_strategy(upload)
        assert upload._chunk_size == chunk_size
        assert upload._stream is None
        assert upload._content_type is None
        assert upload._bytes_uploaded == 0
        assert upload._bytes_checksummed == 0
        assert upload._checksum_object is None
        assert upload._total_bytes is None
        assert upload._resumable_url is None
>       assert upload._checksum_type == "crc32c"  # converted from "auto"
E       AssertionError: assert 'md5' == 'crc32c'
E         
E         - crc32c
E         + md5

tests/resumable_media/unit/test__upload.py:359: AssertionError
______________________________ test_xml_mpu_part _______________________________

filename = '/tmp/tmp6jsl5w9j'

    def test_xml_mpu_part(filename):
        PART_NUMBER = 1
        START = 0
        END = 256
        ETAG = PARTS[1]
    
        part = _upload.XMLMPUPart(
            EXAMPLE_XML_UPLOAD_URL,
            UPLOAD_ID,
            filename,
            START,
            END,
            PART_NUMBER,
            headers=EXAMPLE_HEADERS,
            checksum="md5",
        )
        assert part.upload_url == EXAMPLE_XML_UPLOAD_URL
        assert part.upload_id == UPLOAD_ID
        assert part.filename == filename
        assert part.etag is None
        assert part.start == START
        assert part.end == END
        assert part.part_number == PART_NUMBER
        assert part._headers == EXAMPLE_HEADERS
        assert part._checksum_type == "md5"
        assert part._checksum_object is None
    
        part = _upload.XMLMPUPart(
            EXAMPLE_XML_UPLOAD_URL,
            UPLOAD_ID,
            filename,
            START,
            END,
            PART_NUMBER,
            headers=EXAMPLE_HEADERS,
            checksum="auto",
        )
        assert part.upload_url == EXAMPLE_XML_UPLOAD_URL
        assert part.upload_id == UPLOAD_ID
        assert part.filename == filename
        assert part.etag is None
        assert part.start == START
        assert part.end == END
        assert part.part_number == PART_NUMBER
        assert part._headers == EXAMPLE_HEADERS
>       assert part._checksum_type == "crc32c"  # transformed from "auto"
E       AssertionError: assert 'md5' == 'crc32c'
E         
E         - crc32c
E         + md5

tests/resumable_media/unit/test__upload.py:1425: AssertionError
=============================== warnings summary ===============================
.nox/unit-3-13/lib/python3.13/site-packages/google_crc32c/__init__.py:29
  /tmpfs/src/github/python-storage/.nox/unit-3-13/lib/python3.13/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
    warnings.warn(_SLOW_CRC32C_WARNING, RuntimeWarning)

@parthea parthea added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 13, 2025
@parthea
Copy link
Contributor Author

parthea commented Mar 13, 2025

The issue is that wheels aren't published for google-crc32 yet

@parthea parthea added the status: blocked Resolving the issue is dependent on other work. label Mar 13, 2025
@parthea parthea marked this pull request as ready for review March 18, 2025 10:47
@parthea parthea requested review from a team as code owners March 18, 2025 10:47
@parthea parthea requested a review from balajismaniam March 18, 2025 10:47
@parthea parthea added owlbot:run Add this label to trigger the Owlbot post processor. and removed status: blocked Resolving the issue is dependent on other work. labels Mar 18, 2025
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2025
@parthea parthea added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2025
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2025
@parthea parthea added kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Mar 18, 2025
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Mar 18, 2025
@parthea
Copy link
Contributor Author

parthea commented Mar 18, 2025

@cojenco, Please could you take a look?

Copy link
Contributor

@cojenco cojenco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

CURRENT_DIRECTORY = pathlib.Path(__file__).parent.absolute()

# Error if a python version is missing
nox.options.error_on_missing_interpreters = True

nox.options.sessions = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC this is setting a list of sessions to run by default (to exclude prerelease_deps so that it is only ran in kokoro prerelease_deps)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "client-library-test-universe-domain-credential"
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we've considered running prerelease-deps in continuous instead of presubmits. This increases presubmit running time and the extra two runs of the system test suite are more prone to 429 errors and server-side resource overload etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cl/738167833 and 77a3dc8

@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

@parthea parthea enabled auto-merge (squash) March 18, 2025 22:47
@parthea parthea merged commit 0100916 into main Mar 18, 2025
16 of 17 checks passed
@parthea parthea deleted the allow-protobuf-6 branch March 18, 2025 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants