Enum failed to initialize in PyPy #506

airbender-1 · 2024-11-27T17:08:18Z

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is this a client library issue or a product issue? We will only be able to assist with issues that pertain to the behaviors of this library. If the issue you're experiencing is due to the behavior of the product itself, please visit the Support page to reach the most relevant engineers.

If the support paths suggested above still do not result in a resolution, please provide the following details.

Environment details

Programming language: Python
OS: Mac
Language runtime version: 3.10
Package version: 1.25.0

Steps to reproduce

Install latest PyPy on Mac via brew:

brew install pypy3

Create virtual environment for PyPy

pypy3 -m ensurepip --upgrade
pypy3 -m venv .venv-pypy3

Activate virtual environment

source .venv-pypy3/bin/activate

Upgrade pip in the PyPy environment

pip install --upgrade pip

Install latest google-cloud-aiplatform (version google-cloud-aiplatform==1.73.0) that uses proto-plus==1.25.0

pip install google-cloud-aiplatform

Create test_aiplatform.py script:

from google.cloud import aiplatform

Run the script with PyPy:

pypy ./src/test_aiplatform.py

Expected:
No failures

Actual:
TypeError: 'set' object does not support item deletion

Error is raised from .venv-pypy3/lib/pypy3.10/site-packages/proto/enums.py, see in code:

    del attrs._member_names[pb_options]

Call Stack:

Traceback (most recent call last):
File "/Users/airbender/code/woodpecker/src/test_aiplatform.py", line 1, in
from google.cloud import aiplatform as aiplatform
File "", line 1078, in _handle_fromlist
File "", line 241, in _call_with_frames_removed
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform/init.py", line 24, in
from google.cloud.aiplatform import initializer
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform/initializer.py", line 35, in
from google.cloud.aiplatform import compat
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform/compat/init.py", line 18, in
from google.cloud.aiplatform.compat import services
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform/compat/services/init.py", line 18, in
from google.cloud.aiplatform_v1beta1.services.dataset_service import (
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/init.py", line 21, in
from .services.dataset_service import DatasetServiceClient
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/services/dataset_service/init.py", line 16, in
from .client import DatasetServiceClient
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/services/dataset_service/client.py", line 53, in
from google.cloud.aiplatform_v1beta1.services.dataset_service import pagers
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/services/dataset_service/pagers.py", line 40, in
from google.cloud.aiplatform_v1beta1.types import annotation
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/types/init.py", line 423, in
from .feature_online_store_service import (
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/types/feature_online_store_service.py", line 112, in
class FetchFeatureValuesRequest(proto.Message):
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/google/cloud/aiplatform_v1beta1/types/feature_online_store_service.py", line 143, in FetchFeatureValuesRequest
class Format(proto.Enum):
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/proto/enums.py", line 69, in new
del attrs._member_names[pb_options]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'set' object does not support item deletion

Possible solution:

I think this could be fixed by checking for type of attrs._member_names to be a set, like so (tested on my machine).

Original:

class ProtoEnumMeta(enum.EnumMeta):
    def __new__(mcls, name, bases, attrs):
        # ...
        if pb_options in attrs._member_names:
            if isinstance(attrs._member_names, list):
                idx = attrs._member_names.index(pb_options)
                attrs._member_names.pop(idx)
            else:  # Python 3.11.0b3
                del attrs._member_names[pb_options]

Fixed:

class ProtoEnumMeta(enum.EnumMeta):
    def __new__(mcls, name, bases, attrs):
        # ...
        if pb_options in attrs._member_names:
            if isinstance(attrs._member_names, list):
                idx = attrs._member_names.index(pb_options)
                attrs._member_names.pop(idx)
            # START OF WORKAROUND for PyPy:
            elif isinstance(attrs._member_names, set):
                attrs._member_names.discard(pb_options)
            # END OF WORKAROUND
            else:  # Python 3.11.0b3
                del attrs._member_names[pb_options]

see PR #507

Reproduce with proto-plus only in PyPy

I also managed to reproduce the same issue with the simplified code without any dependency on google aiplatform like so.

test_script.py

import proto


class Format(proto.Enum):
        _pb_options = {"deprecated": True}
        FORMAT_UNSPECIFIED = 0
        KEY_VALUE = 1
        PROTO_STRUCT = 2

Run with PyPy:

pypy test_script.py

Throws the same error:

Traceback (most recent call last):
File "/Users/airbender/code/woodpecker/src/test_script.py", line 5, in
class Format(proto.Enum):
File "/Users/airbender/code/woodpecker/.venv-pypy3/lib/pypy3.10/site-packages/proto/enums.py", line 69, in new
del attrs._member_names[pb_options]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'set' object does not support item deletion

The text was updated successfully, but these errors were encountered:

When removing the _pb_options name from the enum attributes, enums initialization in PyPy should not raise exception. Refs: googleapis#506

parthea · 2025-01-22T11:57:50Z

Thanks @airbender-1 for the detailed error report and fix!

I was able to reproduce the issue in #523
https://github.com/googleapis/proto-plus-python/actions/runs/12907548308/job/35991217125

_____________________________ test_enum_alias_good _____________________________

    def test_enum_alias_good():
        # Have to split good and bad enum alias into two tests so that the generated
        # file descriptor is properly created.
        # For the python runtime, aliases are allowed by default, but we want to
        # make sure that the options don't cause problems.
        # For the cpp runtime, we need to verify that we can in fact define aliases.
>       class GoodMessage(proto.Message):

tests/test_fields_enum.py:386: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_fields_enum.py:387: in GoodMessage
    class GoodEnum(proto.Enum):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

mcls = <class 'proto.enums.ProtoEnumMeta'>, name = 'GoodEnum'
bases = (<enum 'Enum'>,)
attrs = {'_generate_next_value_': <function Enum._generate_next_value_ at 0x00007f14a2f6e020>, '__module__': 'test_fields_enum', '__qualname__': 'test_enum_alias_good.<locals>.GoodMessage.GoodEnum', 'UNKNOWN': 0, 'DEFAULT': 0}

    def __new__(mcls, name, bases, attrs):
        # Do not do any special behavior for `proto.Enum` itself.
        if bases[0] == enum.IntEnum:
            return super().__new__(mcls, name, bases, attrs)
    
        # Get the essential information about the proto package, and where
        # this component belongs within the file.
        package, marshal = _package_info.compile(name, attrs)
    
        # Determine the local path of this proto component within the file.
        local_path = tuple(attrs.get("__qualname__", name).split("."))
    
        # Sanity check: We get the wrong full name if a class is declared
        # inside a function local scope; correct this.
        if "<locals>" in local_path:
            ix = local_path.index("<locals>")
            local_path = local_path[: ix - 1] + local_path[ix + 1 :]
    
        # Determine the full name in protocol buffers.
        full_name = ".".join((package,) + local_path).lstrip(".")
        filename = _file_info._FileInfo.proto_file_name(
            attrs.get("__module__", name.lower())
        )
    
        # Retrieve any enum options.
        # We expect something that looks like an EnumOptions message,
        # either an actual instance or a dict-like representation.
        pb_options = "_pb_options"
        opts = attrs.pop(pb_options, {})
        # This is the only portable way to remove the _pb_options name
        # from the enum attrs.
        # In 3.7 onwards, we can define an _ignore_ attribute and do some
        # mucking around with that.
        if pb_options in attrs._member_names:
            if isinstance(attrs._member_names, list):
                idx = attrs._member_names.index(pb_options)
                attrs._member_names.pop(idx)
            else:  # Python 3.11.0b3
>               del attrs._member_names[pb_options]
E               TypeError: 'set' object does not support item deletion

proto/enums.py:[65](https://github.com/googleapis/proto-plus-python/actions/runs/12907548308/job/35991217125#step:5:66): TypeError

* fix: enums initialization in PyPy When removing the _pb_options name from the enum attributes, enums initialization in PyPy should not raise exception. Refs: #506 * test: add presubmit for PyPy * Add unit (pypy3.10, python) as a required check --------- Co-authored-by: Anthonios Partheniou <[email protected]>

airbender-1 added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Nov 27, 2024

airbender-1 mentioned this issue Nov 27, 2024

fix: fix enums initialization in PyPy #507

Merged

airbender-1 added a commit to airbender-1/proto-plus-python that referenced this issue Nov 28, 2024

fix: enums initialization in PyPy

74747ee

When removing the _pb_options name from the enum attributes, enums initialization in PyPy should not raise exception. Refs: googleapis#506

parthea mentioned this issue Jan 22, 2025

Add presubmit testing for PyPy #521

Closed

parthea added the status: investigating The issue is under investigation, which is determined to be non-trivial. label Jan 22, 2025

parthea self-assigned this Jan 22, 2025

parthea removed the status: investigating The issue is under investigation, which is determined to be non-trivial. label Jan 22, 2025

parthea closed this as completed in #507 Jan 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enum failed to initialize in PyPy #506

Enum failed to initialize in PyPy #506

airbender-1 commented Nov 27, 2024 •

edited

Loading

parthea commented Jan 22, 2025

Enum failed to initialize in PyPy #506

Enum failed to initialize in PyPy #506

Comments

airbender-1 commented Nov 27, 2024 • edited Loading

Environment details

Steps to reproduce

Possible solution:

Reproduce with proto-plus only in PyPy

parthea commented Jan 22, 2025

airbender-1 commented Nov 27, 2024 •

edited

Loading