Thanks to visit codestin.com
Credit goes to github.com

Skip to content

The iteration behavior of IntFlag has changed backwards-incompatibly in python 3.11 and the published documentation is now incorrect #99304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zzzeek opened this issue Nov 9, 2022 · 9 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@zzzeek
Copy link

zzzeek commented Nov 9, 2022

At the Python documentation for Enum at https://docs.python.org/3/library/enum.html, the first paragraph contains the definition of an enum:

An enumeration:

  • is a set of symbolic names (members) bound to unique values
  • can be iterated over to return its members in definition order
  • uses call syntax to return members by value
  • uses index syntax to return members by name

The second bullet is no longer the case in Python 3.11. The demonstration below passes on Python 3.10 and earlier, fails on Python 3.11:

from enum import Enum, IntFlag


class MyEnum(Enum):
    VAL1 = 1
    VAL2 = 2
    VAL3 = 3


class MyIntFlag(IntFlag):
    VAL1 = 1
    VAL2 = 2
    VAL3 = 3


assert list(MyEnum) == [MyEnum.VAL1, MyEnum.VAL2, MyEnum.VAL3]
assert list(MyIntFlag) == [MyIntFlag.VAL1, MyIntFlag.VAL2, MyIntFlag.VAL3], list(MyIntFlag)

on Python 3.10, the script succeeds without output.

output on Python 3.11:

$ /opt/python-3.11.0/bin/python3 test3.py 
Traceback (most recent call last):
  File "/home/classic/dev/sqlalchemy/test3.py", line 17, in <module>
    assert list(MyIntFlag) == [MyIntFlag.VAL1, MyIntFlag.VAL2, MyIntFlag.VAL3], list(MyIntFlag)
AssertionError: [<MyIntFlag.VAL1: 1>, <MyIntFlag.VAL2: 2>]

I would think this might be a regression in Python 3.11, but seeing as there are lots of changes in IntEnum overall, maybe this was intended. It does seem to be counter to the usual spirit of the standard library to change an explicitly documented behavior like this.

@vstinner
Copy link
Member

vstinner commented Nov 9, 2022

cc @ethanfurman @warsaw

@ethanfurman
Copy link
Member

The change is to Flag and IntFlag: in plain enums the canonical name is the one seen first for any particular value, and an alias is a different name for that same value; in flags the canonical name is the first one seen for a power-of-two value, and aliases are different names for that same value, or names of more than one power-of-two value.

So in the example above, MyIntFlag.VAL3 is an alias for MyIntFlag.VAL1 | MyIntFlag.VAL2.

@zzzeek
Copy link
Author

zzzeek commented Nov 9, 2022

Is MyIntFlag.VAL3 a member of the enum? It's still present in __members__. If it's a member, then the phrase, "can be iterated over to return its members in definition order" would appear that it needs to be changed and this is a backwards incompatible change.

@zzzeek
Copy link
Author

zzzeek commented Nov 9, 2022

if it's not a member, then why is it in __members__. That is, I really think this change is a mistake.

@zzzeek
Copy link
Author

zzzeek commented Nov 9, 2022

it's also not in dir(), which IMO is unintuitive (also a behavioral change).

class MyIntFlag(IntFlag):
    VAL1 = 1
    VAL2 = 2
    VAL3 = 3
    VAL4 = 4

print(dir(MyIntFlag))

for python 3.10:

['VAL1', 'VAL2', 'VAL3', 'VAL4', '__class__', '__doc__', '__members__', '__module__']

python 3.11:

['VAL1', 'VAL2', 'VAL4', '__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__contains__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getstate__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__iter__', '__le__', '__len__', '__lshift__', '__lt__', '__members__', '__mod__', '__module__', '__mul__', '__name__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__qualname__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'as_integer_ratio', 'bit_count', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']

This is quite a signifcant change, AFAIK there was no deprecation warning, I'm not sure where this is documented either

@ethanfurman
Copy link
Member

Compared to an enum, 3.11 now matches 3.10 and earlier:

class Kind(IntEnum):
    A = 1
    B = 2
    C = 3
    D = 1

dir(Kind)
# ['A', 'B', 'C', ...]

There is no D because it is an alias

Kind.__members__
# mappingproxy({'A': <Kind.A: 1>, 'B': <Kind.B: 2>, 'C': <Kind.C: 3>, 'D': <Kind.A: 1>})

D is there because it's a valid name, but it returns Kind.A.

The docs should be corrected, possibly by inserting 'canonical' into 'iterated over to return its members'. Flags did not have aliasing properly defined and implemented until 3.11, although enums did since they appeared in 3.4.

@zzzeek
Copy link
Author

zzzeek commented Nov 10, 2022

OK so aliases are returned in __members__, and not in dir() or iter(), is that accurate?

@zzzeek
Copy link
Author

zzzeek commented Nov 10, 2022

Also this is definitely a behavioral change either way and should be noted.

from my end it of course doesn't matter, I have to change my code regardless since we support all Python versions since 3.7.

sqlalchemy-bot pushed a commit to sqlalchemy/sqlalchemy that referenced this issue Nov 11, 2022
in [1], Python 3.11 seems to have changed the behavior of
IntEnum.  We didn't notice this because we have our own
workaround class already, but typing did.   Ensure we remain
compatible with IntFlag.

This change also modifies FastIntFlag to no longer use
global symbols; this is unnecessary as we assign FastIntFlag
members explicitly.  Use of ``symbol()`` should probably
be phased out.

[1] python/cpython#99304
Fixes: #8783

Change-Id: I8ae2e871ff1467ae5ca1f63e66b5dae45d4a6c93
ethanfurman added a commit that referenced this issue Nov 12, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Nov 12, 2022
…H-99395)

(cherry picked from commit 73a921b)

Co-authored-by: Ethan Furman <[email protected]>
Co-authored-by: C.A.M. Gerlach <[email protected]>
ethanfurman added a commit that referenced this issue Nov 12, 2022
…) (GH-99415)

gh-99304: [Enum] clarify what constitutes a flag alias (GH-99395)
(cherry picked from commit 73a921b)

Co-authored-by: Ethan Furman <[email protected]>
Co-authored-by: C.A.M. Gerlach <[email protected]>
@zzzeek
Copy link
Author

zzzeek commented Nov 12, 2022

great changes, thanks for clarifying

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants