-
-
Notifications
You must be signed in to change notification settings - Fork 31.9k
conditional blocks in class definitions seem to be evaluating types even when these conditionals are false #130881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
so I'm reading https://peps.python.org/pep-0749/ now which seems to be where we're going to have to go |
if I can get some guidance on a function in |
this change is really quite bizarre and I continue to hope this isn't intended class MyClass:
somevalue: str
if True:
someothervalue: int
else:
someothervalue: str
print(MyClass.__annotations__) under python 3.13 and below prints python3.14.0a5 is literally running through code and quasi-evaluating it without paying attention to conditionals, this is really the most Perl-ish thing I've ever seen in Python by a mile |
On the other hand, it seems to open new possibilities to interesting experiments that were not possible before 🤔 import enum
import random
class CatState(enum.Enum):
alive = True
dead = False
class SchrödingersCat:
cat: CatState.alive = CatState.alive
if random.choice((True, False)):
cat: CatState.dead = CatState.dead
print(SchrödingersCat.__annotations__, SchrödingersCat.cat) |
I think the answer we're going to get for all of these is that "typing is static and should not have conditionals" I also have vague memories of this discussion re: this hits us in the way our test suite is written, which is one side of it, but we also have some end-user recipes where on the SQLAlchemy side we'd have to tell them to not use TYPE_CHECKING and instead some kind of sentinel value that SQLAlchemy can evaluate, so that when it gets |
Personally I don't think it's correct to evaluate false branches in a way that have effect at runtime. So I hope this is treated as a bug. |
Not at all! Typing is not only static, we also care about our runtime type checkers. |
well while we wait in suspense, it's just this behavior is so weird that I find it hard to believe it was done by accident! |
This is documented in PEP 649: "Code that sets annotations on module or class attributes from inside any kind of flow control statement. It’s currently possible to set module and class attributes with annotations inside an if or try statement, and it works as one would expect. It’s untenable to support this behavior when this PEP is active." However, I agree that the behavior described here is unfortunate. Here's a possible way to fix it: In module and class scopes, assign each annotated assignment an identifier that is unique within that scope (e.g., a small integer starting at 0). Maintain an internal set (hidden to the user) that represents the assignments that have been executed. Add to the set whenever an annotated assignment is executed. Inside the generated Something like this: class SchrödingersCat:
__executed_set = set()
cat: CatState.alive = CatState.alive
__executed_set.add(0)
if random.choice((True, False)):
cat: CatState.dead = CatState.dead
__executed_set.add(1)
def __annotate__(self, format):
if format > 2: raise NotImplementedError
annos = {}
if 0 in __executed_set:
annos["cat"] = CatState.alive
if 1 in __executed_set:
annos["cat"] = CatState.dead
return annos I believe we could optimize away the |
From the pep
I don't think the assumptions are correct, since it includes It's also wrong that type checker don't support such features. They certainly do then using A quick glance at https://github.com/search?q=%22++++if+TYPE_CHECKING%22&type=code&p=1 also seems to indicate that such pattern is in use in at least in class definitions |
I think this will break all tools that use annotations to do runtime operations, for example dataclasses. This does not work in 3.14a5 from dataclasses import dataclass
@dataclass
class Foo:
a: int
b: str
if False:
c: str
print(Foo(1, "2")) raises
Also it breaks other popular libreries. For example pydantic<2 does not even import in python 3.14:
This error if due to the I can't install pydantic>=2 since it has compile errors, but I would expect it to behave like dataclasses. |
@JelleZijlstra if we could have a simple flag on get_annotations() that simply says, "honor_conditionals=True" or something like that, this will allow libraries like pydantic / SQLAlchemy and others to not have to re-think our entire development model. |
@JelleZijlstra in reading pep-649 / pep-749 I am not seeing the correct way that what you propose in #130881 (comment) should be done:
from typing import TYPE_CHECKING
class MyClass:
somevalue: str
if TYPE_CHECKING:
someothervalue: int
# this method is never called
def __annotate__(self, format):
return {"somevalue": str}
import annotationlib
assert annotationlib.get_annotations(MyClass) == {"somevalue": str}
|
Overall I think this boils down to the fact that in python up until this pep untaken branches had no runtime side-effects. This pep introduces side-effects for untaken branches that I think will be hugely surprising for the vast majority of python developers |
That's never been true: Python 3.11.4 (main, Sep 30 2023, 10:54:38) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> X = 1
>>> def f():
... if False:
... x = 2
... print(x)
...
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in f
UnboundLocalError: cannot access local variable 'x' where it is not associated with a value
>>> This situation is very analogous to that, I think. That doesn't change the fact, however, that it's obviously in nobody's interests to release a version of CPython that breaks SQLAlchemy. We'll obviously take the issue seriously and see what we can do about it. No need to keep repeating the same points over and over :-) Thanks so much for reporting the issue! This is exactly why these early CPython alphas exist: so that we can discover problems like this. |
Indeed forgot about this case. |
on the SQLAlchemy side I've proposed two workarounds, one for end-user TYPE_CHECKING recipes we will provide an "ignore" sentinel like this, the existing use case looks like: class MyMappedClass(Base):
# ...
if TYPE_CHECKING:
things: Mapped[list[Stuff]]
# ... later
MyMappedClass.things = relationship(<complicated arguments that can't be inline in the class>) the user wants to apply from sqlalchemy.orm import ignore_mapped_attr
class MyMappedClass(Base):
# ...
if TYPE_CHECKING:
things: Mapped[list[Stuff]]
else:
things = ignore_mapped_attr()
# ... later
MyMappedClass.things = relationship(<complicated arguments that can't be inline in the class>) so while SQLAlchemy declarative will get the For our test suite, we have conditional suites that are just exercising many different syntaxes of annotations to test that SQLAlchemy can interpret them: # unit test code that uses pytest parameterize to provide different cases
class A(Base):
__tablename__ = "a"
id: Mapped[int] = mapped_column(primary_key=True)
data: Mapped[str] = mapped_column()
if brackets.oneset:
if option.not_optional:
json: Mapped[Dict[str, Decimal]] = mapped_column() # type: ignore # noqa: E501
elif option.optional:
json: Mapped[Optional[Dict[str, Decimal]]] = mc
elif option.optional_fwd_ref:
json: Mapped["Optional[Dict[str, Decimal]]"] = mc
# many more conditionals etc.. for these we have to write out the full "A" class each time for each conditional, which makes the test more verbose. This is a handful of test modules where we are trying to exercise parsing of many types of annotation and how they come in for different python versions, so hoping this is manageable. mostly I'm concerned with cognitively being able to catch suites like this which can silently fail to exercise the tests for the previous conditionals that are overwritten. seeing conditionals in source code and having to keep in one's mind that "part of this code will be not subject to the conditionals as though they weren't written, and this other part will be" is just unintuitive and strange. |
Just for fun I managed to make the interpreter run arbitrary code from in file def __getattr__(name):
if name == "MyClass":
print('hi there')
return int
raise AttributeError(f"module {__name__} has no attribute {name}") then in another file import dataclasses
import other
@dataclasses.dataclass
class Foo:
a: int
if False:
b: other.MyClass executing this prints |
I am going to try to implement the fix proposed in my previous comment. Let's keep the discussion focused on how to fix this; we don't need more examples. |
I proposed a fix: #130935. If you're interested in this issue, please try it out and let me know if you encounter any problems. |
ill see if i can try it out |
OK that seems to fix the conditional issue. Something else changed in 3.14 since a5 with Unions but I came up with a workaround for that. |
Can this issue be closed now #130935 has been merged? |
Changes to the test suite to accommodate Python 3.14 as of version 3.14.0b1 Originally this included a major breaking change to how python 3.14 implemented :pep:`649`, however this was resolved by [1]. As of a7, greenlet is skipped due to issues in a7 and later b1 in [2]. 1. the change to rewrite all conditionals in annotation related tests is reverted. 2. test_memusage needed an explicit set_start_method() call so that it can continue to use plain fork 3. unfortunately at the moment greenlet has to be re-disabled for 3.14. 4. Changes to tox overall, remove pysqlcipher which hasn't worked in years, etc. 5. we need to support upcoming typing-extensions also, install the beta 6. 3.14.0a7 introduces major regressions to our runtime typing utilities, unfortunately, it's not clear if these can be resolved 7. for 3.14.0b1, we have to vendor get_annotations to work around [3] [1] python/cpython#130881 [2] python-greenlet/greenlet#440 [3] python/cpython#133684 py314: yes Fixes: #12405 References: #12399 Change-Id: I8715d02fae599472dd64a2a46ccf8986239ecd99
Changes to the test suite to accommodate Python 3.14 as of version 3.14.0b1 Originally this included a major breaking change to how python 3.14 implemented :pep:`649`, however this was resolved by [1]. As of a7, greenlet is skipped due to issues in a7 and later b1 in [2]. 1. the change to rewrite all conditionals in annotation related tests is reverted. 2. test_memusage needed an explicit set_start_method() call so that it can continue to use plain fork 3. unfortunately at the moment greenlet has to be re-disabled for 3.14. 4. Changes to tox overall, remove pysqlcipher which hasn't worked in years, etc. 5. we need to support upcoming typing-extensions also, install the beta 6. 3.14.0a7 introduces major regressions to our runtime typing utilities, unfortunately, it's not clear if these can be resolved 7. for 3.14.0b1, we have to vendor get_annotations to work around [3] [1] python/cpython#130881 [2] python-greenlet/greenlet#440 [3] python/cpython#133684 py314: yes Fixes: #12405 References: #12399 Change-Id: I8715d02fae599472dd64a2a46ccf8986239ecd99
Bug report
Bug description:
using a conditional in a class definition like
if TYPE_CHECKING
whereTYPE_CHECKING
is False, or any kind of false conditional, seems to be ignored when types are evaluated under non-future annotations mode:this seems to be something that might have been done with some intention, which is extremely troubling as this would be a huge showstopper for SQLAlchemy if these conditionals are no longer honored at runtime. A similar example
calling upon
__annotations__
seems to be the trigger that makes the above fail:however this appears to be some very strange quantum-physics type of evaluation that isn't actually running Python fully; below, the "someothervalue" type is evaluted, but not the value function assigned!
This is an enormous and strange behavioral change that is really going to make things extremely difficult for SQLAlchemy, starting with we will have to rewrite
thousands(OK, it turned out to be "dozens") of lines of test suite code into a much bigger exploded form and also a lot of end-user use cases we've defined using TYPE_CHECKING blocks aren't going to work anymore, it's not clear how much further this change will go.I suppose if the whole thing boils down to what
__annotations__
does, and we can call upon something like__annotations_but_not_false_blocks__
, that could save us, but overall I'm really hoping this is just a bad dream that we can wake up from.CPython versions tested on:
3.14
Operating systems tested on:
Linux
Linked PRs
The text was updated successfully, but these errors were encountered: