Thanks to visit codestin.com
Credit goes to github.com

Skip to content

typing.get_type_hints Can Return Incorrect Type for Annotated Metadata #112618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cmg2146 opened this issue Dec 2, 2023 · 6 comments
Closed

typing.get_type_hints Can Return Incorrect Type for Annotated Metadata #112618

cmg2146 opened this issue Dec 2, 2023 · 6 comments
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error

Comments

@cmg2146
Copy link

cmg2146 commented Dec 2, 2023

Bug report

Bug description:

When running the following example:

from typing import Annotated, get_type_hints

class MessageCode(str):
    pass

class DisplayName(str):
    pass

class Message:
    field_a: Annotated[str, MessageCode("A")]

class Form:
    box_a: Annotated[str, DisplayName("A")]

hints = get_type_hints(Message, include_extras=True)
metadata = hints["field_a"].__metadata__[0]
print(f"MessageCode metadata type: {type(metadata)}")

hints = get_type_hints(Form, include_extras=True)
metadata = hints["box_a"].__metadata__[0]
print(f"DisplayName metadata type: {type(metadata)}")

the output will be:

MessageCode metadata type: <class '__main__.MessageCode'>
DisplayName metadata type: <class '__main__.MessageCode'>

when it should be:

MessageCode metadata type: <class '__main__.MessageCode'>
DisplayName metadata type: <class '__main__.DisplayName'>

This issue seems to occur only when:

  • the metadata subclasses an immutable type
  • when multiple instances of such metadata appear in different class definitions and they have the same value

CPython versions tested on:

3.9

Operating systems tested on:

Linux, Windows

Linked PRs

@cmg2146 cmg2146 added the type-bug An unexpected behavior, bug, or error label Dec 2, 2023
@cmg2146 cmg2146 changed the title `typing.get_type_hints" Can Return Incorrect Type for Annotated Metadata typing.get_type_hints Can Return Incorrect Type for Annotated Metadata Dec 2, 2023
@Eclips4
Copy link
Member

Eclips4 commented Dec 2, 2023

Hello!
This happening due to realization of typing._tp_cache(it's used to decorate the __getitem__ method of _SpecialForm to reduce the number of unnecessary calls when objects are identical). Hashes of DisplayName("A") and MessageCode("A") are identical, and they're also identical (DisplayName("A") == MessageCode("A") evaluates to True).
So this example will probably help you understand the problem more deeply:

from typing import Annotated, get_type_hints

class Parent:
    def __init__(self, num: int) -> None:
        self.num = num

class Foo(Parent):
    # Realization of hashable protocol: https://docs.python.org/3.12/glossary.html#term-hashable
    def __eq__(self, other):
        if not isinstance(other, Parent):
            return NotImplemented
        return self.num == other.num
    def __hash__(self):
        return hash(self.num)

class Bar(Parent):  # Same realization of __eq__ and __hash__ as in Foo but semantically it's another class :)
    def __eq__(self, other):
        if not isinstance(other, Parent):
            return NotImplemented
        return self.num == other.num
    def __hash__(self):
        return hash(self.num)

class Form:
    box_a: Annotated[str, Foo(1)]

class Message:
    field_a: Annotated[str, Bar(1)]

hints = get_type_hints(Message, include_extras=True)
metadata = hints["field_a"].__metadata__[0]
print(f"MessageCode metadata type: {type(metadata)}")

hints = get_type_hints(Form, include_extras=True)
metadata = hints["box_a"].__metadata__[0]
print(f"DisplayName metadata type: {type(metadata)}")

Output:

MessageCode metadata type: <class '__main__.Foo'>
DisplayName metadata type: <class '__main__.Foo'>

Honestly, I don't know, should it be fixed or not, this behaviour works for years (as I see, from 2018 year).
cc @AlexWaygood @JelleZijlstra

@sobolevn
Copy link
Member

sobolevn commented Dec 2, 2023

I think that there's an easy fix for that which also keeps all tests green:

>>> hints = get_type_hints(Message, include_extras=True)
>>> metadata = hints["field_a"].__metadata__[0]
>>> print(f"MessageCode metadata type: {type(metadata)}")
MessageCode metadata type: <class '__main__.MessageCode'>
>>> 
>>> hints = get_type_hints(Form, include_extras=True)
>>> metadata = hints["box_a"].__metadata__[0]
>>> print(f"DisplayName metadata type: {type(metadata)}")
DisplayName metadata type: <class '__main__.DisplayName'>
(.venv) ~/Desktop/cpython2  main ✗                                                        
» ./python.exe -m test test_typing
Using random seed: 342322226
0:00:00 load avg: 1.85 Run 1 test sequentially
0:00:00 load avg: 1.85 [1/1] test_typing

== Tests result: SUCCESS ==

1 test OK.

Total duration: 402 ms
Total tests: run=613 skipped=1
Total test files: run=1/1
Result: SUCCESS

Patch:

» git patch
diff --git Lib/typing.py Lib/typing.py
index b3af701f8d5..a7fa763c198 100644
--- Lib/typing.py
+++ Lib/typing.py
@@ -497,6 +497,11 @@ def __getitem__(self, parameters):
         return self._getitem(self, *parameters)
 
 
+class _AnnotatedSpecialForm(_SpecialForm, _root=True):
+    def __getitem__(self, parameters):
+        return self._getitem(self, parameters)
+
+
 class _AnyMeta(type):
     def __instancecheck__(self, obj):
         if self is Any:
@@ -2005,7 +2010,8 @@ def __mro_entries__(self, bases):
         return (self.__origin__,)
 
 
-@_SpecialForm
+@_AnnotatedSpecialForm
+@_tp_cache(typed=True)
 def Annotated(self, params):
     """Add context-specific metadata to a type.

I will experiment a bit with it and decide whether or not I will open a PR for it :)

sobolevn added a commit to sobolevn/cpython that referenced this issue Dec 2, 2023
@AlexWaygood AlexWaygood added the stdlib Python modules in the Lib dir label Dec 2, 2023
@AlexWaygood
Copy link
Member

AlexWaygood commented Dec 2, 2023

Thanks for the bug report @cmg2146! Here's a more minimal repro:

Python 3.13.0a2+ (heads/main-dirty:8f71b349de, Nov 27 2023, 22:42:08) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Annotated
>>> Annotated[str, 1]
typing.Annotated[str, 1]
>>> Annotated[str, True]
typing.Annotated[str, 1]

@Eclips4, I think this is definitely worth fixing :) I think Annotated wasn't used much for the first few years of its existence, but there's now a growing number of third-party libraries that are using it more, so it "makes sense" that this bug might have gone unnoticed until now.

@AlexWaygood AlexWaygood added 3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes labels Dec 2, 2023
AlexWaygood added a commit that referenced this issue Dec 2, 2023
AlexWaygood added a commit to AlexWaygood/cpython that referenced this issue Dec 2, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Dec 3, 2023
AlexWaygood added a commit that referenced this issue Dec 3, 2023
…12628) (#112633)

[3.12] gh-112618: Make Annotated cache typed (GH-112619) (GH-112628)
(cherry picked from commit 2a378ca)

Co-authored-by: Alex Waygood <[email protected]>
@AlexWaygood
Copy link
Member

Fixed on the main branch, the 3.12 branch and the 3.11 branch. The fix will be included in the next patch release for each.

Thanks again @cmg2146 for the report, and thanks @sobolevn for the fix!

@cmg2146
Copy link
Author

cmg2146 commented Dec 3, 2023

Fixed on the main branch, the 3.12 branch and the 3.11 branch. The fix will be included in the next patch release for each.

Thanks again @cmg2146 for the report, and thanks @sobolevn for the fix!

Thanks for fixing this! Do you know if this fix will be ported to typing_extensions?

@JelleZijlstra
Copy link
Member

I'd accept a PR adding this to typing-extensions.

aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 only security fixes 3.13 bugs and security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

5 participants