Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Python 3.12 support in 1.10.x? #9622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
4 of 15 tasks
vfazio opened this issue Jun 10, 2024 · 14 comments
Closed
4 of 15 tasks

Python 3.12 support in 1.10.x? #9622

vfazio opened this issue Jun 10, 2024 · 14 comments

Comments

@vfazio
Copy link

vfazio commented Jun 10, 2024

Initial Checks

  • I have searched GitHub for a duplicate issue and I'm sure this is something new
  • I have searched Google & StackOverflow for a solution and couldn't find anything
  • I have read and followed the docs and still think this is a bug
  • I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description

Right now, we have have users using pydantic on 3.12 despite no advertised support. Part of this is due to the python requirement being only 3.7 and without a cap.

A breaking change was introduced in 3.12.4, otherwise 1.10.15 seems to mostly play nice.

However, a number of changes are required to get tests to pass on python 3.12.

This list is non-exhaustive:

  • flake8>=6.1.0
  • pycodestyle==2.11.1
  • pyflakes==3.1.0
  • flake8-quotes>=3.3.2
  • ignore ast* warnings (or fix them)
  • pytest >=7.3.2
  • mypy>1.5
  • setuptools needs to be added as a dependency (distutils is gone and cython doesn't add it)
  • cython may need a bump for true 3.12 support however we're capped: b9a316e

A number of tests still fail, likely due to string format changes between CPython and tools

      20 failed
         - tests/test_edge_cases.py:1877 test_cython_function_untouched
         - tests/test_generics.py:573 test_partial_specification_name
         - tests/test_generics.py:1423 test_parse_generic_json
         - tests/test_types.py:2712 test_secretfield
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-plugin.ini-plugin_success.py-None]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-plugin.ini-plugin_fail.py-plugin-fail.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-plugin-strict.ini-plugin_success.py-plugin-success-strict.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-plugin-strict.ini-plugin_fail.py-plugin-fail-strict.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-default.ini-success.py-None]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-default.ini-fail1.py-fail1.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-default.ini-fail2.py-fail2.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-default.ini-plugin_success.py-plugin_success.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-default.toml-success.py-None]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-default.toml-fail1.py-fail1.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-default.toml-fail2.py-fail2.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-plugin.toml-plugin_success.py-None]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-plugin.toml-plugin_fail.py-plugin-fail.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-plugin-strict.toml-plugin_success.py-plugin-success-strict.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[pyproject-plugin-strict.toml-plugin_fail.py-plugin-fail-strict.txt]
         - tests/mypy/test_mypy.py:57 test_mypy_results[mypy-plugin-strict.ini-plugin_default_factory.py-plugin_default_factory.txt]
       9 skipped
make: *** [Makefile:69: test] Error 1

In light of #9607, I think a 1.10.16 release should be tagged, but with python capped at <3.13.

If there is a desire to add python 3.12 support, it will take some discussion and some reworking of the tests. If we want python 3.12 support in 1.10.x, a subsequent series of commits can add that support. If we do not, we can tag 1.10.17 with python <3.12.

This would allow users to pin their dependency on 1.10.16 if they need an immediate fix. This will also force a migration to pydantic v2.

But these are pre-caffinated ruminations, so open to thoughts.

@sydney-runkle

Example Code

No response

Python, Pydantic & OS Version

python 3.12, pydantic 1.10.15

Affected Components

@vfazio vfazio added bug V1 Bug related to Pydantic V1.X pending Is unconfirmed labels Jun 10, 2024
@vfazio vfazio changed the title Advertise python 3.12 support Python 3.12 support in 1.10.x? Jun 10, 2024
@sydney-runkle
Copy link
Contributor

In light of #9607, I think a 1.10.16 release should be tagged, but with python capped at <3.13.

Agreed, I can work on this tomorrow once we get a fix out in pydantic-core as well.

If there is a desire to add python 3.12 support, it will take some discussion and some reworking of the tests. If we want python 3.12 support in 1.10.x, a subsequent series of commits can add that support.

Let's try for this!

But these are pre-caffinated ruminations, so open to thoughts.

Haha, thanks for your work on this. Looking forward to collaborating more :).

@vfazio
Copy link
Author

vfazio commented Jun 10, 2024

In light of #9607, I think a 1.10.16 release should be tagged, but with python capped at <3.13.

Agreed, I can work on this tomorrow once we get a fix out in pydantic-core as well.

If there is a desire to add python 3.12 support, it will take some discussion and some reworking of the tests. If we want python 3.12 support in 1.10.x, a subsequent series of commits can add that support.

Let's try for this!

But these are pre-caffinated ruminations, so open to thoughts.

Haha, thanks for your work on this. Looking forward to collaborating more :).

@sydney-runkle pydantic/pydantic-core#1299 this ? or are other fixes necessary?

@sydney-runkle
Copy link
Contributor

@sydney-runkle sydney-runkle added feature request and removed bug V1 Bug related to Pydantic V1.X pending Is unconfirmed labels Jun 11, 2024
@vfazio
Copy link
Author

vfazio commented Jun 12, 2024

So... I don't think we can get compiled pydantic without going to Cython 3... We basically need cython/cython@03c498d and maybe other things, but that is at least what's yelling at me now.

pydantic/__init__.c:1125:32: error: β€˜_PyCFrame’ has no member named β€˜use_tracing’
 1125 |      (unlikely((tstate)->cframe->use_tracing) &&\
      |                                ^~
pydantic/__init__.c:950:43: note: in definition of macro β€˜unlikely’
  950 |   #define unlikely(x) __builtin_expect(!!(x), 0)
      |                                           ^
pydantic/__init__.c:1280:19: note: in expansion of macro β€˜__Pyx_IsTracing’
 1280 |               if (__Pyx_IsTracing(tstate, 0, 0) && tstate->c_tracefunc && __pyx_frame->f_trace) {\
      |                   ^~~~~~~~~~~~~~~
pydantic/__init__.c:2642:3: note: in expansion of macro β€˜__Pyx_TraceLine’
 2642 |   __Pyx_TraceLine(16,0,__PYX_ERR(0, 16, __pyx_L1_error))

Of course, after compiling it, I'm getting additional type errors. I'm able to coerce most of them, but jumping to cython3 was called out as "not happening" per

# Fixed in Cython 3 and Pydantic v1 won't support Cython 3.

Plus, a lot of NameErrors from our friend _evaluate:

        AAModel = create_model(
            'AAModel', __base__=(TestGenericModel, Generic[T]), __cls_kwargs__={'orm_mode': True}, aa=(int, Field(0))
        )
>       result = AAModel[int](aa=1)

tests/test_create_model.py:223: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pydantic/generics.py:116: in __class_getitem__
    type_hints = get_all_type_hints(cls).items()
pydantic/typing.py:80: in pydantic.typing.get_all_type_hints
    return get_type_hints(obj, globalns, localns, include_extras=True)
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:2263: in get_type_hints
    value = _eval_type(value, base_globals, base_locals, base.__type_params__)
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:415: in _eval_type
    return t._evaluate(globalns, localns, type_params, recursive_guard=recursive_guard)
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:943: in _evaluate
    self.__forward_value__ = _eval_type(
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:430: in _eval_type
    ev_args = tuple(
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:431: in <genexpr>
    _eval_type(
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:415: in _eval_type
    return t._evaluate(globalns, localns, type_params, recursive_guard=recursive_guard)
/home/vfazio/.pyenv/versions/3.12.4/lib/python3.12/typing.py:938: in _evaluate
    eval(self.__forward_code__, globalns, locals_to_pass),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Unless we skip the requirement for compiled tests on 3.12, i'm guessing "full support" wont happen.

I'm going to limit further testing until there is a discussion had.

@sydney-runkle
Copy link
Contributor

sydney-runkle commented Jun 12, 2024

@vfazio,

Ahhhh. Good find. We'll stick to that commitment to not support cython 3, so we can halt progress on the 3.12 support for now. Thanks for your investigation here!!

Folks can sort of use it unofficially without the compiled packages, given that we fixed the internal typing stuff.

Going to close this and the related PR, given your findings. Great work!

@vfazio
Copy link
Author

vfazio commented Jun 12, 2024

@vfazio,

Ahhhh. Good find. We'll stick to that commitment to not support cython 3, so we can halt progress on the 3.12 support for now. Thanks for your investigation here!!

Folks can sort of use it unofficially without the compiled packages, given that we fixed the internal typing stuff.

Going to close this and the related PR, given your findings. Great work!

@sydney-runkle i have an alternative solution i'm about to push... give me a minute

@sydney-runkle
Copy link
Contributor

Folks can sort of use it unofficially without the compiled packages, given that we fixed the internal typing stuff.

To clarify - we'll support 3.12, just not compile for 3.12. We can probably add the bound re <3.13 on the next patch release

@vfazio
Copy link
Author

vfazio commented Jun 12, 2024

We cannot use Cython 3 for 1.10.x, but we can get Cython 0.29.x to work on Python 3.12.

The thing that's "blocking" is the CI pipeline, which is requiring build-trace. AFAICT, we're only using tracing on these builds so we can confirm our coverage. Coverage on 3.7-3.11 should be sufficient to give us an idea that code is being exercised as expected.

We should be able to make build and still pass the test suite (i tested this locally), so I'm going to propose isolating 3.12 into it's own job branch to trigger a build without tracing and where a full test suite passes where coverage does not get uploaded so doesn't spoil the results.

@vfazio
Copy link
Author

vfazio commented Jun 12, 2024

My PR has the cap for <3.13 in it, i'll push in a sec.

@sydney-runkle
Copy link
Contributor

Happy to review!

@vfazio
Copy link
Author

vfazio commented Jun 12, 2024

The thing that's "blocking" is the CI pipeline, which is requiring build-trace. AFAICT, we're only using tracing on these builds so we can confirm our coverage. Coverage on 3.7-3.11 should be sufficient to give us an idea that code is being exercised as expected.

@sydney-runkle could you confirm this assumption? Do you cythonize the wheels that are generated? if so, is tracing enabled when they are built? If tracing is a requirement for cythonized wheels then it's all moot.

@michael-k
Copy link

michael-k commented Jun 13, 2024

We can probably add the bound re <3.13 on the next patch release

But why? I don't see the benefit of disallowing pip to install pydantic 1.10.17+ (unless --ignore-requires-python) on Python 3.13 while it'll happily install 1.10.16. Regardless if upper bounds are in install_requires or for Python versions, they just cause pain and confusion for users, esp. if users don't have control over pydantic's version because some other package decided to add a upper bound and pin pydantic < 2.

pydantic 1.0 works fine at least for some use cases on Python 3.13. I'm not asking you to support this, but why prevent it?

>>> import sys
>>> sys.version
'3.13.0b2 (main, Jun  7 2024, 02:27:16) [GCC 12.2.0]'
>>> import pydantic
>>> pydantic.VERSION
StrictVersion ('1.0')
>>> class User(pydantic.BaseModel):
...     id: int
...     name = 'John Doe'
...
>>> user = User(id=1)
>>> print(user.dict())
{'id': 1, 'name': 'John Doe'}

@vfazio
Copy link
Author

vfazio commented Jun 13, 2024

I don't really want to put words in their mouth or answer for them, I'm not a maintainer, but preventing it allows them to determine if support will be extended vs being strong armed into supporting it because thousands of users are able to install it and assume that if they can install it that its supposed to be supported. Its "explicit support" vs "implied support"

It may work in most usecases in 3.13. It may even pass the full test suite, though i have strong doubts on the cythonized builds working. In the end, just because it works doesn't mean its supported.

It may be nontrivial to add full 3.13 support or maybe they don't want to be on the hook for dealing with another full life cycle of a release because there may be compatibility issues between bugfix versions like we just had with 3.12.4 and they need to focus developer time elsewhere, like pydantic v2 or a new product, instead of on a pydantic 1.10.x thats in maintenance mode.

I know #9613 is going to be an issue at some point.

My 2Β’

@vfazio
Copy link
Author

vfazio commented Jun 14, 2024

Going to tear this conversation off into a different issue, because it does seem pertinent and isn't straightforward

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants