[ty] fix the divergence of recursive inference due to ambiguous overload#25548
Conversation
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 92.13%. The percentage of expected errors that received a diagnostic held steady at 87.18%. The number of fully passing files held steady at 92/134. |
Memory usage reportSummary
Significant changesClick to expand detailed breakdownsphinx
prefect
trio
flake8
|
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-argument-type |
0 | 0 | 2 |
invalid-return-type |
0 | 0 | 1 |
not-iterable |
0 | 0 | 1 |
unsupported-operator |
0 | 0 | 1 |
| Total | 0 | 0 | 5 |
Raw diff:
colour (https://github.com/colour-science/colour)
- colour/notation/munsell/centore2014.py:329:12 error[invalid-return-type] Return type does not match returned value: expected `tuple[tuple[tuple[int | float, int | float], int | float], ...]`, found `tuple[tuple[tuple[Unknown, Unknown, Unknown], int | float], ...]`
+ colour/notation/munsell/centore2014.py:329:12 error[invalid-return-type] Return type does not match returned value: expected `tuple[tuple[tuple[int | float, int | float], int | float], ...]`, found `tuple[tuple[Unknown, Unknown], ...] | tuple[tuple[tuple[Unknown, Unknown, Unknown], int | float], ...]`
dd-trace-py (https://github.com/DataDog/dd-trace-py)
- ddtrace/vendor/ply/lex.py:319:40 error[not-iterable] Object of type `None | Unknown | list[tuple[Unknown, Unknown]]` may not be iterable
+ ddtrace/vendor/ply/lex.py:319:40 error[not-iterable] Object of type `None | Unknown | list[Unknown] | list[tuple[Unknown, Unknown]]` may not be iterable
- ddtrace/vendor/ply/yacc.py:3159:42 error[invalid-argument-type] Argument to function `getsourcefile` is incorrect: Expected `ModuleType | type[Any] | ((...) -> Any) | ... omitted 3 union elements`, found `ModuleType | None`
+ ddtrace/vendor/ply/yacc.py:3159:42 error[invalid-argument-type] Argument to function `getsourcefile` is incorrect: Expected `ModuleType | type[Any] | ((...) -> Any) | ... omitted 3 union elements`, found `Unknown | ModuleType | None`
sympy (https://github.com/sympy/sympy)
- sympy/matrices/sparse.py:202:49 error[unsupported-operator] Operator `>=` is not supported between objects of type `(Expr & ~AlwaysFalsy) | (int & ~AlwaysFalsy) | (Unknown & ~AlwaysFalsy)` and `None | Unknown`
+ sympy/matrices/sparse.py:202:49 error[unsupported-operator] Operator `>=` is not supported between objects of type `(Unknown & ~AlwaysFalsy) | (Expr & ~AlwaysFalsy) | (int & ~AlwaysFalsy)` and `None | Unknown`
- sympy/physics/secondquant.py:1883:38 error[invalid-argument-type] Argument to constructor `Mul.__new__` is incorrect: Expected `Expr | int | float | complex`, found `Basic | Unknown`
+ sympy/physics/secondquant.py:1883:38 error[invalid-argument-type] Argument to constructor `Mul.__new__` is incorrect: Expected `Expr | int | float | complex`, found `Unknown | Basic`There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa953aeadc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| reveal_type(NestedListsConcat().x) # revealed: list[int] | list[Divergent] | Unknown | ||
| reveal_type(NestedListsConcat().y) # revealed: list[int] | list[Divergent] | Unknown | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Could you say more about what overload is failing to resolve here? At first I thought it was something to do with the function f, but Codex deduces that it's actually these overloads of list.__add__? (lol @ the comment above those)
# Overloading looks unnecessary, but is needed to work around complex mypy problems
@overload
def __add__(self, value: list[_T], /) -> list[_T]:
"""Return self+value."""
@overload
def __add__(self, value: list[_S], /) -> list[_S | _T]: ...There was a problem hiding this comment.
The core of MRE is not method definition, but implicit recursive attribute type inference (self.x = [self.x] + []).
The binary operation isn't the source of the bug; the same hang occurred when using list.__add__(). Also, the hang disappears if you put a literal or something in the list on the right-hand side. From this, we can see that the root of the bug is indeed a failure in overload resolution (of the generically defined list.__add__).
| let self_degraded_by_overload = | ||
| any_over_type(db, self, false, |ty| { | ||
| matches!(ty, Type::Dynamic(DynamicType::AmbiguousOverload)) | ||
| }) && !any_over_type(db, self, false, |ty| ty.is_divergent()) |
There was a problem hiding this comment.
Is it guaranteed that we only need this new rule when the current cycle's type contains no Divergent? Or is it possible that it could contain Divergent in one spot, while also suffering from this overload resolution issue in another spot?
There was a problem hiding this comment.
The responsibility of cycle_normalized is to guarantee the convergence of fixed-point iterations. If Divergent is properly included in the current iteration type, there is no problem in terms of convergence.
If the type of a previous iteration contained Divergent, but the current type does not, this may represent the progress of sound type inference, but it may also represent a divergence of fixed-point iteration due to the loss of Divergent (this is a case where the language feature itself lacks monotonicity and can lead to a reversal of type inference precision).
Simply unioning the two types in such cases was the behavior in the early stages of cycle_normalized. It was found that this ignored the case of "progress of sound type inference" and included "polluted types from previous iterations", leading to the solution in #23563.
However, the problem in astral-sh/ty#3614 is that the cases for unioning are now too narrow, missing "divergence due to the loss of Divergent". It is difficult to determine this simply from the type's shape, because types generally do not record the inference process they went through. However, using variants of DynamicType allows us to embed certain type inference failures. That's why I added DynamicType::AmbiguousOverload.
In principle, regression can occur in ways other than overload (although I couldn't think of any specific scenarios for that), but even in that case, the same treatment as this time would likely be applicable.
Co-authored-by: Jack O'Connor <[email protected]>
Summary
Fixes astral-sh/ty#3614
As already mentioned in the issue, the problem is that when the mechanism introduced in #23563 is combined with overloading, the convergence of fixed-point iterations is no longer guaranteed.
Therefore, even within "tainted cycles," there are cases where the current type and the previous type must be unioned. However, doing so in all cases would bring back the problem resolved in #23563.
To address this, we introduce a new dynamic type variant called
DynamicType::AmbiguousOverload, which we use to determine whether unioning is necessary in a special case.Test Plan
new mdtest