Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[ty] fix the divergence of recursive inference due to ambiguous overload#25548

Merged
mtshiba merged 3 commits into
astral-sh:mainfrom
mtshiba:fix-ambiguous-divergent-overload-hang
Jun 5, 2026
Merged

[ty] fix the divergence of recursive inference due to ambiguous overload#25548
mtshiba merged 3 commits into
astral-sh:mainfrom
mtshiba:fix-ambiguous-divergent-overload-hang

Conversation

@mtshiba

@mtshiba mtshiba commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes astral-sh/ty#3614

As already mentioned in the issue, the problem is that when the mechanism introduced in #23563 is combined with overloading, the convergence of fixed-point iterations is no longer guaranteed.
Therefore, even within "tainted cycles," there are cases where the current type and the previous type must be unioned. However, doing so in all cases would bring back the problem resolved in #23563.

To address this, we introduce a new dynamic type variant called DynamicType::AmbiguousOverload, which we use to determine whether unioning is necessary in a special case.

Test Plan

new mdtest

@mtshiba mtshiba added the ty Multi-file analysis & type inference label Jun 2, 2026
@astral-sh-bot

astral-sh-bot Bot commented Jun 2, 2026

Copy link
Copy Markdown

Typing conformance results

No changes detected ✅

Current numbers
The percentage of diagnostics emitted that were expected errors held steady at 92.13%. The percentage of expected errors that received a diagnostic held steady at 87.18%. The number of fully passing files held steady at 92/134.

@astral-sh-bot

astral-sh-bot Bot commented Jun 2, 2026

Copy link
Copy Markdown

Memory usage report

Summary

Project Old New Diff Outcome
sphinx 243.95MB 244.03MB +0.03% (87.07kB)
prefect 668.07MB 668.11MB +0.01% (41.95kB)
trio 100.78MB 100.81MB +0.03% (29.02kB)
flake8 39.38MB 39.39MB +0.02% (7.46kB)

Significant changes

Click to expand detailed breakdown

sphinx

Name Old New Diff Outcome
StaticClassLiteral<'db>::try_mro_ 2.55MB 2.56MB +0.36% (9.32kB)
when_constraint_set_assignable_to_owned_impl 2.26MB 2.26MB +0.27% (6.35kB)
Type<'db>::apply_specialization_::interned_arguments 1.79MB 1.80MB +0.34% (6.25kB)
Type<'db>::apply_specialization_ 1.78MB 1.78MB +0.29% (5.36kB)
Specialization 1.29MB 1.29MB +0.36% (4.75kB)
infer_expression_types_impl 21.51MB 21.51MB +0.02% (4.00kB)
Type<'db>::member_lookup_with_policy_ 7.30MB 7.30MB +0.05% (3.79kB)
is_redundant_with_impl::interned_arguments 1.14MB 1.14MB +0.31% (3.61kB)
FunctionType 3.18MB 3.18MB +0.10% (3.28kB)
UnionType 649.89kB 652.88kB +0.46% (2.98kB)
Type<'db>::class_member_with_policy_ 7.96MB 7.97MB +0.03% (2.79kB)
Type<'db>::member_lookup_with_policy_::interned_arguments 2.74MB 2.74MB +0.10% (2.74kB)
CallableType 1.37MB 1.37MB +0.19% (2.73kB)
when_constraint_set_assignable_to_owned_impl::interned_arguments 710.45kB 712.85kB +0.34% (2.41kB)
FunctionType<'db>::signature_ 2.60MB 2.60MB +0.09% (2.34kB)
... 34 more

prefect

Name Old New Diff Outcome
is_redundant_with_impl::interned_arguments 2.34MB 2.34MB +0.26% (6.27kB)
is_redundant_with_impl 2.10MB 2.10MB +0.17% (3.62kB)
StaticClassLiteral<'db>::try_mro_ 5.21MB 5.22MB +0.06% (3.34kB)
Type<'db>::member_lookup_with_policy_ 17.29MB 17.29MB +0.02% (2.98kB)
UnionType 1.35MB 1.36MB +0.21% (2.97kB)
infer_expression_types_impl 60.28MB 60.28MB +0.00% (2.86kB)
infer_definition_types 83.15MB 83.15MB +0.00% (2.40kB)
IntersectionType 995.12kB 997.42kB +0.23% (2.30kB)
Type<'db>::member_lookup_with_policy_::interned_arguments 6.41MB 6.41MB +0.03% (2.03kB)
infer_expression_type_impl 8.41MB 8.41MB +0.02% (1.90kB)
Type<'db>::apply_specialization_::interned_arguments 3.58MB 3.58MB +0.04% (1.33kB)
Specialization 2.58MB 2.58MB +0.05% (1.33kB)
Type<'db>::apply_specialization_ 3.67MB 3.67MB +0.03% (1.01kB)
all_narrowing_constraints_for_expression 8.45MB 8.45MB +0.01% (888.00B)
all_negative_narrowing_constraints_for_expression 8.28MB 8.28MB +0.01% (840.00B)
... 25 more

trio

Name Old New Diff Outcome
StaticClassLiteral<'db>::try_mro_ 853.30kB 855.16kB +0.22% (1.85kB)
FunctionType 1.40MB 1.40MB +0.12% (1.78kB)
Type<'db>::apply_specialization_ 647.60kB 649.36kB +0.27% (1.76kB)
Type<'db>::apply_specialization_::interned_arguments 634.92kB 636.56kB +0.26% (1.64kB)
Type<'db>::class_member_with_policy_ 2.02MB 2.02MB +0.08% (1.62kB)
when_constraint_set_assignable_to_owned_impl 587.05kB 588.64kB +0.27% (1.59kB)
Type<'db>::try_call_dunder_get_ 1.31MB 1.31MB +0.11% (1.49kB)
infer_expression_types_impl 6.89MB 6.89MB +0.02% (1.28kB)
Type<'db>::class_member_with_policy_::interned_arguments 1.10MB 1.10MB +0.11% (1.22kB)
FunctionType<'db>::signature_ 1.13MB 1.13MB +0.11% (1.22kB)
CallableType 685.86kB 687.02kB +0.17% (1.16kB)
Type<'db>::member_lookup_with_policy_ 1.95MB 1.95MB +0.06% (1.13kB)
Specialization 470.06kB 471.16kB +0.23% (1.09kB)
is_redundant_with_impl::interned_arguments 223.27kB 224.21kB +0.42% (968.00B)
UnionType 150.14kB 151.02kB +0.58% (896.00B)
... 25 more

flake8

Name Old New Diff Outcome
StaticClassLiteral<'db>::try_mro_ 353.39kB 355.08kB +0.48% (1.69kB)
when_constraint_set_assignable_to_owned_impl 243.75kB 244.66kB +0.37% (932.00B)
Specialization 179.52kB 180.17kB +0.37% (672.00B)
Type<'db>::apply_specialization_::interned_arguments 221.88kB 222.42kB +0.25% (560.00B)
Type<'db>::try_call_dunder_get_ 378.23kB 378.72kB +0.13% (508.00B)
Type<'db>::apply_specialization_ 217.47kB 217.96kB +0.22% (500.00B)
when_constraint_set_assignable_to_owned_impl::interned_arguments 79.23kB 79.66kB +0.54% (440.00B)
GenericAlias 79.66kB 80.02kB +0.44% (360.00B)
StaticClassLiteral<'db>::try_mro_::interned_arguments 80.09kB 80.44kB +0.44% (360.00B)
Type<'db>::member_lookup_with_policy_::interned_arguments 232.98kB 233.29kB +0.13% (312.00B)
Type<'db>::member_lookup_with_policy_ 574.88kB 575.14kB +0.04% (264.00B)
code_generator_of_static_class 94.27kB 94.50kB +0.24% (236.00B)
Type<'db>::class_member_with_policy_ 597.41kB 597.63kB +0.04% (232.00B)
Type<'db>::class_member_with_policy_::interned_arguments 322.56kB 322.66kB +0.03% (104.00B)
Type<'db>::try_call_dunder_get_::interned_arguments 85.11kB 85.21kB +0.12% (104.00B)
... 5 more

@astral-sh-bot

astral-sh-bot Bot commented Jun 2, 2026

Copy link
Copy Markdown

ecosystem-analyzer results

Lint rule Added Removed Changed
invalid-argument-type 0 0 2
invalid-return-type 0 0 1
not-iterable 0 0 1
unsupported-operator 0 0 1
Total 0 0 5

Raw diff:

colour (https://github.com/colour-science/colour)
- colour/notation/munsell/centore2014.py:329:12 error[invalid-return-type] Return type does not match returned value: expected `tuple[tuple[tuple[int | float, int | float], int | float], ...]`, found `tuple[tuple[tuple[Unknown, Unknown, Unknown], int | float], ...]`
+ colour/notation/munsell/centore2014.py:329:12 error[invalid-return-type] Return type does not match returned value: expected `tuple[tuple[tuple[int | float, int | float], int | float], ...]`, found `tuple[tuple[Unknown, Unknown], ...] | tuple[tuple[tuple[Unknown, Unknown, Unknown], int | float], ...]`

dd-trace-py (https://github.com/DataDog/dd-trace-py)
- ddtrace/vendor/ply/lex.py:319:40 error[not-iterable] Object of type `None | Unknown | list[tuple[Unknown, Unknown]]` may not be iterable
+ ddtrace/vendor/ply/lex.py:319:40 error[not-iterable] Object of type `None | Unknown | list[Unknown] | list[tuple[Unknown, Unknown]]` may not be iterable
- ddtrace/vendor/ply/yacc.py:3159:42 error[invalid-argument-type] Argument to function `getsourcefile` is incorrect: Expected `ModuleType | type[Any] | ((...) -> Any) | ... omitted 3 union elements`, found `ModuleType | None`
+ ddtrace/vendor/ply/yacc.py:3159:42 error[invalid-argument-type] Argument to function `getsourcefile` is incorrect: Expected `ModuleType | type[Any] | ((...) -> Any) | ... omitted 3 union elements`, found `Unknown | ModuleType | None`

sympy (https://github.com/sympy/sympy)
- sympy/matrices/sparse.py:202:49 error[unsupported-operator] Operator `>=` is not supported between objects of type `(Expr & ~AlwaysFalsy) | (int & ~AlwaysFalsy) | (Unknown & ~AlwaysFalsy)` and `None | Unknown`
+ sympy/matrices/sparse.py:202:49 error[unsupported-operator] Operator `>=` is not supported between objects of type `(Unknown & ~AlwaysFalsy) | (Expr & ~AlwaysFalsy) | (int & ~AlwaysFalsy)` and `None | Unknown`
- sympy/physics/secondquant.py:1883:38 error[invalid-argument-type] Argument to constructor `Mul.__new__` is incorrect: Expected `Expr | int | float | complex`, found `Basic | Unknown`
+ sympy/physics/secondquant.py:1883:38 error[invalid-argument-type] Argument to constructor `Mul.__new__` is incorrect: Expected `Expr | int | float | complex`, found `Unknown | Basic`

Full report with detailed diff (timing results)

@mtshiba mtshiba marked this pull request as ready for review June 2, 2026 15:43
@astral-sh-bot astral-sh-bot Bot requested a review from oconnor663 June 2, 2026 15:43

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa953aeadc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ty_python_semantic/src/types/call/bind.rs
reveal_type(NestedListsConcat().x) # revealed: list[int] | list[Divergent] | Unknown
reveal_type(NestedListsConcat().y) # revealed: list[int] | list[Divergent] | Unknown
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you say more about what overload is failing to resolve here? At first I thought it was something to do with the function f, but Codex deduces that it's actually these overloads of list.__add__? (lol @ the comment above those)

    # Overloading looks unnecessary, but is needed to work around complex mypy problems
    @overload
    def __add__(self, value: list[_T], /) -> list[_T]:
        """Return self+value."""
    @overload
    def __add__(self, value: list[_S], /) -> list[_S | _T]: ...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core of MRE is not method definition, but implicit recursive attribute type inference (self.x = [self.x] + []).
The binary operation isn't the source of the bug; the same hang occurred when using list.__add__(). Also, the hang disappears if you put a literal or something in the list on the right-hand side. From this, we can see that the root of the bug is indeed a failure in overload resolution (of the generically defined list.__add__).

let self_degraded_by_overload =
any_over_type(db, self, false, |ty| {
matches!(ty, Type::Dynamic(DynamicType::AmbiguousOverload))
}) && !any_over_type(db, self, false, |ty| ty.is_divergent())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it guaranteed that we only need this new rule when the current cycle's type contains no Divergent? Or is it possible that it could contain Divergent in one spot, while also suffering from this overload resolution issue in another spot?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The responsibility of cycle_normalized is to guarantee the convergence of fixed-point iterations. If Divergent is properly included in the current iteration type, there is no problem in terms of convergence.

If the type of a previous iteration contained Divergent, but the current type does not, this may represent the progress of sound type inference, but it may also represent a divergence of fixed-point iteration due to the loss of Divergent (this is a case where the language feature itself lacks monotonicity and can lead to a reversal of type inference precision).
Simply unioning the two types in such cases was the behavior in the early stages of cycle_normalized. It was found that this ignored the case of "progress of sound type inference" and included "polluted types from previous iterations", leading to the solution in #23563.
However, the problem in astral-sh/ty#3614 is that the cases for unioning are now too narrow, missing "divergence due to the loss of Divergent". It is difficult to determine this simply from the type's shape, because types generally do not record the inference process they went through. However, using variants of DynamicType allows us to embed certain type inference failures. That's why I added DynamicType::AmbiguousOverload.

In principle, regression can occur in ways other than overload (although I couldn't think of any specific scenarios for that), but even in that case, the same treatment as this time would likely be applicable.

Comment thread crates/ty_python_semantic/resources/mdtest/attributes.md
@mtshiba mtshiba merged commit dced0e4 into astral-sh:main Jun 5, 2026
59 checks passed
@mtshiba mtshiba deleted the fix-ambiguous-divergent-overload-hang branch June 5, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hang caused by recursive inference & ambiguous overload resolution

2 participants