[ty] Simplify intersections of invariant generic types with Any specializations#26127
Conversation
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 94.37%. The percentage of expected errors that received a diagnostic held steady at 89.00%. The number of fully passing files held steady at 94/134. |
Memory usage reportMemory usage unchanged ✅ |
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-argument-type |
1 | 0 | 0 |
invalid-assignment |
1 | 0 | 0 |
| Total | 2 | 0 | 0 |
Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.
Raw diff:
prefect (https://github.com/PrefectHQ/prefect)
+ src/integrations/prefect-github/tests/test_graphql.py:28:66 error[invalid-argument-type] Argument to function `_subset_return_fields` is incorrect: Expected `dict[tuple[Unknown, ...], tuple[Unknown, ...]]`, found `dict[tuple[str], list[str]] | dict[Unknown, Unknown]`
spark (https://github.com/apache/spark)
+ python/pyspark/pandas/internal.py:1491:29 error[invalid-assignment] Object of type `list[str]` is not assignable to `list[tuple[Any, ...]]`
Merging this PR will not alter performance
Comparing 
Footnotes
|
ac9c8b5 to
e21d5b4
Compare
Invariant[P] & Invariant[Any] = Invariant[P]Any specializations
a5ae658 to
9cc2667
Compare
|
Hm, the ecosystem timing hit on pycryptodome seems to be persistent. Looking into it. |
dhruvmanila
left a comment
There was a problem hiding this comment.
That was a lot to go through and understand but I think I've a basic understanding of this change. Regardless, I asked Codex to review it and it gave me the following two issues. Do you mind taking a look? You might be able to judge the validity of it quicker than me. I at least think (2) is a valid concern.
- Blocker: builder.rs:105 —
class_specializationlooks through bounded type variables andNewTypewrappers. Consequently, ordinaryTypeIs[list[int]]narrowing erasesTfromT & list[int]and the identity fromListNewType & list[int], producing falseinvalid-return-typeerrors. The same reproduction passes on the base commit.- Blocker: builder.rs:130 —
is_dynamic()includesDivergent. A focused unit test showedlist[Divergent] & list[int]collapsing tolist[int], violating the requirement that divergent markers survive dynamic reduction and risking broken cycle convergence.
(2) is mentioned specifically in
ruff/crates/ty_python_semantic/src/types.rs
Lines 7535 to 7537 in 344c6d1
So, it seems like we should use is_non_divergent_dynamic instead of is_dynamic?
For (1), it gave me the following example for a NewType wrapper:
from typing import Any, NewType, TypeIs
ListId = NewType("ListId", list[Any])
def preserve_id(value: ListId) -> ListId:
if is_int_list(value):
return value # PR incorrectly sees `list[int]`, losing `ListId`
return valueAnd the following for a bounded-TypeVar:
from typing import Any, TypeIs, reveal_type
def is_int_list(value: object) -> TypeIs[list[int]]:
return isinstance(value, list) and all(isinstance(item, int) for item in value)
def preserve_type[T: list[Any]](value: T) -> T:
if is_int_list(value):
reveal_type(value)
return value
return valuead7f8a9 to
52dc1d2
Compare
Thanks. Those are valid concerns, yes. I wouldn't call them "blockers", but that's okay 😄. Both issues should be addressed now.
That behavior is pretty surprising actually. I wouldn't have expected it to resolve typevars to their upper bound. But I guess we do something similar for other type operations as well. |
f17b256 to
5effca5
Compare
|
Hmm, interesting, the perf regression is gone now |
No, it's just below the huge 50% threshold. There's still a 1.42x slowdown in pycryptodome in the latest measurement. I will take another look at this. |
5effca5 to
17f45db
Compare
Now it's gone! 😄 |
| specific: Type<'db>, | ||
| ) -> bool { | ||
| // Fast path to avoid performance regressions. | ||
| if !general.has_dynamic(db) { |
There was a problem hiding this comment.
Should this be has_non_divergent_dynamic?
There was a problem hiding this comment.
It could be (if that method would exist, or if we would implement it).. but it's not necessary. Divergent will pass this fast-path check, but it will be filtered out below. So we don't risk simplifying Divergent out of intersections.
Summary
Implement an intersection simplification for invariant generics, namely
for all gradual types
G. For the full justification, see #26054 (the result there can be generalized from fully-static types to all gradual types).Ecosystem
Both new diagnostic in spark and prefect seems to be true positives.