Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[ty] Guard self-referential TypeOf recursion in generic callables#24668

Merged
charliermarsh merged 4 commits into
mainfrom
charlie/generic-returned-callable
May 12, 2026
Merged

[ty] Guard self-referential TypeOf recursion in generic callables#24668
charliermarsh merged 4 commits into
mainfrom
charlie/generic-returned-callable

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Apr 16, 2026

Copy link
Copy Markdown
Member

Summary

This PR fixes recursive TypeOf handling for generic callables that return callables referencing themselves, as in:

from collections.abc import Callable
from typing import Concatenate
from ty_extensions import TypeOf, generic_context

def self_recursive[**P, T](
    x: Callable[Concatenate[TypeOf[self_recursive], ...], T],
) -> Callable[Concatenate[TypeOf[self_recursive], P], T]:
    return x

reveal_type(generic_context(self_recursive))  # ty_extensions.GenericContext[T@self_recursive]

Previously, rescoping the returned callable could rebuild the function signature from the function literal, which re-
entered TypeOf[self_recursive] and ultimately overflowed. This PR adds recursion guards to avoid re-inferring function signatures in all relevant places: through aliases, descriptors, properties, CallableTypeOf, and mutually recursive callable references.

Like #24683, I initially implemented this by threading a visitor through all type operations (first commit). That worked, but it required threading the visitor through in the correct places. So I did a second pass (second commit) to use a thread-local guard, which significantly reduces the diff at the cost of some implicit magic. I am fine either approach, but figured this was much easier to review.

Closes astral-sh/ty#3135.

@astral-sh-bot astral-sh-bot Bot added the ty Multi-file analysis & type inference label Apr 16, 2026
// the signature. Re-walking the signature here can instead force recursive expansion
// of the function type while that same signature is still under construction.
return;
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When building a function's generic context, a self-referential TypeOf[foo] can't reveal new typevars that aren't already in the signature of foo.

@astral-sh-bot

astral-sh-bot Bot commented Apr 16, 2026

Copy link
Copy Markdown

Typing conformance results

No changes detected ✅

Current numbers
The percentage of diagnostics emitted that were expected errors held steady at 89.36%. The percentage of expected errors that received a diagnostic held steady at 85.49%. The number of fully passing files held steady at 88/134.

Comment thread crates/ty_python_semantic/src/types.rs Outdated
if function.definition(db) == *self_reference_definition =>
{
self
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the "rename return-callable typevars" pass... Previously, we'd recurse into foo while trying to rename foo's own signature.

@charliermarsh charliermarsh changed the title [ty] Guard self-referential TypeOf recursion in generic callables [ty] Guard self-referential TypeOf recursion in generic callables Apr 16, 2026
@charliermarsh charliermarsh added the bug Something isn't working label Apr 16, 2026
@astral-sh-bot

astral-sh-bot Bot commented Apr 16, 2026

Copy link
Copy Markdown

Memory usage report

Memory usage unchanged ✅

@astral-sh-bot

astral-sh-bot Bot commented Apr 16, 2026

Copy link
Copy Markdown

ecosystem-analyzer results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

@charliermarsh charliermarsh marked this pull request as ready for review April 16, 2026 01:48
@astral-sh-bot astral-sh-bot Bot requested a review from oconnor663 April 16, 2026 01:49
@carljm carljm removed their request for review April 16, 2026 04:42
Comment on lines +1217 to +1218
if binding_context
.is_some_and(|binding_context| self.literal(db).definition(db) == binding_context)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this only guards against functions whose signatures are directly recursive. But if we add some indirection like this, it looks like it still hangs/panics?

def foo[**P, T](x: Callable[Concatenate[TypeOf[bar], ...], T]) -> Callable[Concatenate[TypeOf[bar], P], T]:
    return x
def bar[**P, T](x: Callable[Concatenate[TypeOf[foo], ...], T]) -> Callable[Concatenate[TypeOf[foo], P], T]:
    return x

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're absolutely right! I think the fix here is actually simpler... We don't want to recurse at all.

@charliermarsh charliermarsh marked this pull request as draft April 24, 2026 11:38
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch 2 times, most recently from afefe8a to b1e659b Compare April 24, 2026 16:06
@charliermarsh charliermarsh marked this pull request as ready for review April 24, 2026 17:45
@astral-sh-bot astral-sh-bot Bot requested a review from oconnor663 April 24, 2026 17:45
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch from b1e659b to 2968591 Compare April 26, 2026 00:09

@oconnor663 oconnor663 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I was writing the comment suggestion below, I asked Codex whether it's really correct that there's no case where a FunctionLiteral might need to participate in this special rule about return-position Callables. It came up with the following counterexample, which passes on main but fails with this PR:

from collections.abc import Callable
from ty_extensions import TypeOf

class Box[T]:
    @staticmethod
    def method(x: T) -> T:
        return x

def factory[T]() -> Callable[[TypeOf[Box[T].method]], T]:
    raise NotImplementedError

# The following is a case of the "typevar only appears inside a returned
# `Callable`" rule involving a `FunctionLiteral`. The only occurrence of
# `factory`'s `T` is in the return annotation, under `Callable[...]`, so
# `factory` itself should become non-generic and `factory()` should return a
# generic callable instead. In that returned callable, the old `T@factory`
# should be consistently rewritten to the new callable-owned typevar, usually
# displayed as `T'return@factory`. That means its parameter should be the exact
# function type `TypeOf[Box[T'return]. method]`, and passing `Box[int].method`
# should instantiate the returned callable at `T'return = int`.
#
# The current fix skips applying this rewrite recursively into `FunctionLiteral`
# types. `TypeOf[Box[T].method]` is represented as a function-literal-shaped
# type, so the outer returned `Callable` gets the new `T'return`, but the exact
# function type inside its parameter still contains the stale `T@factory`. Since
# `factory` is no longer generic, that stale `T@factory` is no longer something
# the returned callable can bind. As a result, ty rejects `Box[int].method`
# (`def method(x: int) -> int`) against the stale expected type
# (`def method(x: T@factory) -> T@factory`). This line should type-check and
# have type `int`. On main it's currently `Unknown`, but there is no diagnostic.

factory()(Box[int].method)  # error[invalid-argument-type]

The error on this branch:

error[invalid-argument-type]: Argument is incorrect
  --> breaks_good_case.py:32:11
   |
32 | factory()(Box[int].method)  # error[invalid-argument-type]
   |           ^^^^^^^^^^^^^^^ Expected `def method(x: T@factory) -> T@factory`, found `def method(x: int) -> int`
   |
info: incompatible return types: `int` is not assignable to `T@factory`

My first instinct was that this updated fix might not be entirely correct. But actually my second instinct is that both of these examples (the original that motivated this PR, and Codex's counterexample here) are "pathological fuzzer inputs", and the only really important thing is that we don't hang or crash on either of them? So in that sense it's probably a good fix, approved! Though if we believe that Codex's explanation is right here, we should probably edit the "there's no way" part of my comment suggestion into a TODO of some kind? @dcreager I'd be curious to get your take on this example, since I'm only just learning about this whole ReturnCallables rule by reading through your PR notes :)

Comment thread crates/ty_python_semantic/src/types.rs Outdated
Type::FunctionLiteral(function) => visitor.visit(self, type_mapping, || {
match type_mapping {
TypeMapping::ApplySpecialization(ApplySpecialization::ReturnCallables(_)) => {
self

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self
// `ReturnCallables` handles the special case of typevars that are only
// mentioned in a `Callable` in return position. There's no way for a
// `FunctionLiteral` type to participate in that special case, but
// pathological examples can turn into infinite recursion. Short-circuit to
// prevent that.
self

@charliermarsh charliermarsh marked this pull request as draft April 28, 2026 21:54
@charliermarsh

Copy link
Copy Markdown
Member Author

Good catch! I think it's worth getting this right... Let me take another pass at it.

@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch 3 times, most recently from 41666bb to 345c06f Compare April 29, 2026 01:06
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch 3 times, most recently from bf11a8a to 3c2df9e Compare May 3, 2026 17:43
@charliermarsh charliermarsh marked this pull request as ready for review May 4, 2026 05:37
@charliermarsh charliermarsh marked this pull request as draft May 4, 2026 05:48
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch from 3c2df9e to 4be19af Compare May 4, 2026 05:55
@charliermarsh charliermarsh marked this pull request as ready for review May 4, 2026 05:58
@charliermarsh charliermarsh marked this pull request as draft May 4, 2026 16:31
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch 3 times, most recently from 71b0166 to 2a86167 Compare May 4, 2026 17:38
@charliermarsh charliermarsh marked this pull request as ready for review May 4, 2026 17:45
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch 8 times, most recently from 45c12e7 to a3b305b Compare May 11, 2026 17:10
@charliermarsh charliermarsh force-pushed the charlie/generic-returned-callable branch from a3b305b to 8a39037 Compare May 12, 2026 00:12
}
) {
(
self.updated_signature(db).map(|signature| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried that this might incorrectly turn into a no-op in a case where updated_signature wasn't already populated, but Codex wasn't able to come with any real inputs to demonstrate that.

let _guard = ActiveRecursionGuard {
seen: &self.seen,
item,
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this part of the change is for panic safety? Probably worth a comment. Out of curiosity were/are there any panics that do something other than crash the whole process here, where robustly cleaning up this piece of state matters in practice?

nested: bool,
}

std::thread_local! {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely don't love this, and I was going to whip up an example of how easy it would be to stumble over this by sprinkling say rayon::join on the caller. But it turns out that &dyn Db isn't Sync, so you can't actually get an example like that to compile. Ultimately I agree that passing this visitor as an argument everywhere (the spiritually pure option) would be a gigantic change for no concrete benefit.

That said, if you haven't used rayon::join before, definitely give it a try. It's an incredible building block for threaded code :)

@charliermarsh charliermarsh merged commit 21bb256 into main May 12, 2026
57 checks passed
@charliermarsh charliermarsh deleted the charlie/generic-returned-callable branch May 12, 2026 12:36
@charliermarsh

Copy link
Copy Markdown
Member Author

Thanks @oconnor663! Added some explanatory comments. Agree with your analysis on the thread_local stuff.

thejchap pushed a commit to thejchap/ruff that referenced this pull request May 23, 2026
…stral-sh#24668)

## Summary

This PR fixes recursive `TypeOf` handling for generic callables that
return callables referencing themselves, as in:

```python
from collections.abc import Callable
from typing import Concatenate
from ty_extensions import TypeOf, generic_context

def self_recursive[**P, T](
    x: Callable[Concatenate[TypeOf[self_recursive], ...], T],
) -> Callable[Concatenate[TypeOf[self_recursive], P], T]:
    return x

reveal_type(generic_context(self_recursive))  # ty_extensions.GenericContext[T@self_recursive]
```

Previously, rescoping the returned callable could rebuild the function
signature from the function literal, which re-
entered `TypeOf[self_recursive]` and ultimately overflowed. This PR adds
recursion guards to avoid re-inferring function signatures in all
relevant places: through aliases, descriptors, properties,
`CallableTypeOf`, and mutually recursive callable references.

Like astral-sh#24683, I initially
implemented this by threading a visitor through all type operations
(first commit). That worked, but it required threading the visitor
through in the correct places. So I did a second pass (second commit) to
use a thread-local guard, which significantly reduces the diff at the
cost of some implicit magic. I am fine either approach, but figured this
was much easier to review.

Closes astral-sh/ty#3135.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash/hang with self referential function with Concatenate

2 participants