[ty] Allow replacing ordinary methods with compatible functions#26158
Conversation
| identity-sensitive code unsound: after the assignment below, `requires_original_add(Foo.add)` is | ||
| accepted even though the function only allows the original `add`. Its assertion fails at runtime | ||
| because `Foo.add` is now `add_replacement`. Calls to the method itself remain safe because both | ||
| functions have compatible signatures and the same instance-binding behavior. |
There was a problem hiding this comment.
I know this is lengthy but it took me some time to understand it myself so wanted to leave a full explanation.
There was a problem hiding this comment.
I'm changing the example to avoid ty_extensions, maybe I'm still misunderstanding though, hmm...
| replaced, `DescriptorMethods.class_(1)` would stop passing `DescriptorMethods` as the first | ||
| argument, while `DescriptorMethods().class_(1)` would pass the instance instead of the class. | ||
| Supporting these assignments requires replacements with the corresponding decorator (for example, | ||
| `staticmethod(static_replacement)` or `classmethod(class_replacement)`). |
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 94.37%. The percentage of expected errors that received a diagnostic held steady at 89.00%. The number of fully passing files held steady at 94/134. |
Memory usage reportMemory usage unchanged ✅ |
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-assignment |
0 | 23 | 12 |
unresolved-attribute |
8 | 0 | 0 |
| Total | 8 | 23 | 12 |
Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.
Raw diff (43 changes)
dd-trace-py (https://github.com/DataDog/dd-trace-py)
- ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/pybind11/setup_helpers.py:492:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `def compile(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
+ ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/pybind11/setup_helpers.py:492:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
- ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/pybind11/setup_helpers.py:500:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `def compile(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
+ ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/pybind11/setup_helpers.py:500:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
- ddtrace/internal/compat.py:90:5 error[invalid-assignment] Object of type `def _register(self, cls, method=None) -> Unknown` is not assignable to attribute `register` of type `Overload[(self, cls: type[Any], method: None = None) -> (((...) -> Unknown, /) -> ((...) -> Unknown)), (self, cls: (...) -> Unknown, method: None = None) -> ((...) -> Unknown), (self, cls: type[Any], method: (...) -> Unknown) -> ((...) -> Unknown)]`
- ddtrace/internal/coverage/multiprocessing_coverage.py:220:5 error[invalid-assignment] Object of type `def join(self, *args, **kwargs) -> Unknown` is not assignable to attribute `join` of type `def join(self, timeout: int | float | None = None) -> None`
- ddtrace/internal/coverage/multiprocessing_coverage.py:219:5 error[invalid-assignment] Object of type `def close(self) -> Unknown` is not assignable to attribute `close` of type `def close(self) -> None`
+ ddtrace/internal/coverage/multiprocessing_coverage.py:219:5 error[invalid-assignment] Object of type `def close(self) -> Unknown` is not assignable to attribute `close` of type `(self) -> None`
- ddtrace/internal/coverage/multiprocessing_coverage.py:221:5 error[invalid-assignment] Object of type `def kill(self) -> Unknown` is not assignable to attribute `kill` of type `def kill(self) -> None`
+ ddtrace/internal/coverage/multiprocessing_coverage.py:221:5 error[invalid-assignment] Object of type `def kill(self) -> Unknown` is not assignable to attribute `kill` of type `(self) -> None`
- ddtrace/internal/coverage/multiprocessing_coverage.py:222:5 error[invalid-assignment] Object of type `def terminate(self) -> Unknown` is not assignable to attribute `terminate` of type `def terminate(self) -> None`
+ ddtrace/internal/coverage/multiprocessing_coverage.py:222:5 error[invalid-assignment] Object of type `def terminate(self) -> Unknown` is not assignable to attribute `terminate` of type `(self) -> None`
- ddtrace/internal/coverage/threading_coverage.py:88:5 error[invalid-assignment] Object of type `def join(self, *args, **kwargs) -> Unknown` is not assignable to attribute `join` of type `def join(self, timeout: int | float | None = None) -> None`
- tests/profiling/collector/test_lock_reflection.py:190:5 error[invalid-assignment] Object of type `def broken_getattr(self: _ProfiledLock, name: str) -> object` is not assignable to attribute `__getattr__` of type `def __getattr__(self, name: str) -> Any`
discord.py (https://github.com/Rapptz/discord.py)
- discord/enums.py:100:9 error[invalid-assignment] Object of type `(self, other) -> Literal[False] | Unknown` is not assignable to attribute `__le__` of type `def __le__(self, value: tuple[Unknown, ...], /) -> bool`
+ discord/enums.py:100:80 error[unresolved-attribute] Object of type `Self@__le__` has no attribute `value`
+ discord/enums.py:100:94 error[unresolved-attribute] Object of type `Self@__le__` has no attribute `value`
- discord/enums.py:101:9 error[invalid-assignment] Object of type `(self, other) -> Literal[False] | Unknown` is not assignable to attribute `__ge__` of type `def __ge__(self, value: tuple[Unknown, ...], /) -> bool`
+ discord/enums.py:101:80 error[unresolved-attribute] Object of type `Self@__ge__` has no attribute `value`
+ discord/enums.py:101:94 error[unresolved-attribute] Object of type `Self@__ge__` has no attribute `value`
- discord/enums.py:102:9 error[invalid-assignment] Object of type `(self, other) -> Literal[False] | Unknown` is not assignable to attribute `__lt__` of type `def __lt__(self, value: tuple[Unknown, ...], /) -> bool`
+ discord/enums.py:102:80 error[unresolved-attribute] Object of type `Self@__lt__` has no attribute `value`
+ discord/enums.py:102:93 error[unresolved-attribute] Object of type `Self@__lt__` has no attribute `value`
- discord/enums.py:103:9 error[invalid-assignment] Object of type `(self, other) -> Literal[False] | Unknown` is not assignable to attribute `__gt__` of type `def __gt__(self, value: tuple[Unknown, ...], /) -> bool`
+ discord/enums.py:103:80 error[unresolved-attribute] Object of type `Self@__gt__` has no attribute `value`
+ discord/enums.py:103:93 error[unresolved-attribute] Object of type `Self@__gt__` has no attribute `value`
ibis (https://github.com/ibis-project/ibis)
- ibis/backends/singlestoredb/tests/test_client.py:327:5 error[invalid-assignment] Object of type `def mock_do_connect(_self, *_args, **kwargs) -> Unknown` is not assignable to attribute `do_connect` of type `def do_connect(self, host: str | None = None, user: str | None = None, password: str | None = None, port: int | None = None, database: str | None = None, driver: str | None = None, autocommit: bool = True, local_infile: bool = True, **kwargs) -> None`
meson (https://github.com/mesonbuild/meson)
- unittests/failuretests.py:50:5 error[invalid-assignment] Object of type `def new_search(self, name, search_dirs, exclude_paths) -> Unknown` is not assignable to attribute `_search` of type `def _search(self, name: str, search_dirs: list[str | None], exclude_paths: list[str] | None) -> list[str]`
mitmproxy (https://github.com/mitmproxy/mitmproxy)
- examples/contrib/ntlm_upstream_proxy.py:150:9 error[invalid-assignment] Object of type `def patched_start_handshake(self) -> Generator[Command, Any, None]` is not assignable to attribute `start_handshake` of type `def start_handshake(self) -> Generator[Command, Any, None]`
- examples/contrib/ntlm_upstream_proxy.py:151:9 error[invalid-assignment] Object of type `def patched_receive_handshake_data(self, data) -> Generator[Command, Any, tuple[bool, str | None]]` is not assignable to attribute `receive_handshake_data` of type `def receive_handshake_data(self, data: bytes) -> Generator[Command, Any, tuple[bool, str | None]]`
mypy (https://github.com/python/mypy)
- mypyc/build.py:514:5 error[invalid-assignment] Object of type `def patched(self: Any) -> None` is not assignable to attribute `copy_extensions_to_source` of type `def copy_extensions_to_source(self) -> None`
- mypyc/build_setup.py:69:1 error[invalid-assignment] Object of type `def spawn(self, cmd, **kwargs) -> None` is not assignable to attribute `spawn` of type `def spawn(self, cmd: Iterable[str]) -> None`
- mypyc/lib-rt/build_setup.py:69:1 error[invalid-assignment] Object of type `def spawn(self, cmd, **kwargs) -> None` is not assignable to attribute `spawn` of type `def spawn(self, cmd: Iterable[str]) -> None`
pybind11 (https://github.com/pybind/pybind11)
- pybind11/setup_helpers.py:488:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `def compile(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
+ pybind11/setup_helpers.py:488:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
- pybind11/setup_helpers.py:496:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `def compile(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
+ pybind11/setup_helpers.py:496:9 error[invalid-assignment] Object of type `(CCompiler, list[str], str | None, list[tuple[str] | tuple[str, str | None]] | None, list[str] | None, bool, list[str] | None, list[str] | None, list[str] | None, /) -> list[str]` is not assignable to attribute `compile` of type `(self, sources: Sequence[str | PathLike[str]], output_dir: str | None = None, macros: list[tuple[str] | tuple[str, str | None]] | None = None, include_dirs: list[str] | None = None, debug: bool | Literal[0, 1] = 0, extra_preargs: list[str] | None = None, extra_postargs: list[str] | None = None, depends: list[str] | None = None) -> list[str]`
pywin32 (https://github.com/mhammond/pywin32)
- pythonwin/pywin/framework/intpyapp.py:31:1 error[invalid-assignment] Object of type `def _SetupSharedMenu_(self) -> Unknown` is not assignable to attribute `_SetupSharedMenu_` of type `def _SetupSharedMenu_(self) -> Unknown`
scikit-learn (https://github.com/scikit-learn/scikit-learn)
- sklearn/utils/_mocking.py:349:1 error[invalid-assignment] Object of type `RequestMethod` is not assignable to attribute `set_fit_request` of type `def set_fit_request(self, **kwargs) -> Unknown`
+ sklearn/utils/_mocking.py:349:1 error[invalid-assignment] Object of type `RequestMethod` is not assignable to attribute `set_fit_request` of type `(self, **kwargs) -> Unknown`
scipy (https://github.com/scipy/scipy)
- scipy/stats/_distribution_infrastructure.py:4293:9 error[invalid-assignment] Object of type `def _moment_raw_formula(self, order, **kwargs) -> Unknown` is not assignable to attribute `_moment_raw_formula` of type `def _moment_raw_formula(self, order, **params) -> Unknown`
- scipy/stats/_distribution_infrastructure.py:4294:9 error[invalid-assignment] Object of type `def _moment_central_formula(self, order, **kwargs) -> Unknown` is not assignable to attribute `_moment_central_formula` of type `def _moment_central_formula(self, order, **params) -> Unknown`
- scipy/stats/_distribution_infrastructure.py:4295:9 error[invalid-assignment] Object of type `def _moment_standardized_formula(self, order, **kwargs) -> Unknown` is not assignable to attribute `_moment_standardized_formula` of type `def _moment_standardized_formula(self, order, **params) -> Unknown`
spack (https://github.com/spack/spack)
- lib/spack/docs/conf.py:137:1 error[invalid-assignment] Object of type `(self, instance, owner) -> Unknown` is not assignable to attribute `__get__` of type `def __get__(self, instance, owner) -> Unknown`
- lib/spack/spack/test/conftest.py:2409:5 error[invalid-assignment] Object of type `def _libc_from_python(self) -> Unknown` is not assignable to attribute `default_libc` of type `def default_libc(self) -> Unknown | None`
spark (https://github.com/apache/spark)
- python/pyspark/sql/session.py:140:9 error[invalid-assignment] Object of type `def toDF(self, schema=None, sampleRatio=None) -> Unknown` is not assignable to attribute `toDF` of type `Overload[[RowLike](self: RDD[RowLike], schema: list[str] | tuple[str, ...] | None = None, sampleRatio: int | float | None = None) -> DataFrame, [RowLike](self: RDD[RowLike], schema: StructType | str | None = None) -> DataFrame, [AtomicValue](self: RDD[AtomicValue], schema: AtomicType | str) -> DataFrame]`
- python/pyspark/sql/tests/connect/client/test_client.py:251:5 error[invalid-assignment] Object of type `(_) -> None` is not assignable to attribute `_cleanup_ml_cache` of type `def _cleanup_ml_cache(self) -> None`
+ python/pyspark/sql/tests/connect/client/test_client.py:251:5 error[invalid-assignment] Object of type `(_: SparkConnectClient) -> None` is not assignable to attribute `_cleanup_ml_cache` of type `(self) -> None`
- python/pyspark/sql/tests/connect/client/test_client_call_stack_trace.py:33:5 error[invalid-assignment] Object of type `(_) -> None` is not assignable to attribute `_cleanup_ml_cache` of type `def _cleanup_ml_cache(self) -> None`
+ python/pyspark/sql/tests/connect/client/test_client_call_stack_trace.py:33:5 error[invalid-assignment] Object of type `(_: SparkConnectClient) -> None` is not assignable to attribute `_cleanup_ml_cache` of type `(self) -> None`
sympy (https://github.com/sympy/sympy)
- sympy/interactive/printing.py:283:9 error[invalid-assignment] Object of type `def _repr_disabled(self) -> Unknown` is not assignable to attribute `_repr_latex_` of type `def _repr_latex_(self) -> Unknown`
- sympy/interactive/printing.py:275:9 error[invalid-assignment] Object of type `def _print_latex_text(o) -> Unknown` is not assignable to attribute `_repr_latex_` of type `def _repr_latex_(self) -> Unknown`
+ sympy/interactive/printing.py:275:9 error[invalid-assignment] Object of type `def _print_latex_text(o) -> Unknown` is not assignable to attribute `_repr_latex_` of type `(self) -> Unknown`
werkzeug (https://github.com/pallets/werkzeug)
- tests/live_apps/run.py:35:5 error[invalid-assignment] Object of type `(_) -> Any` is not assignable to attribute `address_string` of type `def address_string(self) -> str`
+ tests/live_apps/run.py:35:5 error[invalid-assignment] Object of type `(_: WSGIRequestHandler) -> Any` is not assignable to attribute `address_string` of type `(self) -> str`4ffbc7e to
8212ec3
Compare
8212ec3 to
58a206d
Compare
carljm
left a comment
There was a problem hiding this comment.
With #26253 fixed, this looks fine.
I'm still a bit on the fence about whether we should even do this. The mypy/zuban approach of erroring on this also seems pretty OK to me -- it's an odd thing to do, you can suppress the type error if you insist on doing it.
But if we are going to do it, this implementation looks good.
|
I guess I'm the one who put the issue into Stable milestone, so clearly I'm not thinking consistently over time about this... |
Summary
We currently retain a function-literal type when reading an ordinary method from a class literal. Because each function definition has its own singleton type, assigning a different function is rejected even when its signature and binding behavior match:
This widens the declared method to its function-like callable type only while checking a class attribute assignment. Reads continue to retain the original function-literal type, while replacements must have a compatible signature and ordinary instance-binding behavior.
This is intentionally unsound for function identity: after the assignment, the runtime value of
Foo.methodisreplacement, but its inferred type still identifies the method declared inFoo.staticmethodandclassmethodattributes are not widened because replacing either descriptor with a plain function changes how class and instance access bind arguments. Supporting those cases would require a replacement with the corresponding descriptor.Closes astral-sh/ty#2648.