feat: `json.dumps`, `json.dump` accuracy improvements #13960

max-muoto · 2025-05-08T23:57:54Z

This PR addresses #13781 by making json.dumps and json.dump more accurate:

When default isn't provided, only JSON serializable types are supported (as accurate we can get, we'll go through trade-offs)
When default is provided, the type must match that of the first call argument for default or what we consider JSON-serializable .

Trade-offs:

For reasons described here Define a JSON type typing#182 (comment) less accurate Sequence and Mapping typing is needed to support nested types properly but this will obviously still be a big improvement over Any.
As discussed here, TypedDicts are only compatible with Mapping[str, object] would it would be far preferable to use Mapping[str, _JSON] closing off typed dictionaries is going to lead to a noticeable number of false positives.
Type-checking custom encoders isn't practical. Realistically to do this we would need to introduce generics on JSONEncoder (defaulting to our custom JSON type as the default) however it would be impractical for users to supply said generic param considering it doesn't exist at runtime. As a result an overload exists to accept object in the case that any custom JSONEncoder is provided.

max-muoto · 2025-05-09T00:36:24Z

CC: @srittau if you're interested in reviewing.

github-actions · 2025-05-10T17:17:11Z

Diff from mypy_primer, showing the effect of this PR on open source code:

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/config.py:409: error: Argument 1 to "dumps" has incompatible type "Any | _NoValueType"; expected "_JSON"  [arg-type]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/io/parquet.py:191: error: Argument 1 to "dumps" has incompatible type "dict[Hashable, Any]"; expected "_JSON"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/cacheprovider.py:191: error: No overload variant of "dumps" matches argument types "object", "bool", "int"  [call-overload]
+ src/_pytest/cacheprovider.py:191: note: Possible overload variants:
+ src/_pytest/cacheprovider.py:191: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/_pytest/cacheprovider.py:191: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/_pytest/cacheprovider.py:191: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_json.py:26: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/util/spack_json.py:26: error: No overload variant of "dumps" matches argument types "dict[Any, Any]", "dict[str, tuple[str, str] | None]"  [call-overload]
+ lib/spack/spack/util/spack_json.py:26: note: Error code "call-overload" not covered by "type: ignore" comment
+ lib/spack/spack/util/spack_json.py:26: note: Possible overload variants:
+ lib/spack/spack/util/spack_json.py:26: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:26: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:26: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:27: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/util/spack_json.py:27: error: No overload variant of "dump" matches argument types "dict[Any, Any]", "Any", "dict[str, tuple[str, str] | None]"  [call-overload]
+ lib/spack/spack/util/spack_json.py:27: note: Error code "call-overload" not covered by "type: ignore" comment
+ lib/spack/spack/util/spack_json.py:27: note: Possible overload variants:
+ lib/spack/spack/util/spack_json.py:27: note:     def dump(obj: _JSON, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> None
+ lib/spack/spack/util/spack_json.py:27: note:     def [_T] dump(obj: _JSON | _T, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder] | None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> None
+ lib/spack/spack/util/spack_json.py:27: note:     def dump(obj: object, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> None

strawberry (https://github.com/strawberry-graphql/strawberry)
+ strawberry/http/base.py:51: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ strawberry/http/base.py:51: note: Possible overload variants:
+ strawberry/http/base.py:51: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ strawberry/http/base.py:51: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ strawberry/http/base.py:51: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str

scrapy (https://github.com/scrapy/scrapy)
+ scrapy/commands/settings.py:54: error: Argument 1 to "dumps" has incompatible type "dict[bool | float | int | str | None, Any]"; expected "_JSON"  [arg-type]

cki-lib (https://gitlab.com/cki-project/cki-lib)
+ cki_lib/owners.py:249: error: Argument "key" to "sorted" has incompatible type "Callable[[Mapping[str, object] | Sequence[_JSON] | float | bool | None], object | Any]"; expected "Callable[[Mapping[str, object] | Sequence[_JSON] | float | bool | None], SupportsDunderLT[Any] | SupportsDunderGT[Any]]"  [arg-type]
+ cki_lib/owners.py:249: error: No overload variant of "__getitem__" of "Sequence" matches argument type "str"  [call-overload]
+ cki_lib/owners.py:249: note: Possible overload variants:
+ cki_lib/owners.py:249: note:     def __getitem__(self, int, /) -> _JSON
+ cki_lib/owners.py:249: note:     def __getitem__(self, slice[Any, Any, Any], /) -> Sequence[_JSON]
+ cki_lib/owners.py:249: error: Value of type "Mapping[str, object] | Sequence[_JSON] | float | bool | None" is not indexable  [index]
+ cki_lib/owners.py:249: error: Incompatible return value type (got "object | Any", expected "SupportsDunderLT[Any] | SupportsDunderGT[Any]")  [return-value]

poetry (https://github.com/python-poetry/poetry)
+ src/poetry/utils/cache.py:181: error: No overload variant of "dumps" matches argument type "T"  [call-overload]
+ src/poetry/utils/cache.py:181: note: Possible overload variants:
+ src/poetry/utils/cache.py:181: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:181: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:181: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: error: No overload variant of "dumps" matches argument types "object", "bool", "tuple[str, str]", "bool"  [call-overload]
+ src/poetry/utils/cache.py:211: note: Possible overload variants:
+ src/poetry/utils/cache.py:211: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str

mypy (https://github.com/python/mypy)
+ mypy/util.py:935: error: Returning Any from function declared to return "bytes"  [no-any-return]
+ mypy/util.py:935: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-no-any-return for more info
+ mypy/util.py:935: error: No overload variant of "dumps" matches argument types "object", "int", "bool"  [call-overload]
+ mypy/util.py:935: note: Possible overload variants:
+ mypy/util.py:935: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-call-overload for more info
+ mypy/util.py:938: error: Returning Any from function declared to return "bytes"  [no-any-return]
+ mypy/util.py:938: error: No overload variant of "dumps" matches argument types "object", "bool", "tuple[str, str]"  [call-overload]
+ mypy/util.py:938: note: Possible overload variants:
+ mypy/util.py:938: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:938: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:938: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str

zulip (https://github.com/zulip/zulip)
+ zerver/lib/markdown/api_arguments_table_generator.py:170: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note: Possible overload variants:
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: error: No overload variant of "dumps" matches argument types "object", "bool"  [call-overload]
+ zerver/openapi/markdown_extension.py:243: note: Possible overload variants:
+ zerver/openapi/markdown_extension.py:243: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ zerver/openapi/markdown_extension.py:258: note: Possible overload variants:
+ zerver/openapi/markdown_extension.py:258: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:264: error: Incompatible return value type (got "object", expected "str")  [return-value]
+ zerver/lib/bot_lib.py:47: error: Incompatible types in assignment (expression has type overloaded function, variable has type "Callable[[object], str]")  [assignment]

max-muoto added 2 commits May 8, 2025 18:57

feat: Improve json.dumps, json.dump typing

6060876

Tweak

f7c16d0

max-muoto changed the title ~~feat: Improve json.dumps, json.dump typing~~ feat: json.dumps, json.dump accuracy improvements May 9, 2025

max-muoto and others added 4 commits May 8, 2025 19:09

Add test

c741607

Fixes

2a03418

[pre-commit.ci] auto fixes from pre-commit.com hooks

07f1922

Tweak test

87b10d5

max-muoto marked this pull request as ready for review May 9, 2025 00:24

This comment has been minimized.

Sign in to view

max-muoto and others added 5 commits May 8, 2025 19:26

Tweak

9d548a6

Tweak

07cf9d0

Tweak

c51d852

[pre-commit.ci] auto fixes from pre-commit.com hooks

454786b

Further broaden

6d07be5

Add json.dump tests

a05d94a

This comment has been minimized.

Sign in to view

max-muoto marked this pull request as draft May 9, 2025 01:42

This comment has been minimized.

Sign in to view

max-muoto force-pushed the improve-json-module-typing branch 2 times, most recently from 3ca5208 to a05d94a Compare May 9, 2025 02:13

Add future import

fe70767

This comment has been minimized.

Sign in to view

Support typed dictionaries

b16b4f6

This comment has been minimized.

Sign in to view

max-muoto marked this pull request as ready for review May 9, 2025 02:45

max-muoto marked this pull request as draft May 9, 2025 02:54

max-muoto added 3 commits May 8, 2025 22:12

Support subclasses...

ae677f1

Tweak

5fab9e8

Remove Any

ab39c3f

This comment has been minimized.

Sign in to view

max-muoto marked this pull request as ready for review May 9, 2025 03:48

max-muoto mentioned this pull request May 9, 2025

json.dump/dumps shouldn't allow Any when default is None #13781

Open

Merge branch 'main' into improve-json-module-typing

c4f68ca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `json.dumps`, `json.dump` accuracy improvements #13960

feat: `json.dumps`, `json.dump` accuracy improvements #13960

max-muoto commented May 8, 2025 •

edited

Loading

This comment has been minimized.

max-muoto commented May 9, 2025 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented May 10, 2025

feat: json.dumps, json.dump accuracy improvements #13960

Are you sure you want to change the base?

feat: json.dumps, json.dump accuracy improvements #13960

Conversation

max-muoto commented May 8, 2025 • edited Loading

This comment has been minimized.

max-muoto commented May 9, 2025 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented May 10, 2025

feat: `json.dumps`, `json.dump` accuracy improvements #13960

feat: `json.dumps`, `json.dump` accuracy improvements #13960

max-muoto commented May 8, 2025 •

edited

Loading

max-muoto commented May 9, 2025 •

edited

Loading