Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add experimental_allow_partial support #10748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Nov 4, 2024
Merged

Add experimental_allow_partial support #10748

merged 15 commits into from
Nov 4, 2024

Conversation

samuelcolvin
Copy link
Member

@samuelcolvin samuelcolvin commented Oct 31, 2024 •
edited
Loading

This relies on:

So far allow_partial support is only implemented on validators for the following types:

  • list
  • set
  • frozenset
  • dict (e.g. dict[X, Y])
  • TypedDict

There are 2 fundamental things introduced:

  • (in jiter) support for parsing partial json, e.g. any valid JSON input which "stops early", for example [1, 2, "thr
  • (in pydantic-core) support for ignoring errors while validating the last item in a sequence or last value in a mapping (the last item can be invalid because the original data stopped part way through its definition)

Example:

from pydantic import TypeAdapter, BaseModel
from typing import Literal


class UserRecord(BaseModel):
    id: int
    name: str
    role: Literal['admin', 'user']


ta = TypeAdapter(list[UserRecord])
# allow_partial if the input is a python object
d = ta.validate_python(
    [
        {'id': '1', 'name': 'Alice', 'role': 'user'},
        {'id': '1', 'name': 'Ben', 'role': 'user'},
        {'id': '1', 'name': 'Char'},
    ],
    experimental_allow_partial=True,
)
print(d)
#> [UserRecord(id=1, name='Alice', role='user'), UserRecord(id=1, name='Ben', role='user')]

# allow_partial if the input is a json string
d = ta.validate_json(
    '[{"id":"1","name":"Alice","role":"user"},{"id":"1","name":"Ben","role":"user"},{"id":"1","name":"Char"}]',
    experimental_allow_partial=True,
)
print(d)
#> [UserRecord(id=1, name='Alice', role='user'), UserRecord(id=1, name='Ben', role='user')]

@github-actions github-actions bot added the relnotes-fix Used for bugfixes. label Oct 31, 2024
@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Oct 31, 2024 •
edited
Loading

CodSpeed Performance Report

Merging #10748 will not alter performance

Comparing allow_partial (3ea36b0) with main (b4308e0)

Summary

✅ 44 untouched benchmarks

@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Oct 31, 2024 •
edited
Loading

Deploying pydantic-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3ea36b0
Status: ✅  Deploy successful!
Preview URL: https://ad0fa9a1.pydantic-docs.pages.dev
Branch Preview URL: https://allow-partial.pydantic-docs.pages.dev

View logs

@samuelcolvin samuelcolvin added relnotes-feature and removed relnotes-fix Used for bugfixes. labels Oct 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 31, 2024 •
edited
Loading

Coverage report

This PR does not seem to contain any modification to coverable code.

@samuelcolvin samuelcolvin changed the title Add experimental_allow_partial support to TypeAdapter Add experimental_allow_partial support Oct 31, 2024
@samuelcolvin samuelcolvin mentioned this pull request Oct 31, 2024
Merged
sydney-runkle
sydney-runkle reviewed Oct 31, 2024
View reviewed changes
Copy link
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super exciting overall - definitely a fan of the experimental_ prefix for now. I'm impressed with the turnaround here.

A few follow up questions:

  • We should add a section to the experimental docs talking about this flag (happy to have this between the beta and official release)
  • I'm wondering, does it make sense to only have this as a runtime flag? Should we also have a config setting for this - we have one for cache_strings, which I believe has a similar flow down to jitter? Even if not on config, I think we should make this accessible for fields in models, typed dicts, etc.

As a general note on the above, I feel we don't have a super clear system for when something belongs as a:

  • Runtime flag
  • Config setting
  • Annotation
  • Field setting

And we should probably document that to guide consistency in future development.

@devraj devraj mentioned this pull request Oct 31, 2024
Open
hyperlint-ai[bot]
hyperlint-ai bot reviewed Nov 1, 2024
View reviewed changes
Copy link
Contributor

@hyperlint-ai hyperlint-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 files reviewed, 2 total issue(s) found.

The style guide flagged several spelling errors that seemed like false positives. We skipped posting inline suggestions for the following words:

  • [Aa]nnotated_types

@samuelcolvin samuelcolvin mentioned this pull request Nov 1, 2024
Merged
hyperlint-ai[bot]
hyperlint-ai bot reviewed Nov 2, 2024
View reviewed changes
Copy link
Contributor

@hyperlint-ai hyperlint-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The style guide flagged several spelling errors that seemed like false positives. We skipped posting inline suggestions for the following words:

  • annotated_types

hyperlint-ai[bot]
hyperlint-ai bot reviewed Nov 3, 2024
View reviewed changes
Copy link
Contributor

@hyperlint-ai hyperlint-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 files reviewed, 3 total issue(s) found.

samuelcolvin and others added 4 commits November 3, 2024 11:56
@samuelcolvin @hyperlint-ai
fix typo
80187a7 
Co-authored-by: hyperlint-ai[bot] <154288675+hyperlint-ai[bot]@users.noreply.github.com>
@samuelcolvin
fix example formatting
3d24c16
@sydney-runkle
bump core version
3ceda20
@sydney-runkle
fix dt tests
40c84d7
hyperlint-ai[bot]
hyperlint-ai bot reviewed Nov 4, 2024
View reviewed changes
Copy link
Contributor

@hyperlint-ai hyperlint-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 files reviewed, 3 total issue(s) found.

sydney-runkle
sydney-runkle reviewed Nov 4, 2024
View reviewed changes
hyperlint-ai[bot]
hyperlint-ai bot reviewed Nov 4, 2024
View reviewed changes
Copy link
Contributor

@hyperlint-ai hyperlint-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The style guide flagged several spelling errors that seemed like false positives. We skipped posting inline suggestions for the following words:

  • jiter

sydney-runkle
sydney-runkle approved these changes Nov 4, 2024
View reviewed changes
Copy link
Contributor

@sydney-runkle sydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing!

Thanks for including such thorough docs - those will be easy to move to the concepts section, eventually!

@sydney-runkle sydney-runkle enabled auto-merge (squash) November 4, 2024 17:08
@sydney-runkle sydney-runkle merged commit 678ec30 into main Nov 4, 2024
53 checks passed
@sydney-runkle sydney-runkle deleted the allow_partial branch November 4, 2024 17:11
@bruno-f-cruz bruno-f-cruz mentioned this pull request Nov 8, 2024
Open
@samuelcolvin samuelcolvin mentioned this pull request Nov 10, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
2 more reviewers

@hyperlint-ai hyperlint-ai[bot] hyperlint-ai[bot] left review comments

@sydney-runkle sydney-runkle sydney-runkle approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
relnotes-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@samuelcolvin @sydney-runkle