Thanks to visit codestin.com
Credit goes to github.com

Skip to content

🐛 Prefer custom encoder over defaults if specified in jsonable_encoder#2061

Merged
tiangolo merged 3 commits into
fastapi:masterfrom
viveksunder:master
Jan 23, 2022
Merged

🐛 Prefer custom encoder over defaults if specified in jsonable_encoder#2061
tiangolo merged 3 commits into
fastapi:masterfrom
viveksunder:master

Conversation

@viveksunder

Copy link
Copy Markdown
Contributor

In encoders.py/jsonable_encoder, default encoders for each type are specified first & are used in preference to custom encoders.

This affects custom encoders specified for certain types like enums such that the default encoder is always executed and any custom encoders specified are ignored. See - #1986 - for a description of this issue and example code as well that illustrates the problem.

This PR adds a change to prefer custom encoders if specified in preference to default encoders.

Disclaimer - I am completely new to this code base :-)

@codecov

codecov Bot commented Sep 16, 2020

Copy link
Copy Markdown

Codecov Report

Merging #2061 (fffd897) into master (3de0fb8) will not change coverage.
The diff coverage is 100.00%.

❗ Current head fffd897 differs from pull request most recent head a7efa6a. Consider uploading reports for the commit a7efa6a to get more accurate results
Impacted file tree graph

@@             Coverage Diff             @@
##            master     #2061     +/-   ##
===========================================
  Coverage   100.00%   100.00%             
===========================================
  Files          509       243    -266     
  Lines        12807      7428   -5379     
===========================================
- Hits         12807      7428   -5379     
Impacted Files Coverage Δ
fastapi/encoders.py 100.00% <100.00%> (ø)
tests/test_jsonable_encoder.py 100.00% <100.00%> (ø)
fastapi/params.py 100.00% <0.00%> (ø)
fastapi/routing.py 100.00% <0.00%> (ø)
fastapi/responses.py 100.00% <0.00%> (ø)
fastapi/concurrency.py 100.00% <0.00%> (ø)
fastapi/applications.py 100.00% <0.00%> (ø)
fastapi/openapi/docs.py 100.00% <0.00%> (ø)
fastapi/openapi/utils.py 100.00% <0.00%> (ø)
... and 291 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3de0fb8...a7efa6a. Read the comment docs.

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit e85b8e86880ef1da23a4c2e6a39535499c8eb3ff at: https://5f61bb0c6d3cd4463e8e5c14--fastapi.netlify.app

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit 379762189f4ac877ae9ec32214fc0008a6fe40ba at: https://5f629ba0924ff0920a89f626--fastapi.netlify.app

@viveksunder

Copy link
Copy Markdown
Contributor Author

Hi @ycd - thanks for approving. Just wondering when PRs are typically merged in & what the process is. Is there anything else needed from me in order to get this merged in?

@Kludex

Kludex commented Sep 23, 2020

Copy link
Copy Markdown
Member

Just wait for @tiangolo 's input.

@ycd

ycd commented Sep 23, 2020

Copy link
Copy Markdown
Contributor

Exactly, @tiangolo usually releases a new version in 14-21 days, but he just recently moved to Germany so he is probably too busy too look here, but i believe next release would not take longer than 1-2 weeks.

@viveksunder

Copy link
Copy Markdown
Contributor Author

Gotcha, not a problem at all! Thanks @ycd & @Kludex :)

@mikegrima

mikegrima commented Sep 25, 2020

Copy link
Copy Markdown

Hello! This does appear to fix my issue at #2091. However, if I may make one suggestion to this PR, please also fix the mutable dictionary that is set as a keyword argument with this code:

    custom_encoder: dict = None,     # <------- Make this change
    sqlalchemy_safe: bool = True,
) -> Any:
    custom_encoder = custom_encoder or {}       # <-------- And this change
    if custom_encoder:
        if type(obj) in custom_encoder:
            return custom_encoder[type(obj)](obj)
        else:
            for encoder_type, encoder in custom_encoder.items():
                if isinstance(obj, encoder_type):
                    return encoder(obj)

More details on this is here: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

@viveksunder

Copy link
Copy Markdown
Contributor Author

Hello! This does appear to fix my issue at #2091. However, if I may make one suggestion to this PR, please also fix the mutable dictionary that is set as a keyword argument with this code:

    custom_encoder: dict = None,     # <------- Make this change
    sqlalchemy_safe: bool = True,
) -> Any:
    custom_encoder = custom_encoder or {}       # <-------- And this change
    if custom_encoder:
        if type(obj) in custom_encoder:
            return custom_encoder[type(obj)](obj)
        else:
            for encoder_type, encoder in custom_encoder.items():
                if isinstance(obj, encoder_type):
                    return encoder(obj)

More details on this is here: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

I agree! I can make this change if there are no objections.

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit a92f760b66ecf936ad3bee568c7115220a3e3da8 at: https://5f736b1a6789f9014b8f9245--fastapi.netlify.app

@viveksunder

Copy link
Copy Markdown
Contributor Author

Hate to be a bother @ycd :) but just bumping this because I made the change @mikegrima suggested.

Also, do we think this is going to be merged in soon? Trying to decide if I should use my personal fork to use this change (which I'd rather not on the off-chance that this change doesn't actually get merged in)

@mikegrima

Copy link
Copy Markdown

Would like to make a friendly bump to get this released soon.

@viveksunder

Copy link
Copy Markdown
Contributor Author

Hi @tiangolo - We (and others) would really like to use this ASAP. Really appreciate it if you could review this & lmk if good.

@viveksunder

Copy link
Copy Markdown
Contributor Author

Hello. Is there something preventing this from being merged? It's been open for a while now and we really need this change!

@mikegrima

Copy link
Copy Markdown

@viveksunder Can you rebase this PR?

@ycd one this is rebased, can you merge it in?

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit 3a89879740e47c8760f9ed863e59b5ebef9fc253 at: https://5fe41162b6d01b7e356fabee--fastapi.netlify.app

@viveksunder

viveksunder commented Dec 24, 2020

Copy link
Copy Markdown
Contributor Author

My git commit history got borked while rebasing 🤕 so I started a new one here - #2567

Never mind, github is pretty smart & turns out I didn't need to open a new one.

@github-actions github-actions Bot temporarily deployed to commit December 24, 2020 05:00 Inactive
@viveksunder viveksunder reopened this Dec 24, 2020
@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit fffd897 at: https://5fe42059b6d01baf796fa7df--fastapi.netlify.app

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit fffd897 at: https://5fe42118605fe19c1aa29afe--fastapi.netlify.app

@ycd

ycd commented Dec 25, 2020

Copy link
Copy Markdown
Contributor

@mikegrima I'd love to do it but the only @tiangolo can merge PR's.

@mojimi

mojimi commented Feb 8, 2021

Copy link
Copy Markdown

Really looking forward to this, any news from @tiangolo ?

@mikegrima

Copy link
Copy Markdown

@tiangolo Friendly bump to merge.

@dhofstetter

Copy link
Copy Markdown

@tiangolo Do you see any problems with the approach implemented here? Sometimes it's really useful to override the default encoder. E.g. encoding Decimals to json using a predefined quantization. Would be really great if you would accept this PR

@kovalevvlad

kovalevvlad commented Mar 18, 2021

Copy link
Copy Markdown

+1 would really like to have a custom Enum serializer. @tiangolo

@mikegrima

mikegrima commented Mar 18, 2021

Copy link
Copy Markdown

@tiangolo please click merge. The PR fixes a bug.

EDIT: Sorry if this came off too strong (that was not my intention).

@philipbjorge

philipbjorge commented May 6, 2021

Copy link
Copy Markdown

Is this PR still necessary?

For my usecase at least, it seems like the following resolved.
#1769

Probably there's a different one I'm not understanding :)

@kovalevvlad

Copy link
Copy Markdown

@philipbjorge indeed it is still needed, the check for isinstance(obj, Enum) in fastapi/encoders.py is performed before checking whether custom encoders exist, therefore it is currently impossible to create a functional custom encoder for an Enum type.

@mojimi

mojimi commented May 17, 2021

Copy link
Copy Markdown

For me this is very necessary, as I work with DynamoDB and it requires floats to be transformed to Decimal in the boto3 library.

My current solution makes another iteration on top of the result of jsonable_encoder to fix floats, quite unoptimal

@mikegrima

mikegrima commented May 17, 2021

Copy link
Copy Markdown

Can someone please reach out to @tiangolo to get this merged in? This fixes a legitimate bug and adds a desirable feature for a number of users.

@JP-Globality

Copy link
Copy Markdown

@tiangolo is there anything else we can do to get this PR over the line and merged in?

I too have the same issue as @mikegrima and can verify this PR fixes my issue by allowing the user to specify custom json_encoders for Enums. Simply doing enum.value doesn't produce JSONable values for nested Enums.

@darkpssngr

Copy link
Copy Markdown

waiting for this to be merged as well. @tiangolo could you please look at it?

@justinttl

Copy link
Copy Markdown

It will be fantastic to iron out this very surprising behavior and incompatibility between FastAPI and pydantic! Trying to implement pydantic/pydantic#1004 for my API at the moment and it relies on custom encoders working expectedly. @tiangolo

@pdenapo

pdenapo commented Oct 5, 2021

Copy link
Copy Markdown

+1 It will be very helpful to fix this. For instance, I'm trying to use fastapi with Mongodb, and I'am having trouble with the Decimal128 type for pymongo.bson which is not natively supported by pydantic, so I would need to provide a custom serialization. The problem seems to be with collection types like sets, lists, etc. @tiangolo

crazyscientist pushed a commit to SUSE/aimaas that referenced this pull request Oct 12, 2021
Initial project setup

* Add module with settings
* Add EAV models
* Add some CRUD actions and pydantic models for them
* Add `commit` flag to create_attribute():
  Need this because we can create `Attribute` during `Schema` creation,
  but something can go wrong, so we don't want it to be saved immediately
* Allow non-unique values for entities:
  Now we can create entities with unique attributes having non-unique
  values if existing entity(ies) having this value is marked as deleted
* Forbid binding FK to deleted schemas
* Forbid updates of deteled schemas
* Forbid creating entities for deleted schemas
* Add link to entity for Value models
* Add unique constraint on AttributeDefinition
* Add custom exceptions
* Change DT caster to `fromtimestamp`
* Add AttrTypeMap as serializing workaround:
  Currently, if we try to call `.json()` on any Pydantic model that
  has `AttrType` field in it, Pydantic won't be able to convert it.
  Pydantic allows custom serializers for type, so we can overcome this
  issue for Pydantic. However, it seems that FastAPI ignores custom
  serializers and breaks when it comes across our custom types.
  That's why I wrote this workaround. There will be no need in it when
  this (or similar) PR gets merged to FastAPI repo:
  fastapi/fastapi#2061
* Added some tests for crud
* Check required fields on entity creation
* Catch IntegrityError on update of schema name/slug
@nicknotfun

Copy link
Copy Markdown

Also waiting on this fix, wondering what is holding it up? @ycd @JP-Globality @tiangolo ?

@aliereno

aliereno commented Nov 9, 2021

Copy link
Copy Markdown

I needed to implement custom encoder for decimal and enum types, but I couldn't. I am looking forward to this PR merged.

V3lop5 added a commit to NOWUM/EnSysMod that referenced this pull request Jan 4, 2022
@yribeiro

yribeiro commented Jan 6, 2022

Copy link
Copy Markdown

@tiangolo please merge this. My team rely on FastAPI and Pydantic quite heavily. Having to write custom types and using type ignores is not the best. Would be great if this can get merged into the next release!

@github-actions

Copy link
Copy Markdown
Contributor

📝 Docs preview for commit a7efa6a at: https://61ed82f4bc4aa1c45bb3b669--fastapi.netlify.app

@tiangolo

Copy link
Copy Markdown
Member

Awesome, thank you @viveksunder! ☕

And thanks everyone for the interest! 🍰

This is available in FastAPI 0.73.0, released in a couple of hours. 🎉

@tiangolo tiangolo changed the title Prefer custom encoder over defaults if specified. 🐛 Prefer custom encoder over defaults if specified in jsonable_encoder Jan 23, 2022
@tiangolo tiangolo merged commit 94ca8c1 into fastapi:master Jan 23, 2022
JeanArhancet pushed a commit to JeanArhancet/fastapi that referenced this pull request Aug 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.