gh-96151: Use a private name for passing builtins to dataclass #98143

hauntsaninja · 2022-10-10T07:06:53Z

There's no indication that BUILTINS is a special name. Other names that are special to dataclass are all prefixed by an underscore.

As mentioned in the issue, we can also avoid this locals dance altogether by using ().__class__.__base__ instead of BUILTINS.object.

Issue: BUILTINS is not a valid field name for a frozen dataclass #96151

There's no indication that BUILTINS is a special name. Other names that are special to dataclass are all prefixed by an underscore. As mentioned in the issue, we can also avoid this locals dance altogether by using `().__class__.__base__` instead of `BUILTINS.object`.

sobolevn

I think such subtile change needs a test case :)

ericvsmith · 2022-10-11T21:45:46Z

Lib/test/test_dataclasses.py

        self.assertEqual(c.object, 'foo')
+        self.assertEqual(c.BUILTINS, 5)


Please create a separate function for this, and don't mix it in with existing tests. Maybe test_field_named_BUILTINS or similar.

Thanks for the review, made the change :-)

JelleZijlstra · 2022-10-15T19:08:16Z

Lib/dataclasses.py

@@ -431,8 +431,8 @@ def _create_fn(name, args, body, *, globals=None, locals=None,
    # worries about external callers.
    if locals is None:
        locals = {}
-    if 'BUILTINS' not in locals:


I'd suggest using __BUILTINS__ because dunder names are supposed to be reserved for use by the stdlib.

Before I commit this, I'd like to spend some time researching why this test is even present, instead of just unconditionally assigning to locals. At the very least it could use a comment.

Also, I'm not sure that exposing all of builtins in locals is a good idea, versus just exposing builtins.object.

Good catch, looks like the relevant history is:

https://github.com/python/cpython/pull/9518/files#diff-44ce2dc1c4922b2f5cf7631d8f86cc569a4c25eb003aaecdc2bc22eb9163d5f5L371-L375

bpo-34213: frozen dataclass with "object" attr bug #8452

That is, I think this check was made dead in #9518, but wasn't noticed in that PR

(Also note that the comment at the top of _create_fn is out of date: we do mutate locals, but not via exec)

I can do some double checking (locals should all be created within dataclasses.py) and clean that up + change the PR to only pass along object.

(I'll also note that there's still the inscrutable ().__class__.__base__ option on the table, in case we don't want to expose anything at all)

Okay, I audited all the call sites, it looks like there is one case where this check is not dead, but it's accidental.

Over here we reuse the same locals dict for two different _create_fn calls:

cpython/Lib/dataclasses.py

Line 619 in 87b5fd9

locals = {'cls': cls,

so the second time round we already have the entry for builtins in the dict. shrug

So my conclusion is:
a) It's safe to remove the check.
b) We should actually go a little further. Since we only need builtins for the frozen init, we should pass that in explicitly when creating __init__ and remove this from _create_fn

I've gone ahead and pushed this change to the PR

hauntsaninja · 2022-10-17T02:20:12Z

@JelleZijlstra I made the change, but note that if single underscore names are considered fair game for users, you can elicit all kinds of bad behaviour from dataclass. For example:

from dataclasses import dataclass, field

@dataclass
class X:
    x: int = field(default_factory=lambda: 111)
    _dflt_x: int = field(default_factory=lambda: 222)

X()

I'm happy to open an issue and fix all of these as well, if you think it's worth doing.

TeamSpen210 · 2022-10-17T04:56:44Z

If we’re only actually using setattr, since __init__ is probably a rather hot bit of code would it be a bit more efficient to bind the __setattr__ method itself in the scope? Then there’s less lookups for each attribute being set.

hauntsaninja · 2022-10-17T05:21:13Z

Maybe? I heard a rumour that obj.method(...) is faster these days than m = obj.method; m(...). I'll do some benchmarking and if I get good results putting object or object.__setattr__ into scope, I'll open an issue.

ericvsmith · 2022-10-17T05:24:54Z

About single underscores:

I'm happy to open an issue and fix all of these as well, if you think it's worth doing.

That's worth opening an issue for.

hauntsaninja · 2022-10-30T22:51:58Z

Thanks, I opened #98886 for the single underscores.

Meanwhile, I also modified this PR to use a slightly different name, since my suggestion in #98886 is to have all these special names prefixed with __dataclass_ (like we already do for __dataclass_self__). I think this PR should be good to go.

ericvsmith · 2022-10-31T12:29:09Z

This all looks great. I'm not sure if this will backport cleanly, but I've added the tags for 3.10 and 3.11 backports I see this as a bug fix that should be backported, at least to 3.11.

miss-islington · 2022-10-31T12:39:56Z

Thanks @hauntsaninja for the PR, and @ericvsmith for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2022-10-31T12:39:56Z

Thanks @hauntsaninja for the PR, and @ericvsmith for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

… This now allows for a field named BUILTIN (pythongh-98143) (cherry picked from commit 29f98b4) Co-authored-by: Shantanu <[email protected]>

bedevere-bot · 2022-10-31T12:40:18Z

GH-98899 is a backport of this pull request to the 3.10 branch.

… This now allows for a field named BUILTIN (pythongh-98143) (cherry picked from commit 29f98b4) Co-authored-by: Shantanu <[email protected]>

bedevere-bot · 2022-10-31T12:40:22Z

GH-98900 is a backport of this pull request to the 3.11 branch.

…. This now allows for a field named BUILTIN (gh-98143) (gh-98899) gh-96151: Use a private name for passing builtins to dataclass. This now allows for a field named BUILTIN (gh-98143) (cherry picked from commit 29f98b4) Co-authored-by: Shantanu <[email protected]> Co-authored-by: Shantanu <[email protected]>

…. This now allows for a field named BUILTIN (gh-98143) (gh-98900) gh-96151: Use a private name for passing builtins to dataclass. This now allows for a field named BUILTIN (gh-98143) (cherry picked from commit 29f98b4) Co-authored-by: Shantanu <[email protected]> Co-authored-by: Shantanu <[email protected]>

This commit prefixes `__dataclass` to several things in the locals dict: - Names like _dflt_ (which cause trouble, see first test) - Names like _type_ (not known to be able to cause trouble) - _return_type (not known to able to cause trouble) - _HAS_DEFAULT_FACTORY (which causes trouble, see second test) In addition, this removes `MISSING` from the locals dict. As far as I can tell, this wasn't needed even in the initial implementation of dataclasses.py (and tests on that version passed with it removed). This is basically a continuation of python#96151, where fixing this was welcomed in python#98143 (comment)

…mes (#102032) This commit prefixes `__dataclass` to several things in the locals dict: - Names like `_dflt_` (which cause trouble, see first test) - Names like `_type_` (not known to be able to cause trouble) - `_return_type` (not known to able to cause trouble) - `_HAS_DEFAULT_FACTORY` (which causes trouble, see second test) In addition, this removes `MISSING` from the locals dict. As far as I can tell, this wasn't needed even in the initial implementation of dataclasses.py (and tests on that version passed with it removed). This makes me wary :-) This is basically a continuation of #96151, where fixing this was welcomed in #98143 (comment)

…ore names (python#102032) This commit prefixes `__dataclass` to several things in the locals dict: - Names like `_dflt_` (which cause trouble, see first test) - Names like `_type_` (not known to be able to cause trouble) - `_return_type` (not known to able to cause trouble) - `_HAS_DEFAULT_FACTORY` (which causes trouble, see second test) In addition, this removes `MISSING` from the locals dict. As far as I can tell, this wasn't needed even in the initial implementation of dataclasses.py (and tests on that version passed with it removed). This makes me wary :-) This is basically a continuation of python#96151, where fixing this was welcomed in python#98143 (comment)

hauntsaninja requested a review from ericvsmith as a code owner October 10, 2022 07:06

bedevere-bot added the awaiting review label Oct 10, 2022

📜🤖 Added by blurb_it.

7f9f91e

sobolevn reviewed Oct 10, 2022

View reviewed changes

ericvsmith self-assigned this Oct 10, 2022

add a test case

2d5f290

ericvsmith reviewed Oct 11, 2022

View reviewed changes

separate out test

d7e8782

JelleZijlstra reviewed Oct 15, 2022

View reviewed changes

use dunder builtins

96cc82b

hauntsaninja mentioned this pull request Oct 30, 2022

Fields with single underscore names can mess up dataclasses #98886

Closed

actually, prefix with __dataclass

e7456f8

JelleZijlstra approved these changes Oct 30, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Oct 30, 2022

hauntsaninja added 2 commits October 30, 2022 17:13

Remove builtins logic from _create_fn

8f6e968

also update comment

9d44429

bedevere-bot mentioned this pull request Oct 31, 2022

BUILTINS is not a valid field name for a frozen dataclass #96151

Closed

ericvsmith added 3.11 only security fixes 3.10 only security fixes labels Oct 31, 2022

ericvsmith merged commit 29f98b4 into python:main Oct 31, 2022

bedevere-bot removed the awaiting merge label Oct 31, 2022

AlexWaygood removed the 3.11 only security fixes label Oct 31, 2022

AlexWaygood added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes and removed 3.10 only security fixes labels Oct 31, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 31, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 31, 2022

hauntsaninja deleted the gh-96151 branch November 21, 2022 06:59

hauntsaninja mentioned this pull request Feb 18, 2023

gh-98886: Fix issues with dataclass fields with special underscore names #102032

Merged

		self.assertEqual(c.object, 'foo')
		self.assertEqual(c.BUILTINS, 5)

Uh oh!

gh-96151: Use a private name for passing builtins to dataclass #98143

gh-96151: Use a private name for passing builtins to dataclass #98143

Uh oh!

Conversation

hauntsaninja commented Oct 10, 2022 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

ericvsmith Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Oct 15, 2022

Choose a reason for hiding this comment

Uh oh!

ericvsmith Oct 30, 2022

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hauntsaninja Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hauntsaninja commented Oct 17, 2022

Uh oh!

TeamSpen210 commented Oct 17, 2022

Uh oh!

hauntsaninja commented Oct 17, 2022

Uh oh!

ericvsmith commented Oct 17, 2022

Uh oh!

hauntsaninja commented Oct 30, 2022

Uh oh!

ericvsmith commented Oct 31, 2022

Uh oh!

miss-islington commented Oct 31, 2022

Uh oh!

miss-islington commented Oct 31, 2022

Uh oh!

bedevere-bot commented Oct 31, 2022

Uh oh!

bedevere-bot commented Oct 31, 2022

Uh oh!

Uh oh!

hauntsaninja commented Oct 10, 2022 •

edited by bedevere-bot

Loading

hauntsaninja Oct 30, 2022 •

edited

Loading

hauntsaninja Oct 30, 2022 •

edited

Loading

hauntsaninja Oct 31, 2022 •

edited

Loading