bpo-34213: frozen dataclass with "object" attr bug #8452

VadimPushtaev · 2018-07-24T21:28:54Z

For frozen dataclasses, attributes are set via object.__setattr__(name, value) in __init__. That won't work if object is a function argument which is entirely possible if object is an attribute name.

The same problem is already resolved for self which is replaced by __dataclass_self__ if self is an attribute name.

My solution is to store a reference to object in self and use self.__dataclass_object__ (or even __dataclass_self__.__dataclass_object__) if object doesn't work.

https://bugs.python.org/issue34213

ericvsmith

Thanks for the PR.

ericvsmith · 2018-07-26T00:05:29Z

Lib/dataclasses.py

    # If we're a frozen class, then assign to our fields in __init__
    # via object.__setattr__.  Otherwise, just use a simple
    # assignment.
    #
    # self_name is what "self" is called in this function: don't
    # hard-code "self", since that might be a field name.
    if frozen:
-        return f'object.__setattr__({self_name},{name!r},{value})'
+        return f'{object_expression}.__setattr__({self_name},{name!r},{value})'


I think the only change you need to make in this file is to use __builtins__.object instead of object here. Since identifiers that start with double underscores are reserved for Python, we don't need to support a field named __builtins__.

That's indeed a right thing do, I completely forgot about __builtins__.

However, surprisingly (at least for me) you should do __builtins__['object']__, not __builtins__.object since __builtins__ is a dictionary inside exec, not a module: “If the globals dictionary does not contain a value for the key __builtins__, a reference to the dictionary of the built-in module builtins is inserted under that key.” — https://docs.python.org/3/library/functions.html#exec

ericvsmith · 2018-07-26T00:06:30Z

Misc/NEWS.d/next/Library/2018-07-25-00-40-14.bpo-34213.O15MgP.rst

@@ -0,0 +1,2 @@
+Frozen dataclasses didn't work properly with an attribute called "object".
+Now they do.


Please change to: Allow frozen dataclasses to have a field named "object". Previously this conflicted with an internal use of "object".

ericvsmith · 2018-07-26T00:12:46Z

Lib/test/test_dataclasses.py

+            object: str
+        c=C('foo')
+        self.assertEqual(c.object, 'foo')
+


In addition to these tests, please add a test for field names that are all identifiers in __builtins__ which don't start with double underscores, except for None, True, and False. You can use make_dataclass to dynamically generate a class with those members. I think this will work to generate the field names: [n for n in __builtins__.__dict__.keys() if not n.startswith('__') and not n in ('None', 'False', 'True')]. You'll want to do this for both a normal dataclass and a frozen one. Also, you'll want to test order=True.

Something like:

names = [n for n in __builtins__.__dict__.keys() if not n.startswith('__') and not n in ('None', 'False', 'True')] C = make_dataclass('C', names) C(*([0] * len(names)))

Which works in 3.7.0, and

names = [n for n in __builtins__.__dict__.keys() if not n.startswith('__') and not n in ('None', 'False', 'True')] C = make_dataclass('C', names, frozen=True) C(*([0] * len(names)))

Which fails in 3.7.0.

[Edit: fix frozen example]

ericvsmith · 2018-07-26T00:13:20Z

Lib/test/test_dataclasses.py

+        @dataclass(frozen=True)
+        class C:
+            object: str
+        c=C('foo')


Add spaces around =.

The only reason I omitted spaces is they are already omitted in test_field_named_self which I used as a guideline. Should I fix that c=C as well? In this PR or in a separate one?

I'd just fix this particular case here, and another PR for all other cases of the same problem.

ericvsmith · 2018-07-26T00:13:27Z

Lib/test/test_dataclasses.py

+        @dataclass
+        class C:
+            object: str
+        c=C('foo')


Add spaces around =.

bedevere-bot · 2018-07-26T00:14:37Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

VadimPushtaev · 2018-07-26T12:56:31Z

I have made the requested changes; please review again.

bedevere-bot · 2018-07-26T12:56:34Z

Thanks for making the requested changes!

@ericvsmith: please review the changes made to this pull request.

ericvsmith · 2018-07-26T13:23:37Z

Lib/dataclasses.py

@@ -365,7 +365,7 @@ def _field_assign(frozen, name, value, self_name):
    # self_name is what "self" is called in this function: don't
    # hard-code "self", since that might be a field name.
    if frozen:
-        return f'object.__setattr__({self_name},{name!r},{value})'
+        return f'__builtins__["object"].__setattr__({self_name},{name!r},{value})'


Per your other comment: I did not know this was how you had to access __builtins__ inside exec. Although we could get around this by passing adding {"__builtins__": __builtins__} to the globals passed in to exec. I'll have to think about that: there might be other reasons to set globals["__builtins__"].

That's remarkable, isn't that.

Well, it is dict only if you provide the globals argument:

>>> exec('print(type(__builtins__))') <class 'module'> >>> exec('print(type(__builtins__))', None, None) <class 'module'> >>> exec('print(type(__builtins__))', {}, None) <class 'dict'>

That means that the right thing to do is to include __builtins__ in the globals dict or always use {} instead of None.

ericvsmith

Thanks for making these changes. When I have some free time, I'll run some tests and think about setting __builtins__ in globals in the exec call. If you haven't heard anything in a week or two, please ping me.

VadimPushtaev · 2018-07-26T14:24:40Z

Looks like is misunderstood the __builtins__ problem.

From what I got, __builtins__ may be a reference to the builtins module or to its __dict__. All imported modules have __builtins__ equal to __dict__ while a program that is directly run has __builtins__ equal to the builtins module.

For example, if I run make test that __builtins__ is a dictionary and you can't do __builtins__.__dict__, but it's completely fine as long as I run the test individually.

That is more or less described in the documentation — https://docs.python.org/3/library/builtins.html

Next, the default for any code in exec depends on whether you provide the globals argument or not:

If you don't provide globals or globals is equal to None, then __builtins__ is the same as the caller has. It can be the module or __dict__.
If you do provide globals without the __builtins__ key, then __builtins__ is always the dictionary.
If you do provide globals with the __builtins__ key, then the provided value is used.

This behavior is described here — https://docs.python.org/3/library/functions.html#exec

Having all the in mind, I concluded that we should use builtins instead of __builtins__ wherever possible. However, the generated code can't do that since builtins can still be shadowed by an argument name. So the solution is to use __builtins__ but make sure it's a reference to the module, not to its __dict__.

the-knights-who-say-ni · 2018-07-26T14:24:59Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

When your account is ready, please add a comment in this pull request
and a Python core developer will remove the CLA not signed label
to make the bot check again.

Thanks again for your contribution, we look forward to reviewing it!

VadimPushtaev · 2018-07-26T14:40:08Z

When your account is ready, please add a comment in this pull request
and a Python core developer will remove the CLA not signed label
to make the bot check again.

Done — https://bugs.python.org/user29042

the-knights-who-say-ni · 2018-07-26T14:42:30Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

When your account is ready, please add a comment in this pull request
and a Python core developer will remove the CLA not signed label
to make the bot check again.

Thanks again for your contribution, we look forward to reviewing it!

VadimPushtaev · 2018-07-27T09:30:02Z

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org

Fixed.

VadimPushtaev · 2018-08-06T16:22:15Z

@ericvsmith, you asked to ping you :).

miss-islington · 2018-08-12T11:46:08Z

Thanks @VadimPushtaev for the PR, and @ericvsmith for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

bedevere-bot · 2018-08-12T11:46:30Z

GH-8745 is a backport of this pull request to the 3.7 branch.

…thonGH-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

miss-islington · 2018-08-12T14:15:18Z

Thanks @VadimPushtaev for the PR, and @ericvsmith for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

bedevere-bot · 2018-08-12T14:15:40Z

GH-8746 is a backport of this pull request to the 3.7 branch.

…thonGH-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

…-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

bpo-34213: frozen dataclass with "object" attr bug

fea7cf1

VadimPushtaev requested a review from ericvsmith as a code owner July 24, 2018 21:28

bedevere-bot added the awaiting review label Jul 24, 2018

bpo-34213: news for frozen dataclasses bug

3171e9c

ericvsmith requested changes Jul 26, 2018

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Jul 26, 2018

bpo-34213: frozen dataclass with "object" attr bug

4d7ee27

bedevere-bot added awaiting change review and removed awaiting changes labels Jul 26, 2018

ericvsmith reviewed Jul 26, 2018

View reviewed changes

bpo-34213: __builtins__ bug

4b0ea3e

bpo-34213: "__builtins" typo fix

d08afd9

ericvsmith approved these changes Aug 12, 2018

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Aug 12, 2018

ericvsmith added the needs backport to 3.7 label Aug 12, 2018

ericvsmith merged commit 4d12e4d into python:master Aug 12, 2018

bedevere-bot removed the awaiting merge label Aug 12, 2018

bedevere-bot removed the needs backport to 3.7 label Aug 12, 2018

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 12, 2018

bpo-34213: Allow dataclasses to work with a field named 'object'. (py…

da86307

…thonGH-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

berkerpeksag added the needs backport to 3.7 label Aug 12, 2018

bedevere-bot removed the needs backport to 3.7 label Aug 12, 2018

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 12, 2018

bpo-34213: Allow dataclasses to work with a field named 'object'. (py…

0d62b12

…thonGH-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

miss-islington added a commit that referenced this pull request Aug 13, 2018

bpo-34213: Allow dataclasses to work with a field named 'object'. (GH…

32e58fc

…-8452) (cherry picked from commit 4d12e4d) Co-authored-by: Vadim Pushtaev <[email protected]>

hauntsaninja mentioned this pull request Oct 30, 2022

gh-96151: Use a private name for passing builtins to dataclass #98143

Merged

		@@ -0,0 +1,2 @@
		Frozen dataclasses didn't work properly with an attribute called "object".
		Now they do.

Uh oh!

bpo-34213: frozen dataclass with "object" attr bug #8452

bpo-34213: frozen dataclass with "object" attr bug #8452

Uh oh!

Conversation

VadimPushtaev commented Jul 24, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericvsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericvsmith Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VadimPushtaev Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Jul 26, 2018

Uh oh!

VadimPushtaev commented Jul 26, 2018

Uh oh!

bedevere-bot commented Jul 26, 2018

Uh oh!

ericvsmith Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VadimPushtaev Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericvsmith left a comment

Choose a reason for hiding this comment

Uh oh!

VadimPushtaev commented Jul 26, 2018

Uh oh!

the-knights-who-say-ni commented Jul 26, 2018

Uh oh!

VadimPushtaev commented Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

the-knights-who-say-ni commented Jul 26, 2018

Uh oh!

VadimPushtaev commented Jul 27, 2018

Uh oh!

VadimPushtaev commented Aug 6, 2018

Uh oh!

miss-islington commented Aug 12, 2018

Uh oh!

bedevere-bot commented Aug 12, 2018

Uh oh!

miss-islington commented Aug 12, 2018

Uh oh!

bedevere-bot commented Aug 12, 2018

Uh oh!

Uh oh!

VadimPushtaev commented Jul 24, 2018 •

edited by bedevere-bot

Loading

ericvsmith Jul 26, 2018 •

edited

Loading

VadimPushtaev Jul 26, 2018 •

edited

Loading

ericvsmith Jul 26, 2018 •

edited

Loading

VadimPushtaev Jul 26, 2018 •

edited

Loading

VadimPushtaev commented Jul 26, 2018 •

edited

Loading