gh-112433: Add optional _align_ attribute to ctypes.Structure #113790

monkeyman192 · 2024-01-07T03:10:08Z

This PR adds an _align_ attribute to ctypes.Structure's so that is it possible to set it manually.

Issue: Add ability to force alignment of ctypes.Structure #112433

Issue: Add ability to force alignment of ctypes.Structure #112433

📚 Documentation preview 📚: https://cpython-previews--113790.org.readthedocs.build/

ghost · 2024-01-07T03:10:11Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2024-01-07T03:10:12Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

serhiy-storchaka

It is an interesting idea.

But what should be the size and the alignment of B in the following example?

class A(Structure):
    x: c_uint32

class B(A):
    _align_ = 8
    y: c_uint32

monkeyman192 · 2024-01-28T00:23:17Z

It is an interesting idea.

But what should be the size and the alignment of B in the following example?
class A(Structure):
    x: c_uint32

class B(A):
    _align_ = 8
    y: c_uint32

Thanks for the review!

My intuition would say "8 and 8".
Running the code shows that while the size is 8, the alignment of B is 4, not 8 as specified.
To sanity check I wrote a c++ equivalent of the above and it has size 8, alignment 8.
I'll work on fixing this issue and add some unit tests to cover these cases also, thanks!
I'll also add the news snippet which I have been meaning to do...

Modules/_ctypes/stgdict.c

Lib/test/test_ctypes/test_aligned_structures.py

serhiy-storchaka · 2024-01-28T12:08:11Z

!buildbot s390x

bedevere-bot · 2024-01-28T12:08:14Z

🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 2329fa3 🤖

The command will test the builders whose names match following regular expression: s390x

The builders matched are:

s390x RHEL7 LTO + PGO PR
s390x Fedora LTO PR
s390x RHEL7 LTO PR
s390x SLES PR
s390x Fedora Rawhide LTO PR
s390x Fedora Clang Installed PR
s390x RHEL8 Refleaks PR
s390x Fedora Rawhide LTO + PGO PR
s390x RHEL8 LTO + PGO PR
s390x RHEL8 PR
s390x Fedora Rawhide Refleaks PR
s390x Fedora Refleaks PR
s390x Fedora Clang PR
s390x Fedora LTO + PGO PR
s390x Fedora Rawhide PR
s390x Fedora PR
s390x RHEL8 LTO PR
s390x Fedora Rawhide Clang PR
s390x Debian PR
s390x RHEL7 Refleaks PR
s390x RHEL7 PR
s390x Fedora Rawhide Clang Installed PR

serhiy-storchaka · 2024-01-28T12:49:11Z

https://buildbot.python.org/all/#/builders/390/builds/1350/steps/5/logs/stdio

======================================================================
FAIL: test_aligned_string (test.test_ctypes.test_aligned_structures.TestAlignedStructures.test_aligned_string)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-debian-z/build/Lib/test/test_ctypes/test_aligned_structures.py", line 26, in test_aligned_string
    self.assertEqual(main.first, 7)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: 117440512 != 7
======================================================================
FAIL: test_aligned_struct_in_union (test.test_ctypes.test_aligned_structures.TestAlignedStructures.test_aligned_struct_in_union)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-debian-z/build/Lib/test/test_ctypes/test_aligned_structures.py", line 158, in test_aligned_struct_in_union
    self.assertEqual(main.first, 1)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: 16777216 != 1
======================================================================
FAIL: test_aligned_structures (test.test_ctypes.test_aligned_structures.TestAlignedStructures.test_aligned_structures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-debian-z/build/Lib/test/test_ctypes/test_aligned_structures.py", line 64, in test_aligned_structures
    self.assertEqual(main.y, 7)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: 117440512 != 7
======================================================================
FAIL: test_aligned_subclasses (test.test_ctypes.test_aligned_structures.TestAlignedStructures.test_aligned_subclasses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-debian-z/build/Lib/test/test_ctypes/test_aligned_structures.py", line 106, in test_aligned_subclasses
    self.assertEqual(main.a, 1)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: 16777216 != 1
======================================================================
FAIL: test_aligned_union (test.test_ctypes.test_aligned_structures.TestAlignedStructures.test_aligned_union)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-debian-z/build/Lib/test/test_ctypes/test_aligned_structures.py", line 129, in test_aligned_union
    self.assertEqual(main.first, 1)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: 16777216 != 1
----------------------------------------------------------------------

monkeyman192 · 2024-01-28T23:08:55Z

In the process of making the tests run with both big and little endian versions of the Structure and Union classes I found something curious...

from ctypes import *

class AlignedUnion(BigEndianUnion):
    _fields_ = [
        ("a", c_uint32),
        ("b", c_int32),
    ]

class Main(BigEndianStructure):
    _fields_ = [
        ("first", c_uint32),
        ("union", AlignedUnion),
    ]

m = Main()

If I run this on my machine (little endian), I get the following error:
TypeError: This type does not support other endian: <class '__main__.AlignedUnion'>

This seems like a bug and is the reason the last few tests look the way they do...

serhiy-storchaka · 2024-01-28T23:16:33Z

It was fixed not long time ago in #105102.

monkeyman192 · 2024-01-28T23:22:16Z

It was fixed not long time ago in #105102.

Ok great!

I know that as per the requirements for PR's it's required to make each commit separate and to not rebase, but is it fine to rebase the whole branch onto the current head so that I get the fix for this? I haven't contributed before and just want to make sure I'm following the right procedure...

Edit: Oh I see you already merged the code into the branch, thanks!

serhiy-storchaka · 2024-01-28T23:24:46Z

Don't rebase, just merge. I just merged main into your branch, you have to pull the changes.

serhiy-storchaka

Some questions and suggestions to tests. Please add more checks for offsets, alignments and sizes.

Interesting, how does setting _align_ in a Union subclass affect it?

serhiy-storchaka · 2024-01-29T11:37:06Z

Lib/test/test_ctypes/test_aligned_structures.py

+            self.assertEqual(main.first, 7)
+            self.assertEqual(main.string.value, b'hello world!')
+            self.assertEqual(
+                bytes(main.string.__buffer__(inspect.BufferFlags.SIMPLE)),


Maybe simply bytes(main.string)?

serhiy-storchaka · 2024-01-29T11:38:06Z

Lib/test/test_ctypes/test_aligned_structures.py

+            self.assertEqual(main.string.value, b'hello world!')
+            self.assertEqual(
+                bytes(main.string.__buffer__(inspect.BufferFlags.SIMPLE)),
+                b'\x68\x65\x6c\x6c\x6f\x20\x77\x6f\x72\x6c\x64\x21\x00\x00\x00\x00'


b'hello world!\0\0\0\0' could be more readable.

Lib/test/test_ctypes/test_aligned_structures.py

serhiy-storchaka · 2024-01-29T13:06:16Z

Lib/test/test_ctypes/test_aligned_structures.py

+            if ubase == LittleEndianUnion:
+                self.assertEqual(bytes(main.union.b), b"\3\0\0\0\4\0\0\0")
+            else:
+                self.assertEqual(bytes(main.union.b), b"\0\0\0\3\0\0\0\4")


It is simply data[8:].

serhiy-storchaka · 2024-01-29T13:08:14Z

Lib/test/test_ctypes/test_aligned_structures.py

+class TestAlignedStructures(unittest.TestCase):
+    def test_aligned_string(self):
+        for base, data in (
+            (LittleEndianStructure, bytearray(


You can also use struct.pack() to generate data. It may be more compact and more readable.

monkeyman192 · 2024-01-29T23:24:26Z

Thanks for the thorough review!
I believe I have addressed all the comments and added some further tests to test the case of a subclassed Union where the subclass has an alignment.
I used struct.pack in most places to clean it except for one where the format string was just too ugly looking to make it obvious what the data would be at a glance.

serhiy-storchaka

Thank you for your response @monkeyman192. I have yet few suggestions for polishing test.

serhiy-storchaka · 2024-01-30T08:05:35Z

Lib/test/test_ctypes/test_aligned_structures.py

            self.assertEqual(Main.string.offset, 16)
+            self.assertEqual(Main.string.size, 16)


It is not affected by the _align_ setting, because its size was already 16. Maybe make the value field type c_char * 12? Then the _align_ setting will affect also the size of Aligned.

serhiy-storchaka · 2024-01-30T08:17:04Z

Lib/test/test_ctypes/test_aligned_structures.py

-                    ("y", c_uint32),
+                    ("x", c_ubyte),
+                    ("y", SomeBools),
+                    ("z", c_uint32),


What if make z a byte and make SomeBools containing only 2 fields? Then without the _align_ setting Main can be packed in just 4 bytes. The _align_ setting will made larger difference.

Or just set _align_ to 8. This would cover also the next test.

serhiy-storchaka · 2024-01-30T08:37:20Z

Lib/test/test_ctypes/test_aligned_structures.py

+                    ("y", SomeBoolsTooBig),
+                    ("z", c_uint32),
+                ]
+            with self.assertRaises(ValueError) as ctx:


It is still not very informative test. If you want to test that the _align_ setting affects the size of Main, you can do this more directly by checking sizeof(Main). If you want to check also content, use the data of size 12.

I guess with this test I was just wanting to ensure that the appropriate error is raised when you have a Structure/Union with an alignment which will cause the size it expects to be bigger than what it is given. Rather than checking anything about the content itself.

serhiy-storchaka · 2024-01-30T08:48:44Z

Lib/test/test_ctypes/test_aligned_structures.py

+            self.assertEqual(main.unsigned, 0xD6)
+            self.assertEqual(main.signed, -42)


Oh, it is difficult to test unions with different byte order. But your test will also pass if badding bytes are added before unsigned (i.e. if Main.unsigned.offset is 7). I would use all different bytes in the input data, and tested:

self.assertEqual(main.unsigned, data[0]) self.assertEqual(main.signed, data[0] - 256)

Please add also checks for alignments, offsets and sizes.

serhiy-storchaka · 2024-01-30T08:50:09Z

Lib/test/test_ctypes/test_aligned_structures.py

+            self.assertEqual(bytes(main.string), b'hello world!\0\0\0\0')
+            self.assertEqual(Main.string.offset, 16)
+            self.assertEqual(Main.string.size, 16)
+            self.assertEqual(alignment(main.string), 16)


Does alignment(Main.string) work?

It does not since Main.string is a ctypes.CField which has no alignment info.

monkeyman192 · 2024-02-01T11:35:17Z

I realised that I have neither documented nor tested the case of having both _align_ and _pack_ attributes. I'll add that also as I think it's important to understand how it should behave.

Doc/library/ctypes.rst

monkeyman192 · 2024-02-09T22:55:46Z

Sorry for the ping @serhiy-storchaka just checking to see if there was anything else I needed to address with this.
I believe I have covered the points you raised previously as well as the typo raised in the documentation.
Let me know if there are any further changes I need to make

serhiy-storchaka

LGTM.

serhiy-storchaka · 2024-02-12T11:53:05Z

!buildbot s390x Debian

bedevere-bot · 2024-02-12T11:53:08Z

🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit a7bc0fe 🤖

The command will test the builders whose names match following regular expression: s390x Debian

The builders matched are:

s390x Debian PR

monkeyman192 · 2024-02-14T20:28:45Z

Ah, I just realised that this was already in the awaiting merge stage but I requested a review from aisk... Sorry about that...

…ythonGH-113790)

Add optional _align_ attribute to ctypes.Structure

a6ef3ac

bedevere-app bot added the awaiting review label Jan 7, 2024

bedevere-app bot mentioned this pull request Jan 7, 2024

Add ability to force alignment of ctypes.Structure #112433

Closed

serhiy-storchaka reviewed Jan 26, 2024

View reviewed changes

blurb-it bot and others added 2 commits January 28, 2024 02:46

📜🤖 Added by blurb_it.

e2cceec

Fix issue with aligned subclasses

2329fa3

serhiy-storchaka reviewed Jan 28, 2024

View reviewed changes

Modules/_ctypes/stgdict.c Outdated Show resolved Hide resolved

Lib/test/test_ctypes/test_aligned_structures.py Outdated Show resolved Hide resolved

Improve tests to pass on big endian machines

ebaed7e

Merge branch 'main' into add_structure_alignment

6b183b5

Further improvements to tests

c8b37d4

serhiy-storchaka reviewed Jan 29, 2024

View reviewed changes

Test cleanup and improvements

68e2d61

serhiy-storchaka reviewed Jan 30, 2024

View reviewed changes

More test improvements

f6a93dd

aisk reviewed Feb 1, 2024

View reviewed changes

Doc/library/ctypes.rst Outdated Show resolved Hide resolved

Add extra test and fix documentation typo

a7bc0fe

serhiy-storchaka approved these changes Feb 12, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Feb 12, 2024

monkeyman192 requested a review from aisk February 14, 2024 20:14

serhiy-storchaka merged commit 298bcdc into python:main Feb 15, 2024

bedevere-app bot removed the awaiting merge label Feb 15, 2024

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024

pythongh-112433: Add optional _align_ attribute to ctypes.Structure (p…

d61ab3f

…ythonGH-113790)

LukasWoodtli pushed a commit to LukasWoodtli/cpython that referenced this pull request Jan 22, 2025

pythongh-112433: Add optional _align_ attribute to ctypes.Structure (p…

4a5881c

…ythonGH-113790)

		self.assertEqual(Main.string.offset, 16)
		self.assertEqual(Main.string.size, 16)

		self.assertEqual(main.unsigned, 0xD6)
		self.assertEqual(main.signed, -42)

Uh oh!

gh-112433: Add optional _align_ attribute to ctypes.Structure #113790

gh-112433: Add optional _align_ attribute to ctypes.Structure #113790

Uh oh!

Conversation

monkeyman192 commented Jan 7, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jan 7, 2024 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Jan 7, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

monkeyman192 commented Jan 28, 2024

Uh oh!

Uh oh!

Uh oh!

serhiy-storchaka commented Jan 28, 2024

Uh oh!

bedevere-bot commented Jan 28, 2024

Uh oh!

serhiy-storchaka commented Jan 28, 2024

Uh oh!

monkeyman192 commented Jan 28, 2024

Uh oh!

serhiy-storchaka commented Jan 28, 2024

Uh oh!

monkeyman192 commented Jan 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Jan 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

monkeyman192 commented Jan 29, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

monkeyman192 commented Feb 1, 2024

Uh oh!

Uh oh!

monkeyman192 commented Feb 9, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka commented Feb 12, 2024

Uh oh!

bedevere-bot commented Feb 12, 2024

Uh oh!

monkeyman192 commented Jan 7, 2024 •

edited by github-actions bot

Loading

ghost commented Jan 7, 2024 •

edited by ghost

Loading

monkeyman192 commented Jan 28, 2024 •

edited

Loading

serhiy-storchaka commented Jan 28, 2024 •

edited

Loading