Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 7a3bfc3

Browse files
tim-onetim-one
committed
Added q/Q standard (x-platform 8-byte ints) mode in struct module.
This completes the q/Q project. longobject.c _PyLong_AsByteArray: The original code had a gross bug: the most-significant Python digit doesn't necessarily have SHIFT significant bits, and you really need to count how many copies of the sign bit it has else spurious overflow errors result. test_struct.py: This now does exhaustive std q/Q testing at, and on both sides of, all relevant power-of-2 boundaries, both positive and negative. NEWS: Added brief dict news while I was at it.
1 parent ac4797a commit 7a3bfc3

5 files changed

Lines changed: 337 additions & 77 deletions

File tree

‎Doc/lib/libstruct.tex‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,8 @@ \section{\module{struct} ---
7272
\item[(1)]
7373
The \character{q} and \character{Q} conversion codes are available in
7474
native mode only if the platform C compiler supports C \ctype{long long},
75-
or, on Windows, \ctype{__int64}.
75+
or, on Windows, \ctype{__int64}. They're always available in standard
76+
modes.
7677
\end{description}
7778

7879

@@ -100,8 +101,8 @@ \section{\module{struct} ---
100101
is truncated. If the string is too short, padding is used to ensure
101102
that exactly enough bytes are used to satisfy the count.
102103

103-
For the \character{I} and \character{L} format characters, the return
104-
value is a Python long integer.
104+
For the \character{I}, \character{L}, \character{q} and \character{Q}
105+
format characters, the return value is a Python long integer.
105106

106107
For the \character{P} format character, the return value is a Python
107108
integer or long integer, depending on the size needed to hold a
@@ -139,10 +140,12 @@ \section{\module{struct} ---
139140
order.
140141

141142
Standard size and alignment are as follows: no alignment is required
142-
for any type (so you have to use pad bytes); \ctype{short} is 2 bytes;
143-
\ctype{int} and \ctype{long} are 4 bytes. \ctype{float} and
144-
\ctype{double} are 32-bit and 64-bit IEEE floating point numbers,
145-
respectively.
143+
for any type (so you have to use pad bytes);
144+
\ctype{short} is 2 bytes;
145+
\ctype{int} and \ctype{long} are 4 bytes;
146+
\ctype{long long} (\ctype{__int64} on Windows) is 8 bytes;
147+
\ctype{float} and \ctype{double} are 32-bit and 64-bit
148+
IEEE floating point numbers, respectively.
146149

147150
Note the difference between \character{@} and \character{=}: both use
148151
native byte order, but the size and alignment of the latter is

‎Lib/test/test_struct.py‎

Lines changed: 165 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,16 @@ def simple_err(func, *args):
1212
func.__name__, args)
1313
## pdb.set_trace()
1414

15+
def any_err(func, *args):
16+
try:
17+
apply(func, args)
18+
except (struct.error, OverflowError, TypeError):
19+
pass
20+
else:
21+
raise TestFailed, "%s%s did not raise error" % (
22+
func.__name__, args)
23+
## pdb.set_trace()
24+
1525
simple_err(struct.calcsize, 'Z')
1626

1727
sz = struct.calcsize('i')
@@ -113,7 +123,8 @@ def simple_err(func, *args):
113123
raise TestFailed, "unpack(%s, %s) -> (%s,) # expected (%s,)" % (
114124
`fmt`, `res`, `rev`, `arg`)
115125

116-
# Some q/Q sanity checks.
126+
###########################################################################
127+
# q/Q tests.
117128

118129
has_native_qQ = 1
119130
try:
@@ -124,18 +135,22 @@ def simple_err(func, *args):
124135
if verbose:
125136
print "Platform has native q/Q?", has_native_qQ and "Yes." or "No."
126137

127-
simple_err(struct.pack, "Q", -1) # can't pack -1 as unsigned regardless
138+
any_err(struct.pack, "Q", -1) # can't pack -1 as unsigned regardless
128139
simple_err(struct.pack, "q", "a") # can't pack string as 'q' regardless
129140
simple_err(struct.pack, "Q", "a") # ditto, but 'Q'
130141

142+
def string_reverse(s):
143+
chars = list(s)
144+
chars.reverse()
145+
return "".join(chars)
146+
131147
def bigendian_to_native(value):
132148
if isbigendian:
133149
return value
134-
chars = list(value)
135-
chars.reverse()
136-
return "".join(chars)
150+
else:
151+
return string_reverse(value)
137152

138-
if has_native_qQ:
153+
def test_native_qQ():
139154
bytes = struct.calcsize('q')
140155
# The expected values here are in big-endian format, primarily because
141156
# I'm on a little-endian machine and so this is the clearest way (for
@@ -156,3 +171,147 @@ def bigendian_to_native(value):
156171
verify(retrieved == input,
157172
"%r-unpack of %r gave %r, not %r" %
158173
(format, got, retrieved, input))
174+
175+
if has_native_qQ:
176+
test_native_qQ()
177+
178+
# Standard q/Q (8 bytes; should work on all platforms).
179+
180+
MIN_Q, MAX_Q = 0, 2L**64 - 1
181+
MIN_q, MAX_q = -(2L**63), 2L**63 - 1
182+
183+
import binascii
184+
def test_one_qQ(x, pack=struct.pack,
185+
unpack=struct.unpack,
186+
unhexlify=binascii.unhexlify):
187+
if verbose:
188+
print "trying std q/Q on", x, "==", hex(x)
189+
190+
# Try 'q'.
191+
if MIN_q <= x <= MAX_q:
192+
# Try '>q'.
193+
expected = long(x)
194+
if x < 0:
195+
expected += 1L << 64
196+
assert expected > 0
197+
expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
198+
if len(expected) & 1:
199+
expected = "0" + expected
200+
expected = unhexlify(expected)
201+
expected = "\x00" * (8 - len(expected)) + expected
202+
203+
# >q pack work?
204+
got = pack(">q", x)
205+
verify(got == expected,
206+
"'>q'-pack of %r gave %r, not %r" %
207+
(x, got, expected))
208+
209+
# >q unpack work?
210+
retrieved = unpack(">q", got)[0]
211+
verify(x == retrieved,
212+
"'>q'-unpack of %r gave %r, not %r" %
213+
(got, retrieved, x))
214+
215+
# Adding any byte should cause a "too big" error.
216+
any_err(unpack, ">q", '\x01' + got)
217+
218+
# Try '<q'.
219+
expected = string_reverse(expected)
220+
221+
# <q pack work?
222+
got = pack("<q", x)
223+
verify(got == expected,
224+
"'<q'-pack of %r gave %r, not %r" %
225+
(x, got, expected))
226+
227+
# <q unpack work?
228+
retrieved = unpack("<q", got)[0]
229+
verify(x == retrieved,
230+
"'<q'-unpack of %r gave %r, not %r" %
231+
(got, retrieved, x))
232+
233+
# Adding any byte should cause a "too big" error.
234+
any_err(unpack, "<q", '\x01' + got)
235+
236+
else:
237+
# x is out of q's range -- verify pack realizes that.
238+
any_err(pack, '>q', x)
239+
any_err(pack, '<q', x)
240+
241+
# Much the same for 'Q'.
242+
if MIN_Q <= x <= MAX_Q:
243+
# Try '>Q'.
244+
expected = long(x)
245+
expected = hex(expected)[2:-1] # chop "0x" and trailing 'L'
246+
if len(expected) & 1:
247+
expected = "0" + expected
248+
expected = unhexlify(expected)
249+
expected = "\x00" * (8 - len(expected)) + expected
250+
251+
# >Q pack work?
252+
got = pack(">Q", x)
253+
verify(got == expected,
254+
"'>Q'-pack of %r gave %r, not %r" %
255+
(x, got, expected))
256+
257+
# >Q unpack work?
258+
retrieved = unpack(">Q", got)[0]
259+
verify(x == retrieved,
260+
"'>Q'-unpack of %r gave %r, not %r" %
261+
(got, retrieved, x))
262+
263+
# Adding any byte should cause a "too big" error.
264+
any_err(unpack, ">Q", '\x01' + got)
265+
266+
# Try '<Q'.
267+
expected = string_reverse(expected)
268+
269+
# <Q pack work?
270+
got = pack("<Q", x)
271+
verify(got == expected,
272+
"'<Q'-pack of %r gave %r, not %r" %
273+
(x, got, expected))
274+
275+
# <Q unpack work?
276+
retrieved = unpack("<Q", got)[0]
277+
verify(x == retrieved,
278+
"'<Q'-unpack of %r gave %r, not %r" %
279+
(got, retrieved, x))
280+
281+
# Adding any byte should cause a "too big" error.
282+
any_err(unpack, "<Q", '\x01' + got)
283+
284+
else:
285+
# x is out of Q's range -- verify pack realizes that.
286+
any_err(pack, '>Q', x)
287+
any_err(pack, '<Q', x)
288+
289+
def test_std_qQ():
290+
from random import randrange
291+
292+
# Create all interesting powers of 2.
293+
values = []
294+
for exp in range(70):
295+
values.append(1L << exp)
296+
297+
# Add some random 64-bit values.
298+
for i in range(50):
299+
val = 0L
300+
for j in range(8):
301+
val = (val << 8) | randrange(256)
302+
values.append(val)
303+
304+
# Try all those, and their negations, and +-1 from them. Note
305+
# that this tests all power-of-2 boundaries in range, and a few out
306+
# of range, plus +-(2**n +- 1).
307+
for base in values:
308+
for val in -base, base:
309+
for incr in -1, 0, 1:
310+
x = val + incr
311+
try:
312+
x = int(x)
313+
except OverflowError:
314+
pass
315+
test_one_qQ(x)
316+
317+
test_std_qQ()

‎Misc/NEWS‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,9 @@ Core
8484
sortdict(dict) function for a simple way to display a dict in sorted
8585
order.
8686

87+
- Many other small changes to dicts were made, resulting in faster
88+
operation along the most common code paths.
89+
8790
- Dictionary objects now support the "in" operator: "x in dict" means
8891
the same as dict.has_key(x).
8992

@@ -119,7 +122,7 @@ Core
119122

120123
- Collisions in dicts are resolved via a new approach, which can help
121124
dramatically in bad cases. For example, looking up every key in a dict
122-
d with d.keys() = [i << 16 for i in range(20000)] is approximately 500x
125+
d with d.keys() == [i << 16 for i in range(20000)] is approximately 500x
123126
faster now. Thanks to Christian Tismer for pointing out the cause and
124127
the nature of an effective cure (last December! better late than never).
125128

@@ -145,8 +148,8 @@ Library
145148
native mode, these can be used only when the platform C compiler supports
146149
these types (when HAVE_LONG_LONG is #define'd by the Python config
147150
process), and then they inherit the sizes and alignments of the C types.
148-
XXX TODO In standard mode, 'q' and 'Q' are supported on all platforms, and
149-
XXX TODO are 8-byte integral types.
151+
In standard mode, 'q' and 'Q' are supported on all platforms, and are
152+
8-byte integral types.
150153

151154
Tests
152155

0 commit comments

Comments
 (0)