bpo-35560: Remove incorrect assert statement that caused segfault in debug builds #11288

tirkarthi · 2018-12-22T16:56:16Z

Removed assert statement checking for negative min_width. This caused segfault in debug builds for some format string in float. Reference for negative min_width :

cpython/Python/formatter_unicode.c

Line 529 in 59423e3

/* min_width can go negative, that's okay. format->width == -1 means
Add tests from https://bugs.python.org/msg99839

https://bugs.python.org/issue35560

serhiy-storchaka · 2018-12-22T17:25:57Z

Lib/test/test_float.py

@@ -701,6 +701,12 @@ def test_issue5864(self):
        self.assertEqual(format(1234.56, '.4'), '1.235e+03')
        self.assertEqual(format(12345.6, '.4'), '1.235e+04')

+    def test_issue35560(self):
+        self.assertEqual(format(123.0, '00'), '123.0')
+        self.assertEqual(format(float(123), '00'), '123.0')


float(123) is the same as 123.0. This line is not needed.

The following lines do not add anything for testing purpose too. str.format() and f-strings just use format() internally.

But it may be worth to add tests for format strings containing explicit format char '00f', '00e', '00g', and containing precision: '00.20', '0020f', etc. Test also with negative number, -123.0.

Added test cases as suggested.

Added min_width = Py_MAX(0, min_width); as suggested in https://bugs.python.org/msg332359.

serhiy-storchaka · 2018-12-22T17:44:38Z

Misc/NEWS.d/next/Core and Builtins/2018-12-22-22-19-51.bpo-35560.9vMWSP.rst

@@ -0,0 +1,2 @@
+Removed incorrect assert statement that caused segfault in debug builds.


This is too general and unlikely useful. Please specify place/conditions of the segfault.

Added a note on the function and assert statement. Thanks.

…entry.

serhiy-storchaka · 2018-12-22T19:36:54Z

Lib/test/test_float.py

+        self.assertEqual(format(123.34, '00e'), '1.233400e+02')
+        self.assertEqual(format(123.340, '00g'), '123.34')
+        self.assertEqual(format(123.340, '00.10f'), '123.3400000000')
+        self.assertEqual(format(123.340, '0010f'), '123.340000')


The bug is not reproducible with non-zero width. Add instead tests for non-zero precision and other format characters: '00.10e', '00.10g'.

serhiy-storchaka · 2018-12-22T19:37:11Z

Lib/test/test_float.py

+        self.assertEqual(format(123.0, '00'), '123.0')
+        self.assertEqual(format(123.34, '00f'), '123.340000')
+        self.assertEqual(format(123.34, '00e'), '1.233400e+02')
+        self.assertEqual(format(123.340, '00g'), '123.34')


123.340 is the same as 123.34.

serhiy-storchaka · 2018-12-22T19:39:51Z

Misc/NEWS.d/next/Core and Builtins/2018-12-22-22-19-51.bpo-35560.9vMWSP.rst

@@ -0,0 +1,3 @@
+Removed an incorrect assert statement expecting `min_width`  >= 0 in


min_width and _PyUnicode_InsertThousandsGrouping are implementation details. Describe the issue in terms of Python.

serhiy-storchaka · 2018-12-22T19:43:20Z

Misc/NEWS.d/next/Core and Builtins/2018-12-22-22-19-51.bpo-35560.9vMWSP.rst

@@ -0,0 +1,3 @@
+Removed an incorrect assert statement expecting `min_width`  >= 0 in
+`_PyUnicode_InsertThousandsGrouping` that caused segfault in :func:`format`
+in debug builds for certain cases. Patch by Karthikeyan Singaravelan.


What are certain cases? This is a concrete case: floating point formatting with zero padding and small width.

Is the below content okay? It will be helpful if you have any in mind so that I can use it since I don't know how to explain it without the internals in the news entry.

Removed an incorrect assert statement that caused segfault in :func:`format` in debug builds for floating point formatting with zero padding and small width. Patch by Karthikeyan Singaravelan.

It LGTM, but native speakers can suggest better wording.

tirkarthi · 2018-12-22T20:27:05Z

I printed out min_width of the tests and for negative numbers it's increased by one for the sign bit. I have verified that this passes with a debug build locally and 3.7.1 to make sure there is no change in the output.

./python.exe -c "format(123.0, '00')" # min_width -2 , segfaults
./python.exe -c "format(123.34, '00f')" # min_width -7 , segfaults
./python.exe -c "format(123.34, '00e')" # min_width -11 , segfaults
./python.exe -c "format(123.34, '00g')" # min_width -3 , segfaults
./python.exe -c "format(123.34, '00.10f')" # min_width -11 , segfaults
./python.exe -c "format(123.34, '00.10e')" # min_width -15 , segfaults
./python.exe -c "format(123.34, '00.10g')" # min_width -3 , segfaults
./python.exe -c "format(123.34, '01f')" # min_width -6 , segfaults

./python.exe -c "format(-123.0, '00')" # min_width -3 , segfaults
./python.exe -c "format(-123.34, '00f')" # min_width -8, segfaults
./python.exe -c "format(-123.34, '00e')" # min_width -12, segfaults
./python.exe -c "format(-123.34, '00g')" # min_width -4, segfaults
./python.exe -c "format(-123.34, '00.10f')" # min_width -12 , segfaults
./python.exe -c "format(-123.34, '00.10e')" # min_width -16 , segfaults
./python.exe -c "format(-123.34, '00.10g')" # min_width -4 , segfaults
./python.exe -c "format(-123.34, '01f')" # min_width -7 , segfaults

tirkarthi · 2019-01-05T06:08:01Z

@vstinner Friendly ping if you can review this or if you have a better fix in mind since you wrote the original version I will be happy to close the PR.

vstinner · 2019-01-06T22:08:33Z

Objects/unicodeobject.c

@@ -9381,6 +9381,7 @@ _PyUnicode_InsertThousandsGrouping(
    PyObject *thousands_sep,
    Py_UCS4 *maxchar)
 {
+    min_width = Py_MAX(0, min_width);


I would prefer to fix the caller of this function instead: Python/formatter_unicode.c.

Thanks Victor. Few clarifications :

By fixing the caller do you mean min_width as in min_width = Py_MAX(0, spec->n_min_width); and passing it to _PyUnicode_InsertThousandsGrouping from Python/formatter_unicode.c without modifying spec->n_min_width ? There is an explicit comment that spec->n_min_width and I don't want to mutate the spec->n_min_width .

_PyUnicode_InsertThousandsGrouping is used in two places in formatter_unicode.c and does both places need this change ?

Nevermind, I changed my mind and "min_width = Py_MAX(0, min_width);" in _PyUnicode_InsertThousandsGrouping() is just fine.

bedevere-bot · 2019-01-06T22:08:40Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vstinner · 2019-01-07T14:15:22Z

Objects/unicodeobject.c

@@ -9381,6 +9381,7 @@ _PyUnicode_InsertThousandsGrouping(
    PyObject *thousands_sep,
    Py_UCS4 *maxchar)
 {
+    min_width = Py_MAX(0, min_width);


Nevermind, I changed my mind and "min_width = Py_MAX(0, min_width);" in _PyUnicode_InsertThousandsGrouping() is just fine.

vstinner · 2019-01-07T14:17:59Z

Misc/NEWS.d/next/Core and Builtins/2018-12-22-22-19-51.bpo-35560.9vMWSP.rst

@@ -0,0 +1,3 @@
+Removed an incorrect assert statement that caused segfault in :func:`format`
+in debug builds for floating point formatting with zero padding and small width.


You should mention that the only the "n" format is impacted by the bug. I propose:

Fix an assertion error in :func:format in debug build for floating point formatting with "n" format, zero padding and small width. Release build is not impacted. Patch by Karthikeyan Singaravelan.

Thanks Victor for the suggestion I have reworded the entry.

miss-islington · 2019-01-07T15:09:17Z

Thanks @tirkarthi for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

vstinner · 2019-01-07T15:09:45Z

vstinner removed the needs backport to 3.6 label a minute ago

Python 3.6 no longer accepts bugfixes: https://devguide.python.org/#status-of-python-branches

bedevere-bot · 2019-01-07T15:09:58Z

GH-11458 is a backport of this pull request to the 3.7 branch.

Fix an assertion error in format() in debug build for floating point formatting with "n" format, zero padding and small width. Release build is not impacted. Patch by Karthikeyan Singaravelan. (cherry picked from commit 3f7983a) Co-authored-by: Xtreak <[email protected]>

tirkarthi · 2019-01-07T15:13:06Z

Thanks @vstinner and @serhiy-storchaka for the review.

Fix an assertion error in format() in debug build for floating point formatting with "n" format, zero padding and small width. Release build is not impacted. Patch by Karthikeyan Singaravelan. (cherry picked from commit 3f7983a) Co-authored-by: Xtreak <[email protected]>

charris · 2019-08-18T14:55:13Z

Can this be backported to 3.6? It is causing problems on travisCI with pytest. Note that 3.6 is the default python3 on Ubuntu bionic.

tirkarthi · 2019-08-18T15:44:28Z

I guess this was a problem only with debug builds that has assert statements and as per discussion release builds are not affected. Can you please add a log of travis build that causes this issue in 3.6?

charris · 2019-08-18T16:20:43Z

See https://travis-ci.org/numpy/numpy/jobs/573310511. NumPy is running python3.6-dbg builds on travisCI bionic as one of our CI tests.

charris · 2019-08-18T16:24:21Z

Note that this can affect all users running pytest with the --durations flag on python3.6-dbg, it isn't NumPy specific.

tirkarthi · 2019-08-19T01:53:41Z

Thanks for the details. This fix was made during 3.6.8 RC1 which was the last bugfix for 3.6. It is in security fixes mode so I will defer to core devs if we need to backport this.

vstinner · 2019-08-19T11:40:25Z

Can this be backported to 3.6? It is causing problems on travisCI with pytest. Note that 3.6 is the default python3 on Ubuntu bionic.

Python 3.6 no longer accept bugfixes anymore: https://devguide.python.org/#status-of-python-branches

@ned-deily is the Python 3.6 release manager. Ned: would you be ok to backport this bugfix which cause practical issues on Travis CI?

@charris: Even if Python 3.6 is fixed, I'm not sure that Travis CI is going to get the fix yet. It may take a few weeks or a few months until Travis CI will get the fix. Maybe you should modify pytest tests to skip the test on Python 3.6?

charris · 2019-08-19T12:56:26Z

Thanks for considering a backport.

The workaround for us at this point is to not use --durations. It is a nice thing to have, but not critical, and two years from now the testing might move on to 3.7 :) Another possibility is for pytest to rewrite the format expression to not cause the error. I don't why it causes a problem, so any help in that regard would be welcome. See pytest-dev/pytest#5763, which also shows the troublesome line of code.

ned-deily · 2019-08-19T19:42:05Z

As Victor notes, this issue falls outside the criteria for a security-fix branch like 3.6 and, even if made, a backport to 3.6 is unlikely to make a practical difference to your CI testing any time soon. I think it would be better to concentrate on avoiding the problem in your project’s code since that’s something you control.

miss-islington · 2020-11-10T18:57:49Z

Thanks @tirkarthi for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.6.
🐍🍒⛏🤖

bedevere-bot · 2020-11-10T18:58:01Z

GH-23231 is a backport of this pull request to the 3.6 branch.

Fix an assertion error in format() in debug build for floating point formatting with "n" format, zero padding and small width. Release build is not impacted. Patch by Karthikeyan Singaravelan. (cherry picked from commit 3f7983a) Co-authored-by: Xtreak <[email protected]>

) (GH-10720) Fix str.format(), float.__format__() and complex.__format__() methods for non-ASCII decimal point when using the "n" formatter. Rewrite _PyUnicode_InsertThousandsGrouping(): it now requires a _PyUnicodeWriter object for the buffer and a Python str object for digits. (cherry picked from commit 59423e3) (cherry picked from commit 6f5fa1b)

) Fix an assertion error in format() in debug build for floating point formatting with "n" format, zero padding and small width. Release build is not impacted. Patch by Karthikeyan Singaravelan. (cherry picked from commit 3f7983a) Co-authored-by: Xtreak <[email protected]>

tirkarthi added 2 commits December 22, 2018 22:18

Remove incorrect assert and add tests

877ffb2

Add news entry

d930f32

the-knights-who-say-ni added the CLA signed label Dec 22, 2018

bedevere-bot added the awaiting review label Dec 22, 2018

serhiy-storchaka reviewed Dec 22, 2018

View reviewed changes

Default to 0 when min_width is negative. Add more tests and fix news …

5837592

…entry.

serhiy-storchaka reviewed Dec 22, 2018

View reviewed changes

serhiy-storchaka added type-bug An unexpected behavior, bug, or error needs backport to 3.6 labels Dec 22, 2018

serhiy-storchaka reviewed Dec 22, 2018

View reviewed changes

serhiy-storchaka requested review from mdickinson and ericvsmith December 22, 2018 19:56

More tests and note segfault scenario in news entry

3405406

vstinner requested changes Jan 6, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Jan 6, 2019

vstinner reviewed Jan 7, 2019

View reviewed changes

Reword NEWS entry

eceb18e

serhiy-storchaka approved these changes Jan 7, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting changes labels Jan 7, 2019

vstinner removed the needs backport to 3.6 label Jan 7, 2019

vstinner merged commit 3f7983a into python:master Jan 7, 2019

bedevere-bot removed the awaiting merge label Jan 7, 2019

bedevere-bot removed the needs backport to 3.7 label Jan 7, 2019

gpshead added the needs backport to 3.6 label Nov 10, 2020

bedevere-bot removed the needs backport to 3.6 label Nov 10, 2020

		@@ -0,0 +1,2 @@
		Removed incorrect assert statement that caused segfault in debug builds.

		@@ -0,0 +1,3 @@
		Removed an incorrect assert statement expecting `min_width` >= 0 in

		@@ -0,0 +1,3 @@
		Removed an incorrect assert statement that caused segfault in :func:`format`
		in debug builds for floating point formatting with zero padding and small width.

Uh oh!

bpo-35560: Remove incorrect assert statement that caused segfault in debug builds #11288

bpo-35560: Remove incorrect assert statement that caused segfault in debug builds #11288

Uh oh!

Conversation

tirkarthi commented Dec 22, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tirkarthi commented Dec 22, 2018

Uh oh!

tirkarthi commented Jan 5, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Jan 6, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Jan 7, 2019

Uh oh!

vstinner commented Jan 7, 2019

Uh oh!

bedevere-bot commented Jan 7, 2019

Uh oh!

tirkarthi commented Jan 7, 2019

Uh oh!

charris commented Aug 18, 2019

Uh oh!

tirkarthi commented Aug 18, 2019

Uh oh!

charris commented Aug 18, 2019

Uh oh!

charris commented Aug 18, 2019

Uh oh!

tirkarthi commented Aug 19, 2019

Uh oh!

vstinner commented Aug 19, 2019

Uh oh!

charris commented Aug 19, 2019

Uh oh!

ned-deily commented Aug 19, 2019

Uh oh!

miss-islington commented Nov 10, 2020

Uh oh!

bedevere-bot commented Nov 10, 2020

Uh oh!

Uh oh!

tirkarthi commented Dec 22, 2018 •

edited by bedevere-bot

Loading