Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit f8b71da

Browse files
authored
[3.11] gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96500)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds. This PR comes fresh from a pile of work done in our private PSRT security response team repo. This backports #96499 aka 511ca94 Signed-off-by: Christian Heimes [Red Hat] <[email protected]> Tons-of-polishing-up-by: Gregory P. Smith [Google] <[email protected]> Reviews via the private PSRT repo via many others (see the NEWS entry in the PR). <!-- gh-issue-number: gh-95778 --> * Issue: gh-95778 <!-- /gh-issue-number --> I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#).
1 parent 57116d5 commit f8b71da

27 files changed

+775
-23
lines changed

Doc/data/python3.11.abi

+6-3
Original file line numberDiff line numberDiff line change
@@ -1639,7 +1639,7 @@
16391639
<elf-symbol name='_PyNotImplemented_Type' size='408' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
16401640
<elf-symbol name='_PyOS_ReadlineTState' size='8' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
16411641
<elf-symbol name='_PyParser_TokenNames' size='520' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
1642-
<elf-symbol name='_PyRuntime' size='166680' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
1642+
<elf-symbol name='_PyRuntime' size='166688' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
16431643
<elf-symbol name='_PySet_Dummy' size='8' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
16441644
<elf-symbol name='_PyWeakref_CallableProxyType' size='408' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
16451645
<elf-symbol name='_PyWeakref_ProxyType' size='408' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
@@ -1825,7 +1825,7 @@
18251825
</class-decl>
18261826
<typedef-decl name='PyThreadState' type-id='type-id-9' filepath='./Include/pytypedefs.h' line='24' column='1' id='type-id-25'/>
18271827
<pointer-type-def type-id='type-id-25' size-in-bits='64' id='type-id-10'/>
1828-
<class-decl name='_is' size-in-bits='861952' is-struct='yes' visibility='default' filepath='./Include/internal/pycore_interp.h' line='78' column='1' id='type-id-26'>
1828+
<class-decl name='_is' size-in-bits='862016' is-struct='yes' visibility='default' filepath='./Include/internal/pycore_interp.h' line='78' column='1' id='type-id-26'>
18291829
<data-member access='public' layout-offset-in-bits='0'>
18301830
<var-decl name='next' type-id='type-id-11' visibility='default' filepath='./Include/internal/pycore_interp.h' line='80' column='1'/>
18311831
</data-member>
@@ -1971,6 +1971,9 @@
19711971
<var-decl name='callable_cache' type-id='type-id-51' visibility='default' filepath='./Include/internal/pycore_interp.h' line='177' column='1'/>
19721972
</data-member>
19731973
<data-member access='public' layout-offset-in-bits='859072'>
1974+
<var-decl name='int_max_str_digits' type-id='type-id-8' visibility='default' filepath='./Include/internal/pycore_interp.h' line='179' column='1'/>
1975+
</data-member>
1976+
<data-member access='public' layout-offset-in-bits='859136'>
19741977
<var-decl name='_initial_thread' type-id='type-id-25' visibility='default' filepath='./Include/internal/pycore_interp.h' line='192' column='1'/>
19751978
</data-member>
19761979
</class-decl>
@@ -1995,7 +1998,7 @@
19951998
<typedef-decl name='uint64_t' type-id='type-id-55' filepath='/usr/include/x86_64-linux-gnu/bits/stdint-uintn.h' line='27' column='1' id='type-id-19'/>
19961999
<type-decl name='long int' size-in-bits='64' id='type-id-53'/>
19972000
<typedef-decl name='size_t' type-id='type-id-16' filepath='/usr/lib/gcc/x86_64-linux-gnu/9/include/stddef.h' line='209' column='1' id='type-id-54'/>
1998-
<class-decl name='pyruntimestate' size-in-bits='1333440' is-struct='yes' visibility='default' filepath='./Include/internal/pycore_runtime.h' line='59' column='1' id='type-id-56'>
2001+
<class-decl name='pyruntimestate' size-in-bits='1333504' is-struct='yes' visibility='default' filepath='./Include/internal/pycore_runtime.h' line='59' column='1' id='type-id-56'>
19992002
<data-member access='public' layout-offset-in-bits='0'>
20002003
<var-decl name='_initialized' type-id='type-id-8' visibility='default' filepath='./Include/internal/pycore_runtime.h' line='64' column='1'/>
20012004
</data-member>

Doc/library/functions.rst

+7
Original file line numberDiff line numberDiff line change
@@ -910,6 +910,13 @@ are always available. They are listed here in alphabetical order.
910910
.. versionchanged:: 3.11
911911
The delegation to :meth:`__trunc__` is deprecated.
912912

913+
.. versionchanged:: 3.11
914+
:class:`int` string inputs and string representations can be limited to
915+
help avoid denial of service attacks. A :exc:`ValueError` is raised when
916+
the limit is exceeded while converting a string *x* to an :class:`int` or
917+
when converting an :class:`int` into a string would exceed the limit.
918+
See the :ref:`integer string conversion length limitation
919+
<int_max_str_digits>` documentation.
913920

914921
.. function:: isinstance(object, classinfo)
915922

Doc/library/json.rst

+11
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@ is a lightweight data interchange format inspired by
1818
`JavaScript <https://en.wikipedia.org/wiki/JavaScript>`_ object literal syntax
1919
(although it is not a strict subset of JavaScript [#rfc-errata]_ ).
2020

21+
.. warning::
22+
Be cautious when parsing JSON data from untrusted sources. A malicious
23+
JSON string may cause the decoder to consume considerable CPU and memory
24+
resources. Limiting the size of data to be parsed is recommended.
25+
2126
:mod:`json` exposes an API familiar to users of the standard library
2227
:mod:`marshal` and :mod:`pickle` modules.
2328

@@ -248,6 +253,12 @@ Basic Usage
248253
be used to use another datatype or parser for JSON integers
249254
(e.g. :class:`float`).
250255

256+
.. versionchanged:: 3.11
257+
The default *parse_int* of :func:`int` now limits the maximum length of
258+
the integer string via the interpreter's :ref:`integer string
259+
conversion length limitation <int_max_str_digits>` to help avoid denial
260+
of service attacks.
261+
251262
*parse_constant*, if specified, will be called with one of the following
252263
strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``.
253264
This can be used to raise an exception if invalid JSON numbers

Doc/library/stdtypes.rst

+166
Original file line numberDiff line numberDiff line change
@@ -622,6 +622,13 @@ class`. float also has the following additional methods.
622622
:exc:`OverflowError` on infinities and a :exc:`ValueError` on
623623
NaNs.
624624

625+
.. note::
626+
627+
The values returned by ``as_integer_ratio()`` can be huge. Attempts
628+
to render such integers into decimal strings may bump into the
629+
:ref:`integer string conversion length limitation
630+
<int_max_str_digits>`.
631+
625632
.. method:: float.is_integer()
626633

627634
Return ``True`` if the float instance is finite with integral
@@ -5456,6 +5463,165 @@ types, where they are relevant. Some of these are not reported by the
54565463
[<class 'bool'>]
54575464

54585465

5466+
.. _int_max_str_digits:
5467+
5468+
Integer string conversion length limitation
5469+
===========================================
5470+
5471+
CPython has a global limit for converting between :class:`int` and :class:`str`
5472+
to mitigate denial of service attacks. This limit *only* applies to decimal or
5473+
other non-power-of-two number bases. Hexadecimal, octal, and binary conversions
5474+
are unlimited. The limit can be configured.
5475+
5476+
The :class:`int` type in CPython is an abitrary length number stored in binary
5477+
form (commonly known as a "bignum"). There exists no algorithm that can convert
5478+
a string to a binary integer or a binary integer to a string in linear time,
5479+
*unless* the base is a power of 2. Even the best known algorithms for base 10
5480+
have sub-quadratic complexity. Converting a large value such as ``int('1' *
5481+
500_000)`` can take over a second on a fast CPU.
5482+
5483+
Limiting conversion size offers a practical way to avoid `CVE-2020-10735
5484+
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_.
5485+
5486+
The limit is applied to the number of digit characters in the input or output
5487+
string when a non-linear conversion algorithm would be involved. Underscores
5488+
and the sign are not counted towards the limit.
5489+
5490+
When an operation would exceed the limit, a :exc:`ValueError` is raised:
5491+
5492+
.. doctest::
5493+
5494+
>>> import sys
5495+
>>> sys.set_int_max_str_digits(4300) # Illustrative, this is the default.
5496+
>>> _ = int('2' * 5432)
5497+
Traceback (most recent call last):
5498+
...
5499+
ValueError: Exceeds the limit (4300) for integer string conversion: value has 5432 digits.
5500+
>>> i = int('2' * 4300)
5501+
>>> len(str(i))
5502+
4300
5503+
>>> i_squared = i*i
5504+
>>> len(str(i_squared))
5505+
Traceback (most recent call last):
5506+
...
5507+
ValueError: Exceeds the limit (4300) for integer string conversion: value has 8599 digits.
5508+
>>> len(hex(i_squared))
5509+
7144
5510+
>>> assert int(hex(i_squared), base=16) == i*i # Hexadecimal is unlimited.
5511+
5512+
The default limit is 4300 digits as provided in
5513+
:data:`sys.int_info.default_max_str_digits <sys.int_info>`.
5514+
The lowest limit that can be configured is 640 digits as provided in
5515+
:data:`sys.int_info.str_digits_check_threshold <sys.int_info>`.
5516+
5517+
Verification:
5518+
5519+
.. doctest::
5520+
5521+
>>> import sys
5522+
>>> assert sys.int_info.default_max_str_digits == 4300, sys.int_info
5523+
>>> assert sys.int_info.str_digits_check_threshold == 640, sys.int_info
5524+
>>> msg = int('578966293710682886880994035146873798396722250538762761564'
5525+
... '9252925514383915483333812743580549779436104706260696366600'
5526+
... '571186405732').to_bytes(53, 'big')
5527+
...
5528+
5529+
.. versionadded:: 3.11
5530+
5531+
Affected APIs
5532+
-------------
5533+
5534+
The limition only applies to potentially slow conversions between :class:`int`
5535+
and :class:`str` or :class:`bytes`:
5536+
5537+
* ``int(string)`` with default base 10.
5538+
* ``int(string, base)`` for all bases that are not a power of 2.
5539+
* ``str(integer)``.
5540+
* ``repr(integer)``
5541+
* any other string conversion to base 10, for example ``f"{integer}"``,
5542+
``"{}".format(integer)``, or ``b"%d" % integer``.
5543+
5544+
The limitations do not apply to functions with a linear algorithm:
5545+
5546+
* ``int(string, base)`` with base 2, 4, 8, 16, or 32.
5547+
* :func:`int.from_bytes` and :func:`int.to_bytes`.
5548+
* :func:`hex`, :func:`oct`, :func:`bin`.
5549+
* :ref:`formatspec` for hex, octal, and binary numbers.
5550+
* :class:`str` to :class:`float`.
5551+
* :class:`str` to :class:`decimal.Decimal`.
5552+
5553+
Configuring the limit
5554+
---------------------
5555+
5556+
Before Python starts up you can use an environment variable or an interpreter
5557+
command line flag to configure the limit:
5558+
5559+
* :envvar:`PYTHONINTMAXSTRDIGITS`, e.g.
5560+
``PYTHONINTMAXSTRDIGITS=640 python3`` to set the limit to 640 or
5561+
``PYTHONINTMAXSTRDIGITS=0 python3`` to disable the limitation.
5562+
* :option:`-X int_max_str_digits <-X>`, e.g.
5563+
``python3 -X int_max_str_digits=640``
5564+
* :data:`sys.flags.int_max_str_digits` contains the value of
5565+
:envvar:`PYTHONINTMAXSTRDIGITS` or :option:`-X int_max_str_digits <-X>`.
5566+
If both the env var and the ``-X`` option are set, the ``-X`` option takes
5567+
precedence. A value of *-1* indicates that both were unset, thus a value of
5568+
:data:`sys.int_info.default_max_str_digits` was used during initilization.
5569+
5570+
From code, you can inspect the current limit and set a new one using these
5571+
:mod:`sys` APIs:
5572+
5573+
* :func:`sys.get_int_max_str_digits` and :func:`sys.set_int_max_str_digits` are
5574+
a getter and setter for the interpreter-wide limit. Subinterpreters have
5575+
their own limit.
5576+
5577+
Information about the default and minimum can be found in :attr:`sys.int_info`:
5578+
5579+
* :data:`sys.int_info.default_max_str_digits <sys.int_info>` is the compiled-in
5580+
default limit.
5581+
* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the lowest
5582+
accepted value for the limit (other than 0 which disables it).
5583+
5584+
.. versionadded:: 3.11
5585+
5586+
.. caution::
5587+
5588+
Setting a low limit *can* lead to problems. While rare, code exists that
5589+
contains integer constants in decimal in their source that exceed the
5590+
minimum threshold. A consequence of setting the limit is that Python source
5591+
code containing decimal integer literals longer than the limit will
5592+
encounter an error during parsing, usually at startup time or import time or
5593+
even at installation time - anytime an up to date ``.pyc`` does not already
5594+
exist for the code. A workaround for source that contains such large
5595+
constants is to convert them to ``0x`` hexadecimal form as it has no limit.
5596+
5597+
Test your application thoroughly if you use a low limit. Ensure your tests
5598+
run with the limit set early via the environment or flag so that it applies
5599+
during startup and even during any installation step that may invoke Python
5600+
to precompile ``.py`` sources to ``.pyc`` files.
5601+
5602+
Recommended configuration
5603+
-------------------------
5604+
5605+
The default :data:`sys.int_info.default_max_str_digits` is expected to be
5606+
reasonable for most applications. If your application requires a different
5607+
limit, set it from your main entry point using Python version agnostic code as
5608+
these APIs were added in security patch releases in versions before 3.11.
5609+
5610+
Example::
5611+
5612+
>>> import sys
5613+
>>> if hasattr(sys, "set_int_max_str_digits"):
5614+
... upper_bound = 68000
5615+
... lower_bound = 4004
5616+
... current_limit = sys.get_int_max_str_digits()
5617+
... if current_limit == 0 or current_limit > upper_bound:
5618+
... sys.set_int_max_str_digits(upper_bound)
5619+
... elif current_limit < lower_bound:
5620+
... sys.set_int_max_str_digits(lower_bound)
5621+
5622+
If you need to disable it entirely, set it to ``0``.
5623+
5624+
54595625
.. rubric:: Footnotes
54605626

54615627
.. [1] Additional information on these special methods may be found in the Python

Doc/library/sys.rst

+44-13
Original file line numberDiff line numberDiff line change
@@ -502,9 +502,9 @@ always available.
502502
The :term:`named tuple` *flags* exposes the status of command line
503503
flags. The attributes are read only.
504504

505-
============================= ================================================================
505+
============================= ==============================================================================================================
506506
attribute flag
507-
============================= ================================================================
507+
============================= ==============================================================================================================
508508
:const:`debug` :option:`-d`
509509
:const:`inspect` :option:`-i`
510510
:const:`interactive` :option:`-i`
@@ -521,7 +521,8 @@ always available.
521521
:const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
522522
:const:`utf8_mode` :option:`-X utf8 <-X>`
523523
:const:`safe_path` :option:`-P`
524-
============================= ================================================================
524+
:const:`int_max_str_digits` :option:`-X int_max_str_digits <-X>` (:ref:`integer string conversion length limitation <int_max_str_digits>`)
525+
============================= ==============================================================================================================
525526

526527
.. versionchanged:: 3.2
527528
Added ``quiet`` attribute for the new :option:`-q` flag.
@@ -543,6 +544,9 @@ always available.
543544
.. versionchanged:: 3.11
544545
Added the ``safe_path`` attribute for :option:`-P` option.
545546

547+
.. versionchanged:: 3.11
548+
Added the ``int_max_str_digits`` attribute.
549+
546550

547551
.. data:: float_info
548552

@@ -723,6 +727,13 @@ always available.
723727

724728
.. versionadded:: 3.6
725729

730+
.. function:: get_int_max_str_digits()
731+
732+
Returns the current value for the :ref:`integer string conversion length
733+
limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`.
734+
735+
.. versionadded:: 3.11
736+
726737
.. function:: getrefcount(object)
727738

728739
Return the reference count of the *object*. The count returned is generally one
@@ -996,19 +1007,31 @@ always available.
9961007

9971008
.. tabularcolumns:: |l|L|
9981009

999-
+-------------------------+----------------------------------------------+
1000-
| Attribute | Explanation |
1001-
+=========================+==============================================+
1002-
| :const:`bits_per_digit` | number of bits held in each digit. Python |
1003-
| | integers are stored internally in base |
1004-
| | ``2**int_info.bits_per_digit`` |
1005-
+-------------------------+----------------------------------------------+
1006-
| :const:`sizeof_digit` | size in bytes of the C type used to |
1007-
| | represent a digit |
1008-
+-------------------------+----------------------------------------------+
1010+
+----------------------------------------+-----------------------------------------------+
1011+
| Attribute | Explanation |
1012+
+========================================+===============================================+
1013+
| :const:`bits_per_digit` | number of bits held in each digit. Python |
1014+
| | integers are stored internally in base |
1015+
| | ``2**int_info.bits_per_digit`` |
1016+
+----------------------------------------+-----------------------------------------------+
1017+
| :const:`sizeof_digit` | size in bytes of the C type used to |
1018+
| | represent a digit |
1019+
+----------------------------------------+-----------------------------------------------+
1020+
| :const:`default_max_str_digits` | default value for |
1021+
| | :func:`sys.get_int_max_str_digits` when it |
1022+
| | is not otherwise explicitly configured. |
1023+
+----------------------------------------+-----------------------------------------------+
1024+
| :const:`str_digits_check_threshold` | minimum non-zero value for |
1025+
| | :func:`sys.set_int_max_str_digits`, |
1026+
| | :envvar:`PYTHONINTMAXSTRDIGITS`, or |
1027+
| | :option:`-X int_max_str_digits <-X>`. |
1028+
+----------------------------------------+-----------------------------------------------+
10091029

10101030
.. versionadded:: 3.1
10111031

1032+
.. versionchanged:: 3.11
1033+
Added ``default_max_str_digits`` and ``str_digits_check_threshold``.
1034+
10121035

10131036
.. data:: __interactivehook__
10141037

@@ -1308,6 +1331,14 @@ always available.
13081331

13091332
.. availability:: Unix.
13101333

1334+
.. function:: set_int_max_str_digits(n)
1335+
1336+
Set the :ref:`integer string conversion length limitation
1337+
<int_max_str_digits>` used by this interpreter. See also
1338+
:func:`get_int_max_str_digits`.
1339+
1340+
.. versionadded:: 3.11
1341+
13111342
.. function:: setprofile(profilefunc)
13121343

13131344
.. index::

Doc/library/test.rst

+10
Original file line numberDiff line numberDiff line change
@@ -951,6 +951,16 @@ The :mod:`test.support` module defines the following functions:
951951
.. versionadded:: 3.10
952952

953953

954+
.. function:: adjust_int_max_str_digits(max_digits)
955+
956+
This function returns a context manager that will change the global
957+
:func:`sys.set_int_max_str_digits` setting for the duration of the
958+
context to allow execution of test code that needs a different limit
959+
on the number of digits when converting between an integer and string.
960+
961+
.. versionadded:: 3.11
962+
963+
954964
The :mod:`test.support` module defines the following classes:
955965

956966

0 commit comments

Comments
 (0)