Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 1369f8a

Browse files
ucoderyucodery
committed
Address CVE-2020-10735 for long objects
1 parent df2f13c commit 1369f8a

27 files changed

Lines changed: 683 additions & 21 deletions

‎Doc/library/functions.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -788,6 +788,14 @@ section.
788788

789789
The long type is described in :ref:`typesnumeric`.
790790

791+
.. versionchanged:: 2.7.18.6
792+
:class:`long` string inputs and string representations can be limited to
793+
help avoid denial of service attacks. A :exc:`ValueError` is raised when
794+
the limit is exceeded while converting a string *x* to an :class:`long` or
795+
when converting an :class:`long` into a string would exceed the limit.
796+
See the :ref:`integer string conversion length limitation
797+
<int_max_str_digits>` documentation.
798+
791799

792800
.. function:: map(function, iterable, ...)
793801

‎Doc/library/json.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ is a lightweight data interchange format inspired by
1414
`JavaScript <https://en.wikipedia.org/wiki/JavaScript>`_ object literal syntax
1515
(although it is not a strict subset of JavaScript [#rfc-errata]_ ).
1616

17+
.. warning::
18+
Be cautious when parsing JSON data from untrusted sources. A malicious
19+
JSON string may cause the decoder to consume considerable CPU and memory
20+
resources. Limiting the size of data to be parsed is recommended.
21+
1722
:mod:`json` exposes an API familiar to users of the standard library
1823
:mod:`marshal` and :mod:`pickle` modules.
1924

@@ -249,6 +254,12 @@ Basic Usage
249254
be used to use another datatype or parser for JSON integers
250255
(e.g. :class:`float`).
251256

257+
.. versionchanged:: 2.7.18.6
258+
The default *parse_int* of :func:`int` now limits the maximum length of
259+
the integer string via the interpreter's :ref:`integer string
260+
conversion length limitation <int_max_str_digits>` to help avoid denial
261+
of service attacks.
262+
252263
*parse_constant*, if specified, will be called with one of the following
253264
strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``.
254265
This can be used to raise an exception if invalid JSON numbers

‎Doc/library/stdtypes.rst‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -521,6 +521,13 @@ class`. float also has the following additional methods.
521521

522522
.. versionadded:: 2.6
523523

524+
.. note::
525+
526+
The values returned by ``as_integer_ratio()`` can be huge. Attempts
527+
to render such integers into decimal strings may bump into the
528+
:ref:`integer string conversion length limitation
529+
<int_max_str_digits>`.
530+
524531
.. method:: float.is_integer()
525532

526533
Return ``True`` if the float instance is finite with integral
@@ -3190,6 +3197,167 @@ The following attributes are only supported by :term:`new-style class`\ es.
31903197
[<type 'bool'>]
31913198

31923199

3200+
.. _int_max_str_digits:
3201+
3202+
Integer string conversion length limitation
3203+
===========================================
3204+
3205+
CPython has a global limit for converting between :class:`long` and :class:`str`
3206+
or :class:`unicode` to mitigate denial of service attacks. This limit *only* applies
3207+
to decimal or other non-power-of-two number bases. Hexadecimal, octal, and binary
3208+
conversions are unlimited. The limit can be configured.
3209+
3210+
The :class:`long` type in CPython is an arbitrary length number stored in binary
3211+
form (commonly known as a "bignum"). There exists no algorithm that can convert
3212+
a string to a binary integer or a binary integer to a string in linear time,
3213+
*unless* the base is a power of 2. Even the best known algorithms for base 10
3214+
have sub-quadratic complexity. Converting a large value such as ``long('1' *
3215+
500_000)`` can take over a second on a fast CPU.
3216+
3217+
Limiting conversion size offers a practical way to avoid `CVE-2020-10735
3218+
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_.
3219+
3220+
The limit is applied to the number of digit characters in the input or output
3221+
string when a non-linear conversion algorithm would be involved. Trailing *L*s
3222+
and the sign are not counted towards the limit.
3223+
3224+
When an operation would exceed the limit, a :exc:`ValueError` is raised:
3225+
3226+
.. doctest::
3227+
3228+
>>> import sys
3229+
>>> sys.set_int_max_str_digits(4300) # Illustrative, this is the default.
3230+
>>> _ = long('2' * 5432)
3231+
Traceback (most recent call last):
3232+
...
3233+
ValueError: Exceeds the limit (4300) for integer string conversion: value has 5432 digits.
3234+
>>> i = long('2' * 4300)
3235+
>>> len(str(i))
3236+
4300
3237+
>>> i_squared = i*i
3238+
>>> len(str(i_squared))
3239+
Traceback (most recent call last):
3240+
...
3241+
ValueError: Exceeds the limit (4300) for integer string conversion: value has 8599 digits.
3242+
>>> len(hex(i_squared))
3243+
7144
3244+
>>> assert long(hex(i_squared), base=16) == i*i # Hexadecimal is unlimited.
3245+
3246+
The default limit is 4300 digits as provided in
3247+
:data:`sys.long_info.default_max_str_digits <sys.long_info>`.
3248+
The lowest limit that can be configured is 640 digits as provided in
3249+
:data:`sys.long_info.str_digits_check_threshold <sys.long_info>`.
3250+
3251+
Verification:
3252+
3253+
.. doctest::
3254+
3255+
>>> import sys
3256+
>>> assert sys.int_info.default_max_str_digits == 4300, sys.int_info
3257+
>>> assert sys.int_info.str_digits_check_threshold == 640, sys.int_info
3258+
>>> msg = long('578966293710682886880994035146873798396722250538762761564'
3259+
... '9252925514383915483333812743580549779436104706260696366600'
3260+
... '571186405732').to_bytes(53, 'big')
3261+
...
3262+
3263+
.. versionadded:: 2.7.18.6
3264+
3265+
3266+
Affected APIs
3267+
-------------
3268+
3269+
Because int automatically converts to long if the value is larger than
3270+
:data:`sys.maxint` this limitation applies to potentially slow conversions
3271+
between any of :class:`int` or :class:`long` and :class:`str` or :class:`unicode`:
3272+
3273+
* ``int(string)`` with default base 10.
3274+
* ``int(string, base)`` for all bases that are not a power of 2.
3275+
* ``long(string)`` with default base 10.
3276+
* ``long(string, base)`` for all bases that are not a power of 2.
3277+
* ``int(unicode)`` with default base 10.
3278+
* ``int(unicode, base)`` for all bases that are not a power of 2.
3279+
* ``long(unicode)`` with default base 10.
3280+
* ``long(unicode, base)`` for all bases that are not a power of 2.
3281+
* ``str(long)``.
3282+
* ``repr(long)``.
3283+
* ``unicode(long)``.
3284+
* any other string conversion to base 10, for example ``"{}".format(long)``.
3285+
3286+
The limitations do not apply to functions with a linear algorithm:
3287+
3288+
* ``long(string, base)`` with base 2, 4, 8, 16, or 32.
3289+
* :func:`hex`, :func:`oct`, :func:`bin`.
3290+
* :ref:`formatspec` for hex, octal, and binary numbers.
3291+
* :class:`str` to :class:`float`.
3292+
* :class:`str` to :class:`decimal.Decimal`.
3293+
3294+
Configuring the limit
3295+
---------------------
3296+
3297+
Before Python starts up you can use an environment variable to configure the limit:
3298+
3299+
* :envvar:`PYTHONINTMAXSTRDIGITS`, e.g.
3300+
``PYTHONINTMAXSTRDIGITS=640 python`` to set the limit to 640 or
3301+
``PYTHONINTMAXSTRDIGITS=0 python`` to disable the limitation.
3302+
* :data:`sys.flags.long_max_str_digits` contains the value of
3303+
:envvar:`PYTHONINTMAXSTRDIGITS`. A value of *-1* indicates that none was set,
3304+
thus a value of :data:`sys.int_info.default_max_str_digits` was used during
3305+
initialization.
3306+
3307+
From code, you can inspect the current limit and set a new one using these
3308+
:mod:`sys` APIs:
3309+
3310+
* :func:`sys.get_int_max_str_digits` and :func:`sys.set_int_max_str_digits` are
3311+
a getter and setter for the interpreter-wide limit.
3312+
3313+
Information about the default and minimum can be found in :attr:`sys.long_info`:
3314+
3315+
* :data:`sys.long_info.default_max_str_digits <sys.long_info>` is the compiled-in
3316+
default limit.
3317+
* :data:`sys.long_info.str_digits_check_threshold <sys.long_info>` is the lowest
3318+
accepted value for the limit (other than 0 which disables it).
3319+
3320+
.. versionadded:: 2.7.18.6
3321+
3322+
.. caution::
3323+
3324+
Setting a low limit *can* lead to problems. While rare, code exists that
3325+
contains integer constants in decimal in their source that exceed the
3326+
minimum threshold. A consequence of setting the limit is that Python source
3327+
code containing decimal integer literals longer than the limit will
3328+
encounter an error during parsing, usually at startup time or import time or
3329+
even at installation time - anytime an up to date ``.pyc`` does not already
3330+
exist for the code. A workaround for source that contains such large
3331+
constants is to convert them to ``0x`` hexadecimal form as it has no limit.
3332+
3333+
Test your application thoroughly if you use a low limit. Ensure your tests
3334+
run with the limit set early via the environment so that it applies during
3335+
startup and even during any installation step that may invoke Python to
3336+
precompile ``.py`` sources to ``.pyc`` files.
3337+
3338+
Recommended configuration
3339+
-------------------------
3340+
3341+
The default :data:`sys.long_info.default_max_str_digits` is expected to be
3342+
reasonable for most applications. If your application requires a different
3343+
limit, set it from your main entry point using Python version agnostic code as
3344+
these APIs were ported from the original fix in version 3.12.
3345+
3346+
Example::
3347+
3348+
>>> import sys
3349+
>>> if hasattr(sys, "set_int_max_str_digits"):
3350+
... upper_bound = 68000
3351+
... lower_bound = 4004
3352+
... current_limit = sys.get_int_max_str_digits()
3353+
... if current_limit == 0 or current_limit > upper_bound:
3354+
... sys.set_int_max_str_digits(upper_bound)
3355+
... elif current_limit < lower_bound:
3356+
... sys.set_int_max_str_digits(lower_bound)
3357+
3358+
If you need to disable it entirely, set it to ``0``.
3359+
3360+
31933361
.. rubric:: Footnotes
31943362

31953363
.. [1] Additional information on these special methods may be found in the Python

‎Doc/library/sys.rst‎

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,14 @@ always available.
431431
an argument to :func:`getrefcount`.
432432

433433

434+
.. function:: get_int_max_str_digits()
435+
436+
Returns the current value for the :ref:`integer string conversion length
437+
limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`.
438+
439+
.. versionadded:: 2.7.18.6
440+
441+
434442
.. function:: getrecursionlimit()
435443

436444
Return the current value of the recursion limit, the maximum depth of the Python
@@ -603,19 +611,30 @@ always available.
603611

604612
.. tabularcolumns:: |l|L|
605613

606-
+-------------------------+----------------------------------------------+
607-
| Attribute | Explanation |
608-
+=========================+==============================================+
609-
| :const:`bits_per_digit` | number of bits held in each digit. Python |
610-
| | integers are stored internally in base |
611-
| | ``2**long_info.bits_per_digit`` |
612-
+-------------------------+----------------------------------------------+
613-
| :const:`sizeof_digit` | size in bytes of the C type used to |
614-
| | represent a digit |
615-
+-------------------------+----------------------------------------------+
614+
+----------------------------------------+-----------------------------------------------+
615+
| Attribute | Explanation |
616+
+========================================+===============================================+
617+
| :const:`bits_per_digit` | number of bits held in each digit. Python |
618+
| | integers are stored internally in base |
619+
| | ``2**int_info.bits_per_digit`` |
620+
+----------------------------------------+-----------------------------------------------+
621+
| :const:`sizeof_digit` | size in bytes of the C type used to |
622+
| | represent a digit |
623+
+----------------------------------------+-----------------------------------------------+
624+
| :const:`default_max_str_digits` | default value for |
625+
| | :func:`sys.get_int_max_str_digits` when it |
626+
| | is not otherwise explicitly configured. |
627+
+----------------------------------------+-----------------------------------------------+
628+
| :const:`str_digits_check_threshold` | minimum non-zero value for |
629+
| | :func:`sys.set_int_max_str_digits`, |
630+
| | :envvar:`PYTHONINTMAXSTRDIGITS`. |
631+
+----------------------------------------+-----------------------------------------------+
616632

617633
.. versionadded:: 2.7
618634

635+
.. versionchanged:: 2.7.18.6
636+
Added ``default_max_str_digits`` and ``str_digits_check_threshold``.
637+
619638

620639
.. data:: last_type
621640
last_value
@@ -848,6 +867,15 @@ always available.
848867
.. versionadded:: 2.2
849868

850869

870+
.. function:: set_int_max_str_digits(n)
871+
872+
Set the :ref:`integer string conversion length limitation
873+
<int_max_str_digits>` used by this interpreter. See also
874+
:func:`get_int_max_str_digits`.
875+
876+
.. versionadded:: 2.7.18.6
877+
878+
851879
.. function:: setprofile(profilefunc)
852880

853881
.. index::

‎Doc/library/test.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,16 @@ The :mod:`test.support` module defines the following functions:
443443
.. versionadded:: 2.7
444444

445445

446+
.. function:: adjust_int_max_str_digits(max_digits)
447+
448+
This function returns a context manager that will change the global
449+
:func:`sys.set_int_max_str_digits` setting for the duration of the
450+
context to allow execution of test code that needs a different limit
451+
on the number of digits when converting between an integer and string.
452+
453+
.. versionadded:: 2.7.18.6
454+
455+
446456
The :mod:`test.support` module defines the following classes:
447457

448458
.. class:: TransientResource(exc[, **kwargs])

‎Doc/library/xmlrpclib.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -557,6 +557,12 @@ Convenience Functions
557557
.. versionchanged:: 2.5
558558
The *use_datetime* flag was added.
559559

560+
.. versionchanged:: 2.7.18.6
561+
The default *parse_int* of :func:`int` now limits the maximum length of
562+
the integer string via the interpreter's :ref:`integer string
563+
conversion length limitation <int_max_str_digits>` to help avoid denial
564+
of service attacks.
565+
560566

561567
.. _xmlrpc-client-example:
562568

‎Doc/using/cmdline.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -638,6 +638,15 @@ conflict.
638638

639639
.. versionadded:: 2.7.12
640640

641+
642+
.. envvar:: PYTHONINTMAXSTRDIGITS
643+
644+
If this variable is set to an integer, it is used to configure the
645+
interpreter's global :ref:`integer string conversion length limitation
646+
<int_max_str_digits>`.
647+
648+
.. versionadded:: 2.7.18.6
649+
641650
Debug-mode variables
642651
~~~~~~~~~~~~~~~~~~~~
643652

‎Doc/whatsnew/2.7.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -884,6 +884,18 @@ Some smaller changes made to the core Python language are:
884884
now only cleared if no one else is holding a reference to the
885885
dictionary (:issue:`7140`).
886886

887+
* Converting between :class:`int` or :class:`long` and :class:`str` or
888+
:class:`unicode` in bases other than 2 (binary), 4, 8 (octal), 16
889+
(hexadecimal), or 32 such as base 10 (decimal) now raises a
890+
:exc:`ValueError` if the number of digits in string form is above a
891+
limit to avoid potential denial of service attacks due to the
892+
algorithmic complexity. This is a mitigation for `CVE-2020-10735
893+
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_.
894+
This limit can be configured or disabled by environment variable or
895+
:mod:`sys` APIs. See the :ref:`integer string conversion length
896+
limitation <int_max_str_digits>` documentation. The default limit
897+
is 4300 digits in string form.
898+
887899
.. ======================================================================
888900
889901
.. _new-27-interpreter:

‎Include/longobject.h‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,32 @@ PyAPI_FUNC(PyObject *) _PyLong_FormatAdvanced(PyObject *obj,
129129
char *format_spec,
130130
Py_ssize_t format_spec_len);
131131

132+
#define _MAX_STR_DIGITS_ERROR_FMT "Exceeds the limit (%d) for integer string conversion: value has %zd digits"
133+
/*
134+
* Default long base conversion size limitation: Denial of Service prevention.
135+
*
136+
* Chosen such that this isn't wildly slow on modern hardware
137+
* 4300 decimal digits fits a ~14284 bit number.
138+
*/
139+
#define _PY_LONG_DEFAULT_MAX_STR_DIGITS 4300
140+
/*
141+
* Threshold for max digits check. For performance reasons long() and
142+
* long.__str__() don't checks values that are smaller than this
143+
* threshold. Acts as a guaranteed minimum size limit for bignums that
144+
* applications can expect from CPython.
145+
*
146+
* "640 digits should be enough for anyone." - gps
147+
* fits a ~2126 bit decimal number.
148+
*/
149+
#define _PY_LONG_MAX_STR_DIGITS_THRESHOLD 640
150+
151+
#if ((_PY_LONG_DEFAULT_MAX_STR_DIGITS != 0) && \
152+
(_PY_LONG_DEFAULT_MAX_STR_DIGITS < _PY_LONG_MAX_STR_DIGITS_THRESHOLD))
153+
# error "_PY_LONG_DEFAULT_MAX_STR_DIGITS smaller than threshold."
154+
#endif
155+
156+
int Py_LongMaxStrDigits;
157+
132158
#ifdef __cplusplus
133159
}
134160
#endif

‎Include/pydebug.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ PyAPI_DATA(int) _Py_QnewFlag;
2727
/* Warn about 3.x issues */
2828
PyAPI_DATA(int) Py_Py3kWarningFlag;
2929
PyAPI_DATA(int) Py_HashRandomizationFlag;
30+
PyAPI_DATA(int) Py_LongMaxStrDigitsFlag;
3031

3132
/* this is a wrapper around getenv() that pays attention to
3233
Py_IgnoreEnvironmentFlag. It should be used for getting variables like

0 commit comments

Comments
 (0)