Harden eval_expr - #1744
Merged
Merged
Conversation
Contributor
Author
|
cc @adrinjalali. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1744 +/- ##
==========================================
- Coverage 95.68% 94.24% -1.44%
==========================================
Files 46 46
Lines 7885 7912 +27
==========================================
- Hits 7545 7457 -88
- Misses 340 455 +115 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Thomas Moreau <[email protected]>
adrinjalali
approved these changes
Sep 23, 2025
Sreekant13
added a commit
to Sreekant13/joblib
that referenced
this pull request
Sep 1, 2026
The magnitude limit only inspects a value once it exists, so a power whose operands are individually within the limit was computed in full before being rejected. Evaluating 999999**999999 that way costs several seconds of CPU and builds a number with millions of digits, which is the kind of expression the limit was added to guard against in joblib#1744. A power is now rejected up front when it cannot possibly fit: for any base of magnitude 2 or more, base ** exponent is at least 2 ** exponent, so an exponent above the bit length of the limit can never produce a value under it. Only integer powers are checked, since a float power overflows cheaply and already reports an error. This also fixes the message for a large result. CPython caps integer to string conversion at sys.get_int_max_str_digits() digits, so formatting the value into "Numeric literal ... is too large" raised instead, and pre_dispatch of "10**5000" surfaced a complaint about integer string conversion along with advice to call sys.set_int_max_str_digits(). Both cases reproduce through Parallel(pre_dispatch=...).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
According to the
ast.parsedocumentation, long and complex expressions can cause a Python interpreter crash. While I did not find an example of such an expression, I think we can proactively restrict the expressions that we deem valid to avoid those problems, to more safely use joblib in case the program accepts user-settablepre_dispatchexpressions, for instance.While investigating this, I also discovered that parsing expression with large integer values such as
9**9**9**9can take ages to evaluate (DoS) and that allocating large strings from short expressions such as' ' * 10**10can cause the Python process to get OOM'ed by the operating system.This PR makes the following changes: