Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Harden eval_expr - #1744

Merged
tomMoral merged 4 commits into
joblib:mainfrom
ogrisel:harden_eval_expr
Sep 23, 2025
Merged

Harden eval_expr #1744
tomMoral merged 4 commits into
joblib:mainfrom
ogrisel:harden_eval_expr

Conversation

@ogrisel

@ogrisel ogrisel commented Sep 22, 2025

Copy link
Copy Markdown
Contributor

According to the ast.parse documentation, long and complex expressions can cause a Python interpreter crash. While I did not find an example of such an expression, I think we can proactively restrict the expressions that we deem valid to avoid those problems, to more safely use joblib in case the program accepts user-settable pre_dispatch expressions, for instance.

While investigating this, I also discovered that parsing expression with large integer values such as 9**9**9**9 can take ages to evaluate (DoS) and that allocating large strings from short expressions such as ' ' * 10**10 can cause the Python process to get OOM'ed by the operating system.

This PR makes the following changes:

  • avoid parsing long string inputs;
  • do not evaluate expressions involving non-numeric literals;
  • do not evaluate expression with large intermediate values.

@ogrisel

ogrisel commented Sep 22, 2025

Copy link
Copy Markdown
Contributor Author

cc @adrinjalali.

@codecov

codecov Bot commented Sep 22, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.24%. Comparing base (8642478) to head (c542a65).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
joblib/_utils.py 90.47% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1744      +/-   ##
==========================================
- Coverage   95.68%   94.24%   -1.44%     
==========================================
  Files          46       46              
  Lines        7885     7912      +27     
==========================================
- Hits         7545     7457      -88     
- Misses        340      455     +115     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tomMoral tomMoral left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread joblib/_utils.py Outdated
Co-authored-by: Thomas Moreau <[email protected]>
@tomMoral
tomMoral merged commit 2e9f24e into joblib:main Sep 23, 2025
28 of 30 checks passed
@tomMoral
tomMoral deleted the harden_eval_expr branch September 23, 2025 07:42
Sreekant13 added a commit to Sreekant13/joblib that referenced this pull request Sep 1, 2026
The magnitude limit only inspects a value once it exists, so a power whose
operands are individually within the limit was computed in full before being
rejected. Evaluating 999999**999999 that way costs several seconds of CPU and
builds a number with millions of digits, which is the kind of expression the
limit was added to guard against in joblib#1744.

A power is now rejected up front when it cannot possibly fit: for any base of
magnitude 2 or more, base ** exponent is at least 2 ** exponent, so an
exponent above the bit length of the limit can never produce a value under it.
Only integer powers are checked, since a float power overflows cheaply and
already reports an error.

This also fixes the message for a large result. CPython caps integer to string
conversion at sys.get_int_max_str_digits() digits, so formatting the value
into "Numeric literal ... is too large" raised instead, and pre_dispatch of
"10**5000" surfaced a complaint about integer string conversion along with
advice to call sys.set_int_max_str_digits().

Both cases reproduce through Parallel(pre_dispatch=...).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants