Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-153568: Reuse shared prefixes in generated parser rules - #157465

Merged
pablogsal merged 1 commit into
python:mainfrom
pablogsal:gh-153568-parser-prefix-cache
Sep 15, 2026
Merged

gh-153568: Reuse shared prefixes in generated parser rules#157465
pablogsal merged 1 commit into
python:mainfrom
pablogsal:gh-153568-parser-prefix-cache

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

For obj[index], the parser tries the attribute and call alternatives before the subscript one. Each choice starts by asking for the same obj again. The parse cache saves us from parsing it from scratch, but we still pay for another rule call and cache lookup.

This teaches the generator to keep that result and the position where it ended in local variables. When one choice fails, the next one can pick them up directly and try its own ending. It does this for adjacent choices that start with the same memoized rule, including expression, arithmetic and assignment-target rules.

The grammar stays as it is. The generator finds these shared starts on its own.

On my machine, this makes parsing the standard library and twenty repositories about 10.7% faster overall.

Codebase Python files Parsing speed
Standard library 2,054 11.5% faster
Django 2,932 11.7% faster
pandas 1,527 11.6% faster
NumPy 495 11.7% faster
SciPy 999 10.7% faster
scikit-learn 1,036 10.0% faster
SymPy 1,607 11.6% faster
Transformers 4,884 10.7% faster
PyTorch 4,890 10.5% faster
Home Assistant 18,731 11.1% faster
mypy 442 9.7% faster
Requests 37 11.3% faster
Flask 83 11.3% faster
FastAPI 1,138 10.6% faster
SQLAlchemy 673 11.3% faster
Black 345 9.4% faster
pytest 274 10.7% faster
Ansible 1,853 10.6% faster
Scrapy 491 11.9% faster
Celery 428 10.2% faster
Sphinx 774 10.6% faster

The table compares afe3d53d4d against main at fd0970c0ab, using release builds without PGO or LTO. Each result is the median of five paired runs. The timings include tokenization, building the AST and cleanup, without compiling bytecode or running the files.

This overlaps with #157437, which shares some of these starts by rewriting the grammar. Adding this on top of that PR still gives about 5.2% faster parsing across the standard library, mypy, Home Assistant and PyTorch in a separate run. That comparison does not include the rest of the pending parser changes.

Part of #153568.

@bedevere-app bedevere-app Bot mentioned this pull request Sep 13, 2026
Open
@pablogsal
pablogsal force-pushed the gh-153568-parser-prefix-cache branch from afe3d53 to 27f06b1 Compare September 14, 2026 20:03
@pablogsal
pablogsal force-pushed the gh-153568-parser-prefix-cache branch from 27f06b1 to d0a0ce8 Compare September 14, 2026 20:50
@pablogsal
pablogsal merged commit 89fe985 into python:main Sep 15, 2026
63 checks passed
@pablogsal
pablogsal deleted the gh-153568-parser-prefix-cache branch September 15, 2026 00:26
This was referenced Sep 15, 2026
Merged
Closed
@python python deleted a comment from bedevere-bot Sep 15, 2026
@python python deleted a comment from bedevere-bot Sep 15, 2026
@bedevere-bot

bedevere-bot commented Sep 15, 2026

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 Refleaks 3.x (tier-1) has failed when building commit 89fe985.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/853/builds/4253) and take a look at the build logs.
  4. Check if the failure is related to this commit (89fe985) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/853/builds/4253

Summary of the results of the build (if available):

==

Click to see traceback logs 
Note: switching to '89fe9857dd0832de9b597d63ecb11fa7a09984f1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 89fe9857dd0 gh-153568: Reuse shared prefixes in generated parser rules (#157465)
Switched to and reset branch 'main'

make: *** [Makefile:2481: buildbottest] Error 3

Expanding reachable commits in commit graph: 99281
Expanding reachable commits in commit graph: 171162, done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pablogsal @bedevere-bot