Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-153568: Parse arithmetic with operator loops - #157413

Open
pablogsal wants to merge 2 commits into
python:mainfrom
pablogsal:gh-153568-parser-operator-loops
Open

gh-153568: Parse arithmetic with operator loops#157413
pablogsal wants to merge 2 commits into
python:mainfrom
pablogsal:gh-153568-parser-operator-loops

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 13, 2026

Copy link
Copy Markdown
Member

For a + b + c, the parser first finds a, then calls the rule again to get a + b, then again to get (a + b) + c. It puts each partial result in the parse cache so the next call can pick it up. That works, but it adds rule calls, cache lookups and cache updates to something that is really just a loop.

This adds an (operator_loop) flag to the six arithmetic and bitwise rules. The generated code parses a, keeps the current expression in a local variable, and loops over + b and + c. Each step builds the next AST node directly. It only puts the finished expression in the cache, so all that bookkeeping between steps goes away.

This also helps when there is no operator. Even a plain name passes through these rules, and the general parser still has to check whether it can grow the expression. The loop can just see that the next token isn't an operator and return.

The grammar and AST actions stay the same apart from the flag, and the existing error rules are still there.

The table compares parsing with and without these loops. Both builds include pending changes that reduce allocations, skip grammar choices using the next token, make stack checks and cache updates cheaper, and handle simple one-token expressions directly.

Codebase Python files Release CPU time PGO/LTO CPU time
Standard library 2,060 16.02% less 17.91% less
Home Assistant 18,731 17.19% less 19.18% less
mypy 442 16.50% less 17.83% less
PyTorch 4,890 14.54% less 17.50% less

The timings include tokenization, building the AST and cleanup, without compiling bytecode or running the files.

Turn rules marked with operator_loop into a simple loop over operators and
operands. Keep the cached result and the existing error rules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant