Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-153568: Parse arithmetic with operator loops #157413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pablogsal wants to merge 2 commits into
base: main
Choose a base branch
from
Open

gh-153568: Parse arithmetic with operator loops #157413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
717ce38
gh-153568: Parse arithmetic with operator loops
pablogsal Sep 13, 2026
1c7ff67
gh-153568: Fix generator typing and the news path
pablogsal Sep 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions Grammar/python.gram
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -817,35 +817,35 @@ is_bitwise_or[CmpopExprPair*]: 'is' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, I
# Bitwise operators
# -----------------

bitwise_or[expr_ty]:
bitwise_or[expr_ty] (operator_loop):
| a=bitwise_or '|' b=bitwise_xor { _PyAST_BinOp(a, BitOr, b, EXTRA) }
| invalid_bitwise_or
| bitwise_xor

bitwise_xor[expr_ty]:
bitwise_xor[expr_ty] (operator_loop):
| a=bitwise_xor '^' b=bitwise_and { _PyAST_BinOp(a, BitXor, b, EXTRA) }
| bitwise_and

bitwise_and[expr_ty]:
bitwise_and[expr_ty] (operator_loop):
| a=bitwise_and '&' b=shift_expr { _PyAST_BinOp(a, BitAnd, b, EXTRA) }
| invalid_bitwise_and
| shift_expr

shift_expr[expr_ty]:
shift_expr[expr_ty] (operator_loop):
| a=shift_expr '<<' b=sum { _PyAST_BinOp(a, LShift, b, EXTRA) }
| a=shift_expr '>>' b=sum { _PyAST_BinOp(a, RShift, b, EXTRA) }
| sum

# Arithmetic operators
# --------------------

sum[expr_ty]:
sum[expr_ty] (operator_loop):
| a=sum '+' b=term { _PyAST_BinOp(a, Add, b, EXTRA) }
| a=sum '-' b=term { _PyAST_BinOp(a, Sub, b, EXTRA) }
| invalid_arithmetic
| term

term[expr_ty]:
term[expr_ty] (operator_loop):
| a=term '*' b=factor { _PyAST_BinOp(a, Mult, b, EXTRA) }
| a=term '/' b=factor { _PyAST_BinOp(a, Div, b, EXTRA) }
| a=term '//' b=factor { _PyAST_BinOp(a, FloorDiv, b, EXTRA) }
Expand Down
51 changes: 51 additions & 0 deletions InternalDocs/parser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,57 @@ and "hidden left-recursion" like:
rule: 'optional'? rule '@' some_other_rule
```

Explicit operator loops
----------------------

A C grammar can request an operator loop for a particular left-recursive rule
with the `(operator_loop)` flag:

```
sum[expr_ty] (operator_loop):
| a=sum '+' b=term { _PyAST_BinOp(a, Add, b, EXTRA) }
| a=sum '-' b=term { _PyAST_BinOp(a, Sub, b, EXTRA) }
| term
```

Unmarked rules keep the general left-recursion algorithm. The generator checks
each marked rule and raises a rule-specific error if its shape or operand
dependency is unsupported. The flag takes no value and cannot be combined with
other rule flags. In `skip_actions` mode the generator still validates marked
rules, but emits the general algorithm instead of AST-folding loops.

The flag also asserts an action contract; the generator cannot prove properties
of arbitrary C code. Actions reachable from a marked rule must preserve the
grammar-derived token position and parser-control state when they succeed.
They must not make hidden calls to generated parser rules, inspect or modify
provisional left-recursion memo entries, or mutate the buffered input. The
existing generated `without_invalid` scopes remain supported. AST constructors,
feature checks, warnings and normal error reporting remain permitted. Existing
restrictions on modifying shared AST nodes also apply. Changes to relevant
actions and their C helpers must preserve this contract.

After excluding alternatives guarded by `call_invalid_rules`, each marked rule
must consist of `left=self exact-token right=operand` alternatives followed by
one plain operand alternative. Operator tokens must be distinct. The result and
operand types must be `expr_ty`. Recursive actions must construct `_PyAST_BinOp`
using the bound operands and a constant operator kind, optionally wrapped in
`CHECK_VERSION`. Cuts, predicates, other recursive actions and other alternative
shapes are unsupported.

The operand must consume input and cannot call the marked rule again at the same
position through the grammar. It must use ordinary `(memo)`, or also be marked
`(operator_loop)` and pass validation. This preserves the
original algorithm's final base retry as memo reuse, rather than executing an
operand action again. The loop retains result memoization but does not publish
intermediate seeds. Normal right-operand failure restores the position before
the operator; fatal errors propagate.

Loops run when `call_invalid_rules` is false. The original leader and raw rule
remain available while it is true. This includes diagnostic parsing: an
`expression_without_invalid` scope temporarily disables invalid rules and may
therefore use loops. The transformation changes stack usage and allocation
order; the flag does not promise identical resource-limit boundaries.

Variables in the grammar
------------------------

Expand Down
2 changes: 2 additions & 0 deletions ....d/next/Core_and_Builtins/2026-09-13-10-44-21.gh-issue-153568.operatorloops.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Speed up arithmetic parsing by generating operator loops for explicitly
marked grammar rules.
Loading
Loading