Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 717ce38

Browse files
pablogsalpablogsal
committed
gh-153568: Parse arithmetic with operator loops
Turn rules marked with operator_loop into a simple loop over operators and operands. Keep the cached result and the existing error rules.
1 parent a60343e commit 717ce38

5 files changed

Lines changed: 967 additions & 7 deletions

File tree

‎Grammar/python.gram‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -817,35 +817,35 @@ is_bitwise_or[CmpopExprPair*]: 'is' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, I
817817
# Bitwise operators
818818
# -----------------
819819

820-
bitwise_or[expr_ty]:
820+
bitwise_or[expr_ty] (operator_loop):
821821
| a=bitwise_or '|' b=bitwise_xor { _PyAST_BinOp(a, BitOr, b, EXTRA) }
822822
| invalid_bitwise_or
823823
| bitwise_xor
824824

825-
bitwise_xor[expr_ty]:
825+
bitwise_xor[expr_ty] (operator_loop):
826826
| a=bitwise_xor '^' b=bitwise_and { _PyAST_BinOp(a, BitXor, b, EXTRA) }
827827
| bitwise_and
828828

829-
bitwise_and[expr_ty]:
829+
bitwise_and[expr_ty] (operator_loop):
830830
| a=bitwise_and '&' b=shift_expr { _PyAST_BinOp(a, BitAnd, b, EXTRA) }
831831
| invalid_bitwise_and
832832
| shift_expr
833833

834-
shift_expr[expr_ty]:
834+
shift_expr[expr_ty] (operator_loop):
835835
| a=shift_expr '<<' b=sum { _PyAST_BinOp(a, LShift, b, EXTRA) }
836836
| a=shift_expr '>>' b=sum { _PyAST_BinOp(a, RShift, b, EXTRA) }
837837
| sum
838838

839839
# Arithmetic operators
840840
# --------------------
841841

842-
sum[expr_ty]:
842+
sum[expr_ty] (operator_loop):
843843
| a=sum '+' b=term { _PyAST_BinOp(a, Add, b, EXTRA) }
844844
| a=sum '-' b=term { _PyAST_BinOp(a, Sub, b, EXTRA) }
845845
| invalid_arithmetic
846846
| term
847847

848-
term[expr_ty]:
848+
term[expr_ty] (operator_loop):
849849
| a=term '*' b=factor { _PyAST_BinOp(a, Mult, b, EXTRA) }
850850
| a=term '/' b=factor { _PyAST_BinOp(a, Div, b, EXTRA) }
851851
| a=term '//' b=factor { _PyAST_BinOp(a, FloorDiv, b, EXTRA) }

‎InternalDocs/parser.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,57 @@ and "hidden left-recursion" like:
209209
rule: 'optional'? rule '@' some_other_rule
210210
```
211211

212+
Explicit operator loops
213+
----------------------
214+
215+
A C grammar can request an operator loop for a particular left-recursive rule
216+
with the `(operator_loop)` flag:
217+
218+
```
219+
sum[expr_ty] (operator_loop):
220+
| a=sum '+' b=term { _PyAST_BinOp(a, Add, b, EXTRA) }
221+
| a=sum '-' b=term { _PyAST_BinOp(a, Sub, b, EXTRA) }
222+
| term
223+
```
224+
225+
Unmarked rules keep the general left-recursion algorithm. The generator checks
226+
each marked rule and raises a rule-specific error if its shape or operand
227+
dependency is unsupported. The flag takes no value and cannot be combined with
228+
other rule flags. In `skip_actions` mode the generator still validates marked
229+
rules, but emits the general algorithm instead of AST-folding loops.
230+
231+
The flag also asserts an action contract; the generator cannot prove properties
232+
of arbitrary C code. Actions reachable from a marked rule must preserve the
233+
grammar-derived token position and parser-control state when they succeed.
234+
They must not make hidden calls to generated parser rules, inspect or modify
235+
provisional left-recursion memo entries, or mutate the buffered input. The
236+
existing generated `without_invalid` scopes remain supported. AST constructors,
237+
feature checks, warnings and normal error reporting remain permitted. Existing
238+
restrictions on modifying shared AST nodes also apply. Changes to relevant
239+
actions and their C helpers must preserve this contract.
240+
241+
After excluding alternatives guarded by `call_invalid_rules`, each marked rule
242+
must consist of `left=self exact-token right=operand` alternatives followed by
243+
one plain operand alternative. Operator tokens must be distinct. The result and
244+
operand types must be `expr_ty`. Recursive actions must construct `_PyAST_BinOp`
245+
using the bound operands and a constant operator kind, optionally wrapped in
246+
`CHECK_VERSION`. Cuts, predicates, other recursive actions and other alternative
247+
shapes are unsupported.
248+
249+
The operand must consume input and cannot call the marked rule again at the same
250+
position through the grammar. It must use ordinary `(memo)`, or also be marked
251+
`(operator_loop)` and pass validation. This preserves the
252+
original algorithm's final base retry as memo reuse, rather than executing an
253+
operand action again. The loop retains result memoization but does not publish
254+
intermediate seeds. Normal right-operand failure restores the position before
255+
the operator; fatal errors propagate.
256+
257+
Loops run when `call_invalid_rules` is false. The original leader and raw rule
258+
remain available while it is true. This includes diagnostic parsing: an
259+
`expression_without_invalid` scope temporarily disables invalid rules and may
260+
therefore use loops. The transformation changes stack usage and allocation
261+
order; the flag does not promise identical resource-limit boundaries.
262+
212263
Variables in the grammar
213264
------------------------
214265

‎Misc/NEWS.d/next/Core and Builtins/2026-09-13-10-44-21.gh-issue-153568.operatorloops.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Speed up arithmetic parsing by generating operator loops for explicitly
2+
marked grammar rules.

0 commit comments

Comments
 (0)