gh-129157: Change the location of with statement AST nodes to spawn the first line #129162

pablogsal · 2025-01-21T23:43:09Z

Issue: Add to AST location info for compound exception headers or keywords #129157

iritkatriel · 2025-01-22T00:52:31Z

Grammar/python.gram

+    | start='async' 'with' '(' a[asdl_withitem_seq*]=','.with_item+ ','? ')' end=':' b=block {
+       CHECK_VERSION(stmt_ty, 5, "Async with statements are", _PyAST_AsyncWith(a, b, NULL, EXTRA_EXPR(start, end))) }
+    | start='async' 'with' a[asdl_withitem_seq*]=','.with_item+ end=':' tc=[TYPE_COMMENT] b=block {
+       CHECK_VERSION(stmt_ty, 5, "Async with statements are", _PyAST_AsyncWith(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA_EXPR(start, end))) }


If I understand correctly, this will create a situation where the source location of a with node does not cover all the source code of the body sub-tree hanging under it. I think in the past we considered this undesirable.

If you think this is ok we can go with it, but I assumed it was not, and my intention was that we add a header child to a with node, so the tree has a with node spanning header + body, with two sub-trees, one for header and one for body. The other compound statements can also have a header + body split.

If I understand correctly, this will create a situation where the source location of a with node does not cover all the source code of the body sub-tree hanging under it. I think in the past we considered this undesirable.

If you think this is ok we can go with it, but I assumed it was not, and my intention was that we add a header child to a with node, so the tree has a with node spanning header + body, with two sub-trees, one for header and one for body. The other compound statements can also have a header + body split.

Ah, that is actually a breaking change for tools :(

If you think this is ok we can go with it, but I assumed it was not, and my intention was that we add a header child to a with node, so the tree has a with node spanning header + body, with two sub-trees, one for header and one for body. The other compound statements can also have a header + body split.

Yeah, I can go with this instead 👍

Just to be clear, how different are you picturing this to be from the current version. Currently for

with A, B, C: ...

we have:

Module( body=[ With( items=[ withitem( context_expr=Name(id='A', ctx=Load())), withitem( context_expr=Name(id='B', ctx=Load())), withitem( context_expr=Name(id='C', ctx=Load()))], body=[ Expr( value=Constant(value=Ellipsis))])])

are you picturing some new node wrapping the items?

I have made a version of this just adding a "header" wrapper.

I am thinking that maybe we can generalize this and create a simple "container" node that just wraps and adds location info instead of making it "header". WDYT?

I think we need the location of the whole with statement, up to the :. So it would need a new node wrapping all the items.

But let's make sure we have a plan. How do we want to render a traceback for an exception coming out of the __enter__ or __exit__ of one of the context managers? Should we hilight the whole statement and put carets under the expression that creates the specific context manager that caused the error?

CC @ericsnowcurrently

Lib/test/test_ast/snippets.py

Grammar/python.gram

…pawn the first line Signed-off-by: Pablo Galindo <[email protected]>

pablogsal · 2025-01-22T18:06:21Z

@iritkatriel I have switch to adding an extra optional header type to every statement. The reason is that wrapping is MUCH more verbose and require changes in everyone handling these nodes in Python as well which I would prefer to avoid. Additionally to create a generic one we would need to hack and to type erasing on the sequences which I don't like as that reduces safety.

I think this approach is the simplest but is also good enough to be generic.

I have only tacked with here for us to agree on the approach and we can do more statements in other PRs.

iritkatriel · 2025-01-22T18:09:28Z

@iritkatriel I have switch to adding an extra optional header type to every statement. The reason is that wrapping is MUCH more verbose and require changes in everyone handling these nodes in Python as well which I would prefer to avoid. Additionally to create a generic one we would need to hack and to type erasing on the sequences which I don't like as that reduces safety.

I think this approach is the simplest but is also good enough to be generic.

Yes, I agree. Good idea.

iritkatriel

We're getting something, but I think this is not enough yet. Currently we get more precise information than what your change will give us - we know which of the context managers failed:

>>> class C:
...     def __init__(self, arg): self.arg = arg
...     def __enter__(self): 1/self.arg
...     def __exit__(*args): pass
...     
>>> with C(-1), C(0), C(1):
...     1
...     2
...     
Traceback (most recent call last):
  File "<python-input-21>", line 1, in <module>
    with C(-1), C(0), C(1):                                                      # <---
                ~^^^
  File "<python-input-20>", line 3, in __enter__
    def __enter__(self): 1/self.arg
                         ~^~~~~~~~~
ZeroDivisionError: division by zero

Unless we can add carets to show that, this change actually makes things worse in the traceback.

iritkatriel · 2025-01-22T18:33:50Z

Parser/action_helpers.c

@@ -1693,4 +1693,4 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,

    assert(current_pos == n_elements);
    return _PyAST_JoinedStr(values, lineno, col_offset, end_lineno, end_col_offset, p->arena);
-}
+}


revert this.

Will do! In any case don't worry too much for now as we are just in the exploratory phase. Once we are happy I will clean everything

pablogsal · 2025-01-22T23:32:28Z

Hummmmm how do you picture the carets then? Can you give me an example of what you have in mind?

I think it would also be helpful if you can draft how you would like to propagate the locations. Currently you have in the context manager the header but also every one of the individual items so in theory the compiler is given all it needs no?

pablogsal · 2025-01-22T23:36:50Z

Another problem is to add carets we need to be able to parse the entire context manager with the body for the ast parse function to be happy we cannot just pass the header line because that will fail to parse

iritkatriel · 2025-01-22T23:40:50Z

Currently you have in the context manager the header but also every one of the individual items so in theory the compiler is given all it needs no?

True. I guess this isn't the AST's problem then.

iritkatriel · 2025-01-22T23:47:43Z

So we want something like this? (And analogously for except you hilight the relevant type expression (if there is one), and in a loop you hilight the iterator expression (for exception in __next__)).

>>> class C:
...     def __init__(self, arg): self.arg = arg
...     def __enter__(self): 1/self.arg
...     def __exit__(*args): pass
...     
>>> with C(-1), C(0), C(1):
...     1
...     2
...     
Traceback (most recent call last):
  File "<python-input-21>", line 1, in <module>
    with C(-1), C(0), C(1):                                                      # <---
    ~~~~~~~~~~~~^^^~~~~~~~
  File "<python-input-20>", line 3, in __enter__
    def __enter__(self): 1/self.arg
                         ~^~~~~~~~~
ZeroDivisionError: division by zero

iritkatriel · 2025-01-23T00:03:52Z

Another problem is to add carets we need to be able to parse the entire context manager with the body for the ast parse function to be happy we cannot just pass the header line because that will fail to parse

We can add a pass as the body and then it will compile. But maybe we can find a way to do it without having to compile the code again.

Suppose the code object has locations as they are now (with just the context manager expression for the with statement), but there is also a "compound statement table" which maps instructions ranges to header locations. So all instructions implementing a compound statement header show up there (as one entry for the range of instructions). The space this would take would depend on the number of such blocks in the function (like the exception table grows when you have more try-excepts). Lookup time doesn't matter that much because it's only used for tracebacks.

iritkatriel · 2025-01-23T01:00:06Z

I think you can revert the changes to the compiler and merge this PR for the AST changes. Then we'll follow up with something in the compiler for the tracebacks in another PR.

pablogsal requested review from Eclips4 and lysnikolaou as code owners January 21, 2025 23:43

bedevere-app bot added the awaiting core review label Jan 21, 2025

bedevere-app bot mentioned this pull request Jan 21, 2025

Add to AST location info for compound exception headers or keywords #129157

Open

pablogsal force-pushed the gh-129157 branch 2 times, most recently from f6564e9 to 38f90ff Compare January 22, 2025 00:18

pablogsal requested review from markshannon and iritkatriel as code owners January 22, 2025 00:18

pablogsal force-pushed the gh-129157 branch from 38f90ff to 9f84c26 Compare January 22, 2025 00:37

iritkatriel reviewed Jan 22, 2025

View reviewed changes

pablogsal force-pushed the gh-129157 branch from 9f84c26 to ba0404f Compare January 22, 2025 01:31

pablogsal requested review from isidentical, JelleZijlstra, ericsnowcurrently and carljm as code owners January 22, 2025 01:31

iritkatriel reviewed Jan 22, 2025

View reviewed changes

Lib/test/test_ast/snippets.py Outdated Show resolved Hide resolved

Grammar/python.gram Outdated Show resolved Hide resolved

pythongh-129157: Change the location of with statement AST nodes to s…

09d87e8

…pawn the first line Signed-off-by: Pablo Galindo <[email protected]>

pablogsal force-pushed the gh-129157 branch from ba0404f to 09d87e8 Compare January 22, 2025 18:04

iritkatriel reviewed Jan 22, 2025

View reviewed changes

iritkatriel mentioned this pull request Jan 24, 2025

gh-129025: fix too wide source location for bytecodes emitted for except* #129026

Open

JelleZijlstra removed their request for review March 3, 2025 23:09

Uh oh!

gh-129157: Change the location of with statement AST nodes to spawn the first line #129162

Are you sure you want to change the base?

gh-129157: Change the location of with statement AST nodes to spawn the first line #129162

Uh oh!

Conversation

pablogsal commented Jan 21, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iritkatriel Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iritkatriel Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pablogsal commented Jan 22, 2025

Uh oh!

iritkatriel commented Jan 22, 2025

Uh oh!

iritkatriel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iritkatriel Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

pablogsal commented Jan 22, 2025

Uh oh!

pablogsal commented Jan 22, 2025

Uh oh!

iritkatriel commented Jan 22, 2025

Uh oh!

iritkatriel commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iritkatriel commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iritkatriel commented Jan 23, 2025

Uh oh!

Uh oh!

pablogsal commented Jan 21, 2025 •

edited by bedevere-app bot

Loading

pablogsal Jan 22, 2025 •

edited

Loading

iritkatriel left a comment •

edited

Loading

iritkatriel commented Jan 22, 2025 •

edited

Loading

iritkatriel commented Jan 23, 2025 •

edited

Loading