Why not fix the new parser to produce an error message that doesn't end in a newline? This could affect other uses as well.

Why not fix the new parser to produce an error message that doesn't end in a newline? This could affect other uses as well.

👍 I was not sure if we were relying on those newlines elsewhere.

I will try to do the normalization in pegen.c

I was not sure if we were relying on those newlines elsewhere.

Indeed, there are tests that fail if the newline is not there. For instance:

/python -m test test_exceptions
0:00:00 load avg: 0.68 Run tests sequentially
0:00:00 load avg: 0.68 [1/1] test_exceptions
test test_exceptions failed -- Traceback (most recent call last):
  File "/home/pablogsal/github/python/master/Lib/test/test_exceptions.py", line 221, in testSyntaxErrorOffset
    check(b"""if 1:
  File "/home/pablogsal/github/python/master/Lib/test/test_exceptions.py", line 190, in check
    self.assertIn(line, cm.exception.text)
AssertionError: "                '''quux'''" not found in "                '''"

test_exceptions failed

== Tests result: FAILURE ==

1 test failed:
    test_exceptions

Total duration: 1.6 sec
Tests result: FAILURE

Ok, I think I have something that works. I have reverted my changes to codeop.py and I have added trimming code in pegen.c

Indeed, there are tests that fail if the newline is not there.

But then you must not be doing the same thing as the old parser. I created a little test program:

for suffix in "", "\n", "\n\n":
    try:
        compile("raise=1" + suffix, "", "exec")
    except SyntaxError as err:
        print(repr(err))

Output with old parser always includes a newline:

(v38) ~/cpython$ ./python.exe  -X oldparser t.py
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1\n'))
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1\n'))
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1\n'))

But with new parser only when there's a newline in the source:

(v38) ~/cpython$ ./python.exe  t.py
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1'))
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1'))
SyntaxError('invalid syntax', ('', 1, 6, 'raise=1\n'))

FWIW there are also differences with eval and single modes. Try it.

• New parser shows newline only for exec with two newlines appended
• Old parser shows newline whenever present in the source, but always for exec

IIRC the old parser adds a newline in exec mode "just in case".

Not sure how to handle this. Maybe always append the newline???

FWIW there are also differences with eval and single modes. Try it.

Ugh. The current version of the patch is always trimming all newlines. But seems that this is not what the old parser does. Seems that we need to maintain the last one when using exec, isn't it?

(See my edited comment.)

Not sure how to handle this. Maybe always append the newline???

Apparently that makes some other tests in test_exceptions to fail because byte_offset_to_character_offset does not like the fiddling with the newlines. I am checking what's going on there

Seems that with this version something fails in test_cmdline

Seems that it was something wrong with my .pythonrc