Fix IndexError on source mixing lone CR with LF line endings (#105) - #174
Conversation
Fixes gristlabs#105. `ASTTokens(src, parse=True)` raised `IndexError: list index out of range` on valid source that mixes a lone "\r" (old-Mac line ending) with "\n" and ends with an attribute access, e.g. the minimal reproducer "\ry.y\n". `LineNumbers` computed line starts with a `re.M` `^`, which does not treat a lone "\r" as a line boundary. Python's tokenizer and `ast` module do, so AST node positions on the line after a "\r" were mapped to a character offset past the source text and thus onto the ENDMARKER token. `MarkTokens.handle_attr` then called `next_token` on the ENDMARKER, walking past the end of the token list. Compute line starts from the actual end-of-line sequences ("\r\n", "\r" or "\n") instead, matching how source lines are numbered elsewhere. "\r\n"- and "\n"-only sources are unaffected. Add regression tests for the line-offset handling and for marking tokens on CR/LF-mixed sources.
dsagal
left a comment
There was a problem hiding this comment.
Thank you!! The fix looks great. I have some comments about strengthening tests.
| def test_carriage_returns(self): | ||
| # A lone "\r" (old-Mac line ending) and "\r\n" (Windows) must both count as line boundaries, | ||
| # matching how Python's tokenizer and ast module number source lines. See issue #105. | ||
| ln = asttokens.LineNumbers("a\rb\r\nc\nd") |
There was a problem hiding this comment.
I suggest a stronger test example like "a\rb\r\nc\n\rd\r\re\r\r\nf" to cover situations like two \r in a row, and \n\r combination
| # related cases parse and mark without error. | ||
| for source in ( | ||
| "\ry.y\n", # the maintainer's minimal reproducer | ||
| "\ry\n", |
There was a problem hiding this comment.
All these only exercise one lone \r, always before \n. I'd ask for a test string that exercises more combinations, like "a=1\rb.c\r\nd=2\n\re.f\r\rg=3\rh.i". (Then many of the simpler cases are probably not necessary.)
|
|
||
|
|
||
| def test_generate_tokens_normalizes_lone_carriage_returns(): | ||
| tokens = list(asttokens.util.generate_tokens("\ry.y\n")) |
There was a problem hiding this comment.
Would it be useful to test triple-quoted strings too?
| for node in util.walk(atok.tree): | ||
| first, last = getattr(node, 'first_token', None), getattr(node, 'last_token', None) | ||
| if first is not None: | ||
| self.assertNotEqual(last.type, token.ENDMARKER, node) |
There was a problem hiding this comment.
The assertions check that marking doesn't run off the end of the token stream, but not that nodes get mapped to the right tokens. Maybe you could assert the actual value of atok.get_text(stmt) for some statements?
|
Thanks for the review! I addressed all four test suggestions in commit
The affected tests pass on Python 3.8, 3.11, and 3.13, and the full fast suite passes on Python 3.13 ( |
Summary
Fixes #105.
asttokens.ASTTokens(src, parse=True)raisedIndexError: list index out of rangeon valid source that mixes a lone\rwith\nand ends with an attribute access. The minimal reproducer is"\ry.y\n".Root cause
Two position systems disagreed:
ast.parse()treats\r,\r\n, and\nas source line boundaries and reportsy.yon line 2.LineNumberspreviously recognized only boundaries matched by multiline^.tokenizeemits the entire"\ry.y\n"input as a non-codingNLtoken, so noy,., orytokens exist for the AST nodes.The AST positions therefore landed on the
ENDMARKER;MarkTokens.handle_attr()callednext_token()on it and walked past the token list.Fix
\r\n, lone\r, and\nsequences.\rwith\n. This is length-preserving, so offsets still map to the original source.\r\nunchanged to avoid shifting later offsets.asttokens.util.generate_tokens()helper, covering default tokenization and callers that pass generated tokens explicitly.Tests
astroid<3on Python 3.8 and 3.11.Disclosure: prepared with AI assistance; reviewed and verified locally.