Fix #815 - "error: unexpected input" from unicode character token #816
Conversation
Fixes fail obtained when a syntax error interrupts the parsing at a Unicode character and, as a consequence, the parsed content returned by 'getParseData(source_file)' includes an incomplete UTF-8 multibyte representation of that character (which is invalid). See the GitHub issue discussion for details. The proposed solution merely "coerces" the getParseData's 'text' column to what it is supposed to be according to its documentation. An alternative would be to actually fix the table before use: > content[!content$terminal, "text"] <- ""
|
Thanks for providing a fix. |
|
Done. I didn't know exactly how to test that a specific error doesn't occur, but it seems extensive enough like this. Note: I had to force-push because, when I ran the unit test with |
FYI that shouldn't be an issue, just |
| tmp <- tempfile() | ||
| on.exit(unlink(tmp), add = TRUE) | ||
| writeLines("y <- x \U2013 42", tmp) # Unicode en-dash | ||
| content <- expect_error(get_source_expressions(tmp), NA) | ||
| expect_equal(content$error$message, "unexpected input") | ||
| expect_equal(content$error$column_number, 8L) |
There was a problem hiding this comment.
Not sure if possible, but have you tried using with_content_to_parse() used in the other tests?
There was a problem hiding this comment.
I took a look at it, but it seemed that with_content_to_parse() itself would fail with the bug. And as it doesn't use the expect family of functions internally, the error would not be well formatted. I didn't verify it though.
There was a problem hiding this comment.
Did some experiments with the tests just now. The error generated by with_content_to_parse() ends well formatted anyway by testthat. So in principle we could use it, but it doesn't expose the error item from get_source_expressions() result. I'm sending an alternative version of the tests with a modification to that helper function.
|
Looks good now. Please add a |
|
Thank you for your time and attention. 😉 |
Fixes fail obtained when a syntax error interrupts the parsing at a
Unicode character and, as a consequence, the parsed content returned
by
getParseData(source_file)includes a truncated UTF-8 multibyterepresentation of that character (which is invalid). See the GitHub
issue discussion (#815) for details.
The proposed solution merely "coerces" the
getParseData'stextcolumn to what it is supposed to be according to its documentation.
An alternative would be to actually fix the table before use: