Currently when you have an invalid setting like [Setpu] in a test (or keyword, or task), error is reported on the console, but the test is nevertheless executed and in the log file the invalid setting isn't visible under the test at all. This isn't great. It would be better if the invalid setting would be visible under the test in the log file and it should also fail the test.

A similar situation occurs with invalid syntax like END or ELSE without an opening IF. At the moment such invalid syntax is considered to be a keyword. We have Reserved library that contains matching keywords, so when these "keywords" are run they fail. Invalid syntax reported as failing keywords is odd and the resulting error messages aren't great either. A part of #4210 is detecting this kind of invalid syntax already at the parsing time, but we need to also show them in the log file somehow.

A solution to both of these error reporting issues is adding new Error object to our TestSuite structure. When the parser detects invalid syntax, it can create such objects, and when they are run they can simply fail. These errors should create new <error> elements to output.xml and they obviously need to be show in log.html as well.

A problem with this enhancement is that it's not fully backwards compatible:

• Invalid settings like [Setpu] don't currently affect test status, but in the future they will fail the test. An obvious solution for this kind of problem is not having such invalid settings in tests in the first place.
• External tools parsing output.xml need to take <error> elements into account. In general such tools should ignore elements they don't recognize, but there certainly can be tools that don't behave like that.
• Tools working with the TestSuite model need to take new Error objects into account.

Invalid syntax failing a test isn't a big concern for me, but this change possibly breaking external tools is a lot bigger problem. I don't see any other ways this could be handed, though. Doing the change in a non-major version is a bit questionable, but I don't think postponing this change to RF 7 (or changing RF 6.1 to RF 7) is a good idea either. We just need to inform external tool developers about this change in release notes and possibly also otherwise.