Refactor execution and result side model objects #3749

pekkaklarck · 2020-10-30T21:11:25Z

The TestSuite structures we use on the execution and result sides (robot.running.model and robot.result.model, respectively, with a common base classes in robot.model) have certain things that need to be changed:

Remove keywords and messages from execution side Keyword objects. These keywords never actually have child keywords or messages, they just represent the keyword call with the keyword name and arguments. Basically this means moving the related code from the common base class to the result side model objects.
TestSuite objects should not have keywords attribute but instead should have setup and teardown directly. Suites having keywords is odd in general and being able to use e.g. suite.setup instead of suite.keywords.setup is convenient.
keywords attribute used by TestCase, For, If (new in RF 4.0) and UserKeyword objects should be renamed to body. The body can contain FOR loops and soon also IF/ELSE in addition to keywords so body is a much better name than keywords and that's also the term parsing side model uses. The forthcoming IfElse objects should naturally have body, not keywords, as well.
Add setup and teadown directly to TestCase and UserKeyword objects. Accessing test.setup is more convenient than the current test.keywords.setup and test.body.setup would be just wrong.
Make setup and teardown objects, not None, also when they are not defined. This makes it easier for external code like pre-run modifiers to modify them. Their initial truth value should be false to make it possible to use code like if test.setup:.

This kind of changes are naturally backwards incompatible but in my opinion better model is worth that. We should, however, preserve keywords as a deprecated read-only property that is generated based on body and setup/teardown when accessed.

[Update on 2021-01-18] Everything discussed above has now been done, but doing these changes has uncovered other highly related problems in the model that also need to be changed:

On the execution side make For and If independent classes instead of extending Keyword. They don't need keyword related attributes like name or args, but instead they should have context specific attributes like For.flavor and If.condition. Earlier those context specific attributes were implemented as properties but they should be normal attributes. Old keyword specific attributes should probably be preserved as deprecated properties for backwards compatibility.
Add separate visit/start/end_for and visit/start/end_if methods to the visitor interface. Earlier For and If were visited using visit/start/end_keyword but that doesn't work anymore when they don't extend Keyword.
Add separate For and If classes on result side. Currently FOR and IF construct are reported as keywords with type set accordingly. This makes inspecting FOR/IF hard because their context specific information is not directly available and need to be parsed from keyword attributes. For example, all FOR information is in keyword name in format like ${x} IN [ a | b | c ]. This change requires some changes to output.xml as well and I added a separate comment about that.
On the result side move keywords' messages to body to preserve their relative order with keywords and FOR/IF constructs. This makes it easier to preserve their order also in log (Relative order of messages and keywords is not preserved in log #2086). The old messages attribute should be preserved as a property to make accessing only messages easy.

The text was updated successfully, but these errors were encountered:

pekkaklarck · 2020-10-30T21:14:01Z

These changes affect also pre-run modifier examples in the User Guide:
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#example-skip-setups-and-teardowns

Related to those examples, we need an easy way to disable setup/teardown.

pekkaklarck · 2020-11-26T11:30:32Z

Important things to decide:

Should visitor and listener APIs be changed? They both currently have start/end_keyword method which are called also with FOR and IF constructs. Should we add separate start/end_for and start/end_if methods? If we do, they could call start/end_keyword by default to avoid backwards incompatibly problems.
Should we change <keywords> to <body> also in output.xml? It would make this change even more backwards incompatible, but output.xml being inconsistent with internal model would also be a smallish problem. If we would decide to change output.xml format, we need to submit a separate issue about it and enhance output.xml also otherwise. For example, attributes currently using values yes or no should be changed to use true or false (i.e. xsd:boolean type).

Related to #3749.

prescod · 2020-12-12T19:25:41Z

Relevant to your refactor:

I'm trying to measure and log the execution time of tests with Setup and Teardown subtracted. I'm having a tough time using the current Listener V3 Results object to do so.

Given this test:

Test Perf Measure
    [Teardown]      Sleep   0.5
    [Setup]         Sleep   0.5
    [Tags]  perf
    Log       Noop

And this code:

    def end_test(self, data, result):
        print(type(result), len(result.keywords))
        print(type(data), len(data.keywords))

I get:

<class 'robot.result.model.TestCase'> 0
<class 'robot.running.model.TestCase'> 3

So the result model thinks I have 0 keywords and the running model thinks I have 3.

But the running model has no start-time or end-time. So I have not figured out how to get a hold of the elapsed time for the Setup and Teardown in the end_test listener hook.

pekkaklarck · 2020-12-12T21:36:24Z

That problem is due to keyword results being only written to output.xml and not stored in memory during execution. That's not really related to this issue, but we need to do something to that if/when we add start/end_keyword methods to the listener v3 (#3296). If you are interested in execution times, it's easiest to read them after execution using result models. If you really need that info during execution, you need to use listener v2.

prescod · 2020-12-13T09:13:38Z

Thank you. I've switched to an XML post-process system.

Related to robotframework#3749.

- Separate For and If base classes in robot.model. - For and If not anymore based on Keyword. - They have common BodyItem base class instead. - Running side For and If use right runners automatically. - Replace Body.create with Body.create_(keyword|for|if). This is part of model refactoring (#3749).

…3749.

This is part of model refactoring discussed in issue #3749. Storing keywords and messages in body, and preserving their order also in log files, fixes #2086.

- Test/keyword setup/teardown were not visited. - Visiting FOR/IF was totally broken. Both problems due to the ongoing model refactoring (#3749).

pekkaklarck · 2021-01-18T11:33:26Z

This issue has turned out to be bigger than originally anticipated. Everything discussed in the original description is now done, but these enhancements have uncovered other highly related problems in the model that also need to be changed. I'll update the description to list these other changes as well.

Had we seen how big change this is, we might have postponed it to future releases. Because we've already done so much, and because changes are really good, I strongly believe we should finish this still before RF 4.0.

This makes For/If being independent classes instead of extending Keyword less backwards incompatible. Related to #3749.

pekkaklarck · 2021-01-18T12:55:35Z

The main remaining task is adding For and If to the result side instead of reporting FOR and IF as keywords. This isn't itself that big a task, but because results are saved to output.xml, and we need to be able to read that info back, we need to think what kind of changes to do there. I see two possibilities:

Introduce new <for> and <if> elements that contain FOR/IF related information. I believe this is the best approach in general, but such a big change to output.xml would affect many external tools and I'm not sure is that a good idea anymore in RF 4.0. If we do this, we should also cleanup some other problems in output.xml at the same time.
Keep using <kw> with type attribute differentiating FOR/IF from normal keywords and store FOR/IF information into them. We could add new attributes to <kw> or add new elements under it, and tools processing output.xml ought to in general just ignore them. Changes would be smaller but output.xml wouldn't directly represent model used internally and there would be some unnecessary cruft. We could do bigger output.xml cleanup in some future release, though.

pekkaklarck · 2021-01-18T17:39:24Z

A decision was made today to add separate <for> and <if> elements to output.xml. We also decided to cleanup output.xml otherwise a bit and I'll submit a new issue about all that.

pekkaklarck · 2021-01-28T16:26:15Z

After all these changes this issue is about to be ready and ought to be in good enough shape for the long overdue beta 2.It's likely some tuning is still done after the beta.

pekkaklarck · 2021-01-28T16:33:03Z

Things to be still consider include:

How to store FOR loop variables and values in output.xml? Currently they are stored using <assign> and <arguments> elements that also keywords use, but using something more FOR related might be a better idea.
How to store FOR loop iterations in output.xml and also in the result model? Currently the model has Iteration class with only info attribute and output.xml has <iter info=''>. The info contains variables and values used in that iteration as a single sting like ${x} = 1 or ${x} = 1, ${y} = 2 if there are multiple variables. It's not possibly to reliable parse such a string, so it might be better to store them as a dictionary in Iteration and like <var name='${x}'>1</var> in output.xml. That would make Iteration and <iter> non-generic but that ought to fine.

pekkaklarck · 2021-02-01T18:56:54Z

I'm started to think about representing IF/ELSE in the model and in output.xml. We currently do it so that the model has a root If object with the IF branch in body and the next ELSE IF or ELSE branch in orelse. ELSE IF and ELSE branches have their own body and the next branch in orelse. In output.xml these branches are nowadays nested like discussed in the above commit. We use the same design also in the parsing model and that was inspired by how Python itself stores if/else structures in its AST.

Another approach would be storing IF/ELSE structures so that we have a root model with branches attribute (or possibly body that other model objects have) that contains actual IF/ELSE branches. Then in output.xml we could use something like this:

<if>
    <branch condition='$x > 0' type='if'>
        <status status='NOT_RUN'/>
    </branch>
    <branch type='else'>
        <kw>...</kw>
        <status status='PASS'/>
    </branch>
</if>

This approach might feel more natural than the current approach with orelse and could also simplify some internal code related to IF/ELSE. There are, however, some problems as well:

Representing a single IF without ELSE IF or ELSE would require using two objects, the root and the branch object, compared to one.
In output.xml we'd also need <if><if type='if'>...</if></if> instead of just <if type='if'>...</if>.
Extra layer would likely require changes to the visitor interface.
All changes require more work and it would be great to get RF 4.0 out at some point.

Although this could still be changed, it certainly won't be changed before RF 4.0 beta 2.

boakley · 2021-02-02T17:15:58Z

Having an <if> inside an <if>, and having <if> both represent an "if" and "else" seems very odd. Why not a single tag such as <block> or <compound> which could be used for all statements that start a block of statements? For example, <block type='for'>, <block type='if'>, <block type='else'>, etc? Then, if robot ever gets a "with" or "while" or "try" statement, we don't have to invent more tags.

pekkaklarck · 2021-02-03T10:50:30Z

A problem with a generic element like <block> is that IF and FOR need to store different information. This means that depending on the usage <block> would have different attributes and/or child elements. This makes, for example, creating a schema harder.

pekkaklarck · 2021-02-11T15:24:32Z

Some more work to be done still:

Change IF/ELSE structure to use If object with all its branches in body and not the next ELSE IF or ELSE in orelse. Also change output.xml accordingly. This was discussed in the comment above.
Better type constant names and values.
Consistent naming of internal classes, visitor methods, etc.
Store FOR loop iteration information in output.xml so that it can be fully recovered.

Earlier IF/ELSE structures were represented so that `If` objects represented IF branches and they has `orelse` attribute containing next ELSE IF or ELSE branch. ELSE (IF) branches were also represented by `If` objects which, again, had `orelse`. Due to this nested structure branches needed to have both the normal `status` attribute (tells the overall status, including `orelse` status) and a separate `branch_status` attribute (tells the status of this particular branch). The nested structure was also used in output.xml: <if type='if' condition='$x > 0'> <if type='else'> <kw .../> <status status='PASS'/> </if> <status status='PASS' branchstatus='NOT RUN'/> </if> After this commit IF/ELSE structures are represented so that there's a root `If` object which has `body` containing actual IF/ELSE branches. This removes the need for `orelse` and `branch_status`, but requires adding a new model object representing the root. The old `If` class was renamed to `IfBranch` that now represent a single branch. Same structure used also in output.xml: <if> <branc type='if' condition='$x > 0'> <status status='NOT RUN'/> </branch> <branch type='else'> <kw .../> <status status='PASS'/> </if> <status status='PASS'/> </if> There are pros and cons with both ways to represent IF/ELSE structures, but overall the new approach feels more straightforward and is probably easier to understand and use for others. Notice that the parsing model still uses the `orelse` approach. There it works well and I don't think changing it would be worth the effort.

For example: - 'kw' -> 'KEYWORD' - 'foritem' -> 'FOR ITERATION' - 'elseif' -> 'ELSE IF' This is part of model refactoring (#3749). These enhanced type values are now also used with listeners. That fixes #3851. They are also used in output.xml. That needs to be documented in the schema (#3726).

pekkaklarck · 2021-02-11T16:47:15Z

IF/ELSE model model and type constants were enhanced by the comments linked above. Class, method and attribute names were enhanced in 00c282f

1. Internally store iteration variables in an OrderedDict instead as single string. 2. In output.xml store each variable name and value separately to be able to reconstruct variables. Part of model refactoring (#3749). Hopefully the final part.

pekkaklarck · 2021-02-12T10:32:39Z

Everything listed above now done. Closing this again.

pekkaklarck · 2021-03-03T22:42:20Z

As #3868 demonstrates, external tools using test.keywords.create() need to be updated because keywords is nowadays only a read-only property. The new way to write that is test.body.create_keyword(). This code is compatible both with the old and new model:

if hasattr(tc, 'body'):    # RF >= 4.0
    create_keyword = tc.body.create_keyword
else:    # RF < 4.0
    create_keyword = tc.keywords.create
create_keyword(name=kwname, args=args)

pekkaklarck added enhancement priority: medium backwards incompatible alpha 3 labels Oct 30, 2020

pekkaklarck added this to the v4.0 milestone Oct 30, 2020

yanne added alpha4 and removed alpha 3 labels Nov 16, 2020

pekkaklarck added beta 2 priority: high and removed alpha4 priority: medium labels Nov 25, 2020

pekkaklarck pushed a commit that referenced this issue Nov 27, 2020

model: setters for setup and teardown (#3782)

5dca989

Related to #3749.

vokiput pushed a commit to vokiput/robotframework that referenced this issue Dec 27, 2020

model: setters for setup and teardown (robotframework#3782)

71d34bf

Related to robotframework#3749.

pekkaklarck mentioned this issue Jan 4, 2021

How do I setup multiple keywords with robot api #3811

Closed

pekkaklarck mentioned this issue Jan 12, 2021

Relative order of messages and keywords is not preserved in log #2086

Closed

pekkaklarck added a commit that referenced this issue Jan 14, 2021

Fix creating keywords property that has been deprecated as part of #…

3baa89c

…3749.

pekkaklarck added a commit that referenced this issue Jan 14, 2021

Store keywords and messages in common 'body'.

fc09d21

This is part of model refactoring discussed in issue #3749. Storing keywords and messages in body, and preserving their order also in log files, fixes #2086.

pekkaklarck added a commit that referenced this issue Jan 14, 2021

Fix visitor interface.

56f3ebe

- Test/keyword setup/teardown were not visited. - Visiting FOR/IF was totally broken. Both problems due to the ongoing model refactoring (#3749).

pekkaklarck added a commit that referenced this issue Jan 18, 2021

Add Keyword attrs to For/If as deprecated properties.

959544f

This makes For/If being independent classes instead of extending Keyword less backwards incompatible. Related to #3749.

This was referenced Jan 18, 2021

Use standard XSD data types in output.xml #3829

Closed

Update and enhance output.xml schema #3726

Closed

pekkaklarck closed this as completed Jan 28, 2021

This was referenced Feb 11, 2021

robot.testdoc not generating Json properly for reserved keywords #3846

Open

Listener: Use consistent upper case type values with start/end_keyword #3851

Closed

pekkaklarck reopened this Feb 11, 2021

pekkaklarck closed this as completed Feb 12, 2021

pekkaklarck added beta 3 and removed beta 2 labels Feb 12, 2021

pekkaklarck mentioned this issue Feb 12, 2021

Support keywords and control structures with listener version 3 #3296

Closed

This was referenced Feb 26, 2021

Execution side refactoring on v4.0 #3668

Closed

AttributeError: 'keywords' attribute is read-only and deprecated since Robot Framework 4.0. Use 'body', 'setup' or 'teardown' instead. #3868

Closed

Support messages directly under test in output.xml #3762

Closed

abhishek12sarkar mentioned this issue Mar 12, 2021

'keywords' attribute is read-only and deprecated since RF 4.0. Use 'body', 'setup', or 'teardown' instead Snooz82/robotframework-datadriver#37

Closed

This was referenced Mar 15, 2021

AttributeError: 'keywords' attribute is read-only and deprecated since Robot Framework 4.0. Use 'body', 'setup' or 'teardown' instead. #3886

Closed

Visitor and listener interfaces can only access top level keywords #3887

Closed

pekkaklarck mentioned this issue Mar 30, 2021

Using BuiltIn().run_keyword in listener start/end_suite method creates invalid output.xml #3893

Closed

pekkaklarck mentioned this issue Oct 4, 2021

robot case only generator output.xml but not generator log.html and report.html #4040

Closed

pekkaklarck mentioned this issue Aug 25, 2023

Result model: Loudly deprecate not needed attributes and remove already deprecated ones #4846

Closed

bbornhau mentioned this issue Feb 21, 2025

Support for RF 4.0+ ericchiachi/BDD-Library#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor execution and result side model objects #3749

Refactor execution and result side model objects #3749

pekkaklarck commented Oct 30, 2020 •

edited

Loading

pekkaklarck commented Oct 30, 2020

pekkaklarck commented Nov 26, 2020

prescod commented Dec 12, 2020

pekkaklarck commented Dec 12, 2020

prescod commented Dec 13, 2020

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 28, 2021

pekkaklarck commented Jan 28, 2021

pekkaklarck commented Feb 1, 2021 •

edited

Loading

boakley commented Feb 2, 2021

pekkaklarck commented Feb 3, 2021

pekkaklarck commented Feb 11, 2021 •

edited

Loading

pekkaklarck commented Feb 11, 2021

pekkaklarck commented Feb 12, 2021

pekkaklarck commented Mar 3, 2021 •

edited

Loading

Refactor execution and result side model objects #3749

Refactor execution and result side model objects #3749

Comments

pekkaklarck commented Oct 30, 2020 • edited Loading

pekkaklarck commented Oct 30, 2020

pekkaklarck commented Nov 26, 2020

prescod commented Dec 12, 2020

pekkaklarck commented Dec 12, 2020

prescod commented Dec 13, 2020

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 18, 2021

pekkaklarck commented Jan 28, 2021

pekkaklarck commented Jan 28, 2021

pekkaklarck commented Feb 1, 2021 • edited Loading

boakley commented Feb 2, 2021

pekkaklarck commented Feb 3, 2021

pekkaklarck commented Feb 11, 2021 • edited Loading

pekkaklarck commented Feb 11, 2021

pekkaklarck commented Feb 12, 2021

pekkaklarck commented Mar 3, 2021 • edited Loading

pekkaklarck commented Oct 30, 2020 •

edited

Loading

pekkaklarck commented Feb 1, 2021 •

edited

Loading

pekkaklarck commented Feb 11, 2021 •

edited

Loading

pekkaklarck commented Mar 3, 2021 •

edited

Loading