Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Alternative status when suite setup or suite teardown fail. #5346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
gohierf opened this issue Feb 20, 2025 · 1 comment
Open

Alternative status when suite setup or suite teardown fail. #5346

gohierf opened this issue Feb 20, 2025 · 1 comment

Comments

@gohierf
Copy link
Contributor

gohierf commented Feb 20, 2025

Enhancement Request

This is my use case:

  • When my suite setup fails, I would like to mark the tests as skipped rather than failed. And I would like the final suite status to be failed.
  • When my suite teardown fails, I would like to preserve each tests results, rather than have them marked as failed. And I would like the final suite status to be failed.

The current behaviour is as described in the documentation:

2.4.6 Suite setup and teardown
...

If a suite setup fails, all test cases in it and its child test suites are immediately assigned a fail status and they are not actually executed. [...]

[...] If the suite teardown fails, all test cases in the suite are marked failed, regardless of their original execution status. [...]

...

Maybe my request could be implemented as an alternative behaviour that could be selected from the CLI?
For example: --skip-tests-on-suite-setup-failure and --keep-test-status-on-suite-teardown-failure.

It would also be nice to be able to activate from inside the file using tags: robot:skip-tests-on-suite-setup-failure and robot:keep-test-status-on-suite-teardown-failure. Though I am not sure how this would be possible since tags are attached to tests rather than suites.

Current work around

Currently, I have tried to use try/except statement, which kinds of works but comes with limitations (example here). The tests are marked as skipped in case of suite setup failure, but the whole suite is not marked as failed. Same for suite teardown failure, I can preserve the tests results, but not change the suite setup to failed. Moreover, my keywords have limitations: it is not possible to provide arguments to the keywords and tools like RobotCode do not detect my keywords as used when I call them through my custom Run Suite Setup Keywords and Run Suite Teardown Keywords.

Do you have other suggestions on how to implement this in a custom way if the change request is denied? I would prefer avoiding listeners or post-processing.
@pekkaklarck
Copy link
Member

Changing how suite setup is handled could be accomplished with the following changes:

  1. Change the algorithm to get the suite status so that if suite setup/teardown is failed, the status is FAIL regardless of test and child suite statuses. Then you could have a suite with skipped tests but failed setup and the status would be what you want.

  2. Change handling failures in suite setup so that tests in the suite are marked skipped instead of failed. This would give exactly the behavior you want.

The first change shouldn't be too problematic and I believe it would make sense in general. The second change would be problematic for the following reasons, though:

  • The report file is very much test case based and doesn't show suite setup/teardown at all. You thus could have a situation where all tests have succeeded, but a failed suite setup in some suite makes the overall status failed. Fixing this requires major changes to log/report. We've been planning giving them an overhaul, but there's no timeline for that.
  • The return code to the system from a test execution is currently the number of failed tests. That needed to be changed, because we absolutely don't want to return zero if the suite status is failed.
  • The change would be backwards incompatible. If it would be considered important enough, it could be done in a major version, though.
  • Marking tests skipped if suite setup fails would be somewhat odd, because they are marked skipped also if suite setup is skipped. Setup failure leading to test failures and setup skip leading to tests being skipped sounds more logical to me.

Due to the reasons explained above, I don't think change 2. is a good idea at least until log/report are enhanced. It would have a benefit that we could use the same approach when handling suite teardowns, though.

Change 1. ought to be fine already in RF 7.3, but it alone doesn't directly help you. You custom suite setup with TRY/EXCEPT apparently wouldn't have FAIL status, so if all tests are skipped then the overall status is SKIP as well. It would be possible to change the suite setup status to FAIL after execution, but that's not too convenient. Perhaps it would be better than the current situation, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pekkaklarck @gohierf