Randomize test order for unit tests #3108
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2f29631 to
1bc42dd
Compare
mandre
reviewed
Jul 16, 2024
|
Looks like I've just uncovered a whole class of bugs 😂 😭 |
1bc42dd to
436fdbc
Compare
2e7bd84 to
d96065f
Compare
d96065f to
4de3d75
Compare
4de3d75 to
172a538
Compare
Global tests objects are harmful, in so far as any attempt to modify them can result in failures further down the food chain. Don't use them and instead generate new objects on the fly. Signed-off-by: Stephen Finucane <[email protected]>
The fake service clients rely on the fake server being initialised, even if we don't actually use it. Add the missing calls, renaming some tests in the process. PS: You can reproduce this without the '-shuffle on' by simply deleting any of the test cases preceding those that we are modifying here and running the tests. Signed-off-by: Stephen Finucane <[email protected]>
Ahead of some large-scale rework here. Signed-off-by: Stephen Finucane <[email protected]>
All aliases were discovered with:
ag '[^\t]"github.com/gophercloud/gophercloud/v2/testhelper/client"'
They were then replaced with judicious use of sed and ag. Starting with
the most common alias, 'fake', first replace the alias:
sed -i 's/\<fake\>/client/' $(ag 'fake "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l)
This was repeated for the other aliases, 'fakeclient' and 'thclient'. We
then dropped the now unnecessary aliasing:
sed -i 's/\<client\> "/"/' $(ag 'client "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l)
Signed-off-by: Stephen Finucane <[email protected]>
I don't know why this isn't the default behavior, but relying on test order is a bad idea (TM) that we should do our best to avoid. Signed-off-by: Stephen Finucane <[email protected]>
bdffc9d to
9251e99
Compare
|
Weird: the designate dalmatian job failed to deploy due to a missing dependency, but I haven't touched that and the upstream branch is mostly unchanged 😕 I've rerun the job in case it was a once-off. |
|
The networking test failed because of a duplicate security group rule. This is likely because we use I've rerun this too. |
No such luck. I am investigating. I have also asked the ironic team about the persistent ironic failures, since it seems related to a recently-merged feature in their devstack plugin. I expect the messaging tests to fail until we fix it. UPDATE: I've reproduced this locally on an Ubuntu 22.04 VM. This is not specific to this PR. |
EmilienM
approved these changes
May 22, 2025
PTAL: #3392 |
This was referenced Jun 19, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I don't know why this isn't the default behavior, but relying on test order is a bad idea (TM) that we should do our best to avoid.
This is currently only done for unit tests so that we can actually get something merged. We can extend to acceptance tests in a subsequent PR.