-
Notifications
You must be signed in to change notification settings - Fork 557
[v2] Randomize test order for unit tests #3434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
stephenfin
wants to merge
8
commits into
gophercloud:v2
Choose a base branch
from
shiftstack:randomize-tests-v2
base: v2
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
+8,013
−7,884
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Global tests objects are harmful, in so far as any attempt to modify them can result in failures further down the food chain. Don't use them and instead generate new objects on the fly. Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit 610f545)
The fake service clients rely on the fake server being initialised, even if we don't actually use it. Add the missing calls, renaming some tests in the process. PS: You can reproduce this without the '-shuffle on' by simply deleting any of the test cases preceding those that we are modifying here and running the tests. Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit 47b361b)
This is a mildly backwards-incompatible change, but since most people only use this API on the level of pagination.Pager, it ought not be a big deal in practice. Closes gophercloud#3222. (cherry picked from commit 97c0a32)
Ahead of some large-scale rework here. Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit 1476dae)
All aliases were discovered with: ag '[^\t]"github.com/gophercloud/gophercloud/v2/testhelper/client"' They were then replaced with judicious use of sed and ag. Starting with the most common alias, 'fake', first replace the alias: sed -i 's/\<fake\>/client/' $(ag 'fake "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l) This was repeated for the other aliases, 'fakeclient' and 'thclient'. We then dropped the now unnecessary aliasing: sed -i 's/\<client\> "/"/' $(ag 'client "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l) Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit beeed5b)
Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit 2f52485)
The meat of this changes is in 'testhelper/http_responses.go' and 'testhelper/client/fake.go'. The rest of the change is bulk generated updates to handle those changes: Create a "FakeServer" instance instead of configuring the global fake: sed -i 's/th.SetupHTTP/fakeServer := th.SetupHTTP/g' $(ag 'th.SetupHTTP' -l) Tear down this instance on test completion: sed -i 's/defer th.TeardownHTTP/defer fakeServer.Teardown/g' $(ag 'defer th.TeardownHTTP' -l) Update calls to the global 'Mux' instance with calls to the FakeServer version: sed -i 's/th.Mux.HandleFunc/fakeServer.Mux.HandleFunc/g' $(ag 'th.Mux.HandleFunc' -l) Update calls to ServiceClient to include a 'FakeServer' parameter: sed -i 's/client.ServiceClient()/client.ServiceClient(fakeServer)/' $(ag -w client.ServiceClient -l) Plus alises for the same: sed -i 's/fake.ServiceClient()/fake.ServiceClient(fakeServer)/' $(ag 'fake.ServiceClient' -l) Update definitions and calls to various MockFoo and HandleFoo helpers - plus lowercase, private variants of same - to include a 'FakeServer' parameter: sed -i 's/\(func Mock\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func Mock\w+\(t \*testing.T' -l) sed -i 's/\(\tMock\w\+(t\)/\1, fakeServer/' $(ag '\tMock\w+\(t' -l) sed -i 's/\(func mock\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func mock\w+\(t \*testing.T' -l) sed -i 's/\(\tmock\w\+(t\)/\1, fakeServer/' $(ag '\tmock\w+\(t' -l) sed -i 's/\(func Handle\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func Handle\w+\(t \*testing.T' -l) sed -i 's/\(\tHandle\w\+(t\)/\1, fakeServer/' $(ag '\tHandle\w+\(t' -l) sed -i 's/\(func handle\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func handle\w+\(t \*testing.T' -l) sed -i 's/\(\thandle\w\+(t\)/\1, fakeServer/' $(ag '\thandle\w+\(t' -l) Finally, there were some manual fixes to handle stuff that the regexes missed. Conflicts: openstack/baremetal/v1/portgroups/testing/fixtures.go openstack/baremetal/v1/portgroups/testing/requests_test.go openstack/compute/v2/servers/testing/requests_test.go openstack/db/v1/quotas/testing/fixtures_test.go openstack/db/v1/quotas/testing/requests_test.go openstack/dns/v2/zones/testing/requests_test.go openstack/loadbalancer/v2/flavors/testing/fixtures.go openstack/loadbalancer/v2/flavors/testing/requests_test.go openstack/networking/v2/extensions/security/rules/testing/requests_test.go Changes: openstack/blockstorage/v3/manageablevolumes/testing/fixtures_test.go openstack/blockstorage/v3/manageablevolumes/testing/requests_test.go openstack/blockstorage/v3/volumes/testing/fixtures_test.go openstack/blockstorage/v3/volumes/testing/requests_test.go openstack/networking/v2/extensions/security/addressgroups/testing/requests_test.go NOTE(stephenfin): Conflicts are generally due to tests or test files that don't exist on the v2 branch. Changes are due to tests that have been removed or moved on the main branch. Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit eed41cd)
I don't know why this isn't the default behavior, but relying on test order is a bad idea (TM) that we should do our best to avoid. Signed-off-by: Stephen Finucane <[email protected]> (cherry picked from commit 9251e99)
idk if we want to include fcf44f4 here. It's technically a breaking change but it's also a bugfix. Input welcome. |
@stephenfin I don't see anything breaking there. |
From the
|
I've never seen that someone used direct access to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
edit:baremetal
This PR updates baremetal code
edit:baremetalintrospection
This PR updates baremetalintrospection code
edit:blockstorage
This PR updates blockstorage code
edit:common
This PR updates common code
edit:compute
This PR updates compute code
edit:container
This PR updates container code
edit:containerinfra
This PR updates containerinfra code
edit:db
This PR updates db code
edit:dns
This PR updates dns code
edit:gophercloud
This PR updates common Gophercloud code
edit:identity
This PR updates identity code
edit:image
This PR updates image code
edit:keymanager
This PR updates keymanager code
edit:loadbalancer
This PR updates loadbalancer code
edit:messaging
This PR updates messaging code
edit:networking
This PR updates networking code
edit:objectstorage
This PR updates objectstorage code
edit:openstack
This PR updates common OpenStack code
edit:orchestration
This PR updates orchestration code
edit:placement
This PR updates placement code
edit:sharedfilesystems
This PR updates sharedfilesystems code
edit:testinfra
This PR updates testing infrastructure code
edit:utils
This PR updates utils code
edit:workflow
This PR updates workflow code
semver:major
Breaking change
v2
This PR targets v2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is manual backport of #3108, with trivial conflicts on the penultimate patch addressed.
This is not critical, but backporting it will ease future backports from the
main
branch.