flake: TestProvision/returns-modules #322
Comments
|
Sorry for the delay, I completely forgot about this one. The root cause of the issue seems to be terraform downloading providers over and over again. In the first error a connection to registry.terraform.io failed, in the second - a connection to github.com. I think we could instead download the providers once and cache them between CI runs. I'd enforce that provider_installation {
filesystem_mirror {
path = "/path/to/local/providers"
include = ["*/*"]
}
direct {
exclude = ["*/*"]
}
}Each test would try to add its providers to the local mirror if it hasn't already while holding a system-wide lock with this command: terraform providers mirror /path/to/local/providersThis approach doesn't completely remove the dependency on the external registry, but it dramatically reduces the number of network requests required. In fact, if terraform tests don't change between runs, no new network requests should be made at all. Do you see any issues with this approach, @spikecurtis? If not, I'll implement it this week. |
|
Is there any way we can completely remove the dependency on the external registry? Unit tests like this should be fast, self-contained, and bullet proof. |
|
@spikecurtis From what I can see many of the TestProvider tests depend on the coder provider and check whether it behaves correctly. I don't think there's a way to remove the dependency without deleting the tests completely. |
|
Let's start with caching between runs in CI, and see if the flakes go away. |
|
I've implemented the coder-side caching logic in coder/coder#17373. I'm still working out the caching strategy in GitHub Actions. I'll have an update soon, and then the PR will be ready for review. |
Addresses coder/internal#322. This PR starts caching Terraform providers used by `TestProvision` in `provisioner/terraform/provision_test.go`. The goal is to improve the reliability of this test by cutting down on the number of network calls to external services. It leverages GitHub Actions cache, which [on depot runners is persisted for 14 days by default](https://depot.dev/docs/github-actions/overview#cache-retention-policy). Other than the aforementioned `TestProvision`, I couldn't find any other tests which depend on external terraform providers.
|
Closing the issue since coder/coder#17373 should address it. If the problem resurfaces, we can reopen it. |
https://github.com/coder/coder/actions/runs/12983674870/job/36205319350?pr=16265
Looks like this test requires reaching out to
registry.terraform.io, which is generally a bad idea in unit tests, since it subjects us to network issues, throttling/limits, etc.The text was updated successfully, but these errors were encountered: