• fix: reduce parallelism and increase worker size on go-test-race #15106 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @spikecurtis and the rest of your teammates on Graphite

@Emyrk @johnstcn @dannykopping @ammario WDYT?

My hope is that this will reduce occurrences of flakes like coder/internal#102 where things just get inexplicably slow for multiple seconds.

We could also consider throwing beefier hardware at the race tests, but I still think we need to explicitly limit parallelism, or Go will see the increased HW threads and do even more stuff in parallel.

We can also experiment with a 4-core runner to reduce CI costs if we get the same execution time. For context, using a smaller runner on test-e2e didn't impact execution time.

Edit:

We could also consider throwing beefier hardware at the race tests

My suggestion was exactly opposite to what you are saying.

With this PR, the go-test-race goes from about 6 min to 10 min elapsed time.

@matifali I think what's going on is that in CI, we are running too many things in parallel and starving individual tests of CPU they need to execute in something close to the amount of time they were designed for. That is, we need more resources for the go-test-race target, either by taking a longer elapsed time, or if that is unacceptable, running on better hardware.

@spikecurtis Feel free to use a larger runner if it reduces CI time. if a 16-core runner does this in 5 minutes it will cost us the same amount while reducing time.

@Emyrk, each job in the workflow is run on a new ephemeral VM. They are single tenant. See https://depot.dev/docs/github-actions/overview

@spikecurtis do you know if we're setting GOMAXPROCS on our test runners?

@spikecurtis do you know if we're setting GOMAXPROCS on our test runners?

I think we are not, which makes it default to the number of HW threads

I increased to 4 concurrent packages and the worker to a 16 core box and we are back at ~6min.

@spikecurtis do you know if we're setting GOMAXPROCS on our test runners?

I think we are not, which makes it default to the number of HW threads

Are we only running one CI job per host, or multiple in parallel?
I assume the latter.

In that case we could consider tuning GOMAXPROCS a bit.

Are we only running one CI job per host, or multiple in parallel?
I assume the latter.

Its one CI job per host, running on depot.dev VMs

Its one CI job per host, running on depot.dev VMs

Ah. In that case the default GOMAXPROCS is fine.