Set up continous benchmarking - #1075
Conversation
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
sobolevn
left a comment
There was a problem hiding this comment.
Awesome! I think that we also need a import msgspec timing test! 👍
Added, with some caveats |
sobolevn
left a comment
There was a problem hiding this comment.
Awesome! Thanks a lot for your work!
Co-authored-by: sobolevn <[email protected]>
|
Mhm. Seems like codspeed is indeed a bit unstable: #962 (comment), because checks are running in different envs. Re-ran these 3x now. When run in the same env, they do same stable, but getting the same env as the previous runs, with 3 jobs seems like quite a bit of a gamble. Could probably improve this with self-hosted runners. |
|
I tried to run the manual profiling job to check if the results are similar to Codspeed but it doesn't work now https://github.com/msgspec/msgspec/actions/runs/27883637649/job/82515828562 |
Probably because Also, the results won't be comparable at all, since the codspeed tests run via the CPU simulation |
|
So, I set up the threshold to 2%, down from the 10% default, and now pretty much every PR is failing again. Seems that there's still a lot of variance. @ofek can we try setting up macro runners? I don't have the permissions for that: https://codspeed.io/docs/features/macro-runners#repository-access-configuration |
|
I just followed that guide so everything should be set up now. Yes, I don't have high confidence due to the current variance and was trying to run the old workflow to verify. |
I feel the same. Which is a shame because otherwise codspeed is really nice. |
|
Do you think using the same measurement framework as CPython would help us with our micro-benchmarks? https://github.com/psf/pyperf |
|
I'm not sure what we can do on GitHub-provided runners but what about some of the advice here? https://google.github.io/benchmark/reducing_variance.html I found a nice quickstart setup gist https://gist.github.com/itzmeanjan/05dc3e946f635d00c5e0b21aae6203a7 |
Unfortunately not, as on-machine variance isn't our issue. As long as we get the same runner hardware, the results are stable. What tools like these can help with is stabilising results across runs on the same hardware, which codspeed already does through similar means.
I'll check those out! |
Set up continous benchmarking via codespee.io. Closes #310.
For reasoning and decision making process, refer to #310 (comment)
Re-implement existing benchmarks in
/benchmarkas pytest style benchmarks in/tests/prof/perf, so they work with codspeed. I've kept around the existing benchmarks, for local use and to produce comparisons against other libraries. Over time we might chose to consolidate them, but I think we should evaluate codspeed for a bit before doing so.Limitations
Benchmarks only run on Linux, because codspeeds instrumentation requires a Linux toolchain. Wallclock benchmarking is available for macOS as well, however, much larger variance is expected there.