Make benchmarking script more fault-tolerant #7674
Conversation
25c621a to
3822e6b
Compare
3822e6b to
5c141e2
Compare
antiagainst
left a comment
There was a problem hiding this comment.
LGTM, thanks for adding this! But given this keep going feature does not have some count down, I assume it's only meant for local usage.
My main comment is mostly around having more comments explaining the intent. I know it's an art as when/where to comment to avoid being excessive; but generally if the logic is too branchy and lengthy I'd suggest to put more to keep following the logic. :)
Not sure what you mean by a countdown? My main use case was for running locally, although I think it would also be nice to be able to do this on CI so you could see the results. It would require plumbing more things through though, so I wouldn't do it right now.
Yeah, like I said, I don't really love this. There are a lot of switches and parameters all over the place 😕 I'll add some more comments and docstrings. |
(╯°□°)╯︵ ┻━┻ I hate dynamic languages |
|
FYI: this breaks uploading to dashboard: https://buildkite.com/iree/iree-benchmark/builds/1534#68e90f49-9c6e-4c4a-81ff-36d8a4895f57. Looks we missed one place that should be updated. Fix here: #7694. |
|
Argh, thanks for fixing |
This is not the prettiest Python I've ever written, but it does allow restarting benchmark runs rather than losing all progress after a single failure. This makes the workflow of starting a benchmarking run and then coming back when it is finished far more workable. I tried out incremental output of the final json results and reloading from that, but decided against it because I had to manually construct json (no native incremental support) and use context handlers to ensure structures were closed even on a failure exit. Overall it ended up being pretty gross. Since we were already using temporary files for captures, this seemed like a reasonable way to go.
This is not the prettiest Python I've ever written, but it does allow
restarting benchmark runs rather than losing all progress after a
single failure. This makes the workflow of starting a benchmarking run
and then coming back when it is finished far more workable.
I tried out incremental output of the final json results and reloading
from that, but decided against it because I had to manually construct
json (no native incremental support) and use context handlers to ensure
structures were closed even on a failure exit. Overall it ended up
being pretty gross. Since we were already using temporary files for
captures, this seemed like a reasonable way to go.