CI: Ensure code coverage is always uploaded #28164
Conversation
|
OK, this looks good; coverage is being uploaded even on failure on all the runs. I'm not sure if we should upload Cygwin coverage, since it's not done on all PRs too. Maybe there's a way to keep that separate. Also, the script to remove passing images is now working, though I see it doesn't really clean up as much as it can. One other thing is that it appears that Azure only uploads on Linux; it is using the Bash uploader, which is supposed to be broken, but still works there. However, it stopped working on other platforms. I think we would need to set up a token in order to get the new uploader working. |
|
I think preventing cygwin from uploading is probably fine. If having it would cause Azure may be worth either fixing (could be here or in a followup) or removing coverage entirely (if its not getting us additional coverage/hasn't been working anyway). I would leave any adjustments to the upload cleaner for a separate PR. |
If code coverage is not uploaded on failure, then overall coverage can suffer mysteriously (e.g., if a mac system failed, then it would appear as if `backend_macosx.py` was not tested). Codecov would then show several "indirect changes" that are spurious. On GitHub Actions, if you don't have any status check function in the `if` entry, then it is treated as `success() && (whatever else you had)`, so put in an explicit check. Azure also had no condition, so defaulted to only-on-success. Cygwin is only run on `main` and sometimes on PRs, so disable code coverage reporting from it. That would cause random "reductions" in coverage for all PRs that don't run it (which is most of them.) Also, fix the image-cleanup script, which would only run on success instead of failure, where it would be useful.
45d5dc0 to
25847dd
Compare
|
I removed Cygwin, as it would just cause extra flapping, since only |
|
Agree about both not uploading from cygwin and defering fixing uploading on non-linux azure to a follow on PR. |
|
Owee, I'm MrMeeseeks, Look at me. There seem to be a conflict, please backport manually. Here are approximate instructions:
And apply the correct labels and milestones. Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon! Remember to remove the If these instructions are inaccurate, feel free to suggest an improvement. |
…3.9.x Backport PR #28164 on branch v3.9.x (CI: Ensure code coverage is always uploaded)
PR summary
If code coverage is not uploaded on failure, then overall coverage can suffer mysteriously (e.g., if a mac system failed, then it would appear as if
backend_macosx.pywas not tested). Codecov would then show several "indirect changes" that are spurious.On GitHub Actions, if you don't have any status check function in the
ifentry, then it is treated assuccess() && (whatever else you had), so put in an explicit check.Azure also had no condition, so defaulted to only-on-success.
Also, fix the image-cleanup script, which would only run on success instead of failure, where it would be useful.
PR checklist