Fix status check comment for timed out or failed checks #565
Conversation
Codecov Report
@@ Coverage Diff @@
## main #565 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 10 10
Lines 1012 1025 +13
Branches 71 72 +1
=========================================
+ Hits 1012 1025 +13
|
| elem in [None, "failure", "timed_out"] | ||
| for elem in all_check_run_conclusions | ||
| ) | ||
| success = result["state"] == "success" and not failure |
There was a problem hiding this comment.
I would like to understand better what's going on here, and possibly capture this in a comment.
result["state"] seems to be the final outcome of all the checks combined. I assume that some of the optional checks might fail or timeout without affecting this final result. I'm also guessing that all_check_run_conclusions might include the optional checks, even though I'm not sure in which case the result is None.
If this assumptions are right, we have a few different possible outcomes:
- the status check is done, and it's a success
- the status check is done, it's a success, but some optional tests failed/timed out
- the status check is done, but it's a failure
- the status check is (not?) done, but it's a failure because of a timeout
We could therefore keep reporting the result["state"], and then elaborate as described above. We could also create two variables for failures and timeouts if we want a more accurate message.
What do you think?
There was a problem hiding this comment.
Best example is python/cpython@40ccf13 from python/cpython#94734 where I originally spotted this bug.
The result['state'] comes from:
https://api.github.com/repos/python/cpython/commits/40ccf1321b819ce383d23841868004c18d368eb0/status
This refers to all checks from bedevere such as issue number and CLA signing.
failure is based on:
https://api.github.com/repos/python/cpython/commits/40ccf1321b819ce383d23841868004c18d368eb0/check-runs
These are all the Github Actions that get triggered. These do not include the checks that bedevere sets.
Thus, it is not so much about checks are required or optional, but rather about the source of the type of check.
Do we want to handle those differently?
There was a problem hiding this comment.
Is there any reason why the bedevere checks are listed separately from the other GitHub actions? Are they not considered checks?
There was a problem hiding this comment.
Bedevere creates statutes, see https://docs.github.com/en/rest/commits/statuses
Since they are not triggered by a GitHub Action directly they need to be recovered from another endpoint. Honestly, the amount of different types of checks GitHub provides is quite long and their documentation is not optimal. Of the top of my head I can think of the terms actions, check suites, check runs, checks, statutes. There are probably more 😅
| }, | ||
| } | ||
| ] | ||
| } |
There was a problem hiding this comment.
This object seem duplicated several time. Can it be defined once and reused? If not, could the "message" be trimmed and the object made more compact so that it takes 3 lines instead of several?
There was a problem hiding this comment.
The pattern in these tests seems to be redefining. Objects like items could also be re-used.
Do you want me to refactor? Or follow the current style with a shorter message?
There was a problem hiding this comment.
Perhaps we could refactor in a separate PR.
Co-authored-by: Ezio Melotti <[email protected]>
|
@DanielNoord, it seems there are still some issues, see e.g.: |
| if success: | ||
| emoji = "✅" | ||
| status = "it's a success" | ||
| if failure: | ||
| emoji = "❌" | ||
| status = "it's a failure or timed out" | ||
| else: | ||
| emoji = "❌" | ||
| message = f"Status check is done, and it's a {result['state']} {emoji} ." | ||
| status = "it's a failure" |
There was a problem hiding this comment.
@ezio-melotti Now that I think about this, this seem to be faulty if ... if logic.
If success is True failure is None and thus we will always go into the third path.
I'll open a PR to address this tomorrow. I'll tag you for a review so we can fix this and remove confusion from the cpython repo.
|
Follow up PR in #576 😄 |
Closes #564.
Split out the
"succes"check from the timed out check so we can record both states.