Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix status check comment for timed out or failed checks #565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 1, 2022
+83 −9
Merged

Fix status check comment for timed out or failed checks #565

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
61d5ace
Fix status check comment for timed out or failed checks
DanielNoord Jul 11, 2022
b536399
Fix coverage issue
DanielNoord Jul 11, 2022
1ffbfb0
Remove unrelated change
DanielNoord Jul 11, 2022
e7af1b0
Last unrelated change
DanielNoord Jul 11, 2022
f358e2f
Update tests/test_status_change.py
DanielNoord Aug 8, 2022
00cf6d3
Merge branch 'main' into fail-success
ezio-melotti Aug 18, 2022
14e0903
Merge branch 'main' into fail-success
ezio-melotti Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions miss_islington/status_change.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,16 +100,22 @@ async def check_ci_status_and_approval(

title_match = TITLE_RE.match(normalized_pr_title)
if title_match or is_automerge:
success = result["state"] == "success" and not any(
failure = any(
elem in [None, "failure", "timed_out"]
for elem in all_check_run_conclusions
)
success = result["state"] == "success" and not failure
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to understand better what's going on here, and possibly capture this in a comment.
result["state"] seems to be the final outcome of all the checks combined. I assume that some of the optional checks might fail or timeout without affecting this final result. I'm also guessing that all_check_run_conclusions might include the optional checks, even though I'm not sure in which case the result is None.

If this assumptions are right, we have a few different possible outcomes:

  • the status check is done, and it's a success
  • the status check is done, it's a success, but some optional tests failed/timed out
  • the status check is done, but it's a failure
  • the status check is (not?) done, but it's a failure because of a timeout

We could therefore keep reporting the result["state"], and then elaborate as described above. We could also create two variables for failures and timeouts if we want a more accurate message.

What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best example is python/cpython@40ccf13 from python/cpython#94734 where I originally spotted this bug.

The result['state'] comes from:
https://api.github.com/repos/python/cpython/commits/40ccf1321b819ce383d23841868004c18d368eb0/status

This refers to all checks from bedevere such as issue number and CLA signing.

failure is based on:
https://api.github.com/repos/python/cpython/commits/40ccf1321b819ce383d23841868004c18d368eb0/check-runs

These are all the Github Actions that get triggered. These do not include the checks that bedevere sets.

Thus, it is not so much about checks are required or optional, but rather about the source of the type of check.

Do we want to handle those differently?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why the bedevere checks are listed separately from the other GitHub actions? Are they not considered checks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bedevere creates statutes, see https://docs.github.com/en/rest/commits/statuses

Since they are not triggered by a GitHub Action directly they need to be recovered from another endpoint. Honestly, the amount of different types of checks GitHub provides is quite long and their documentation is not optimal. Of the top of my head I can think of the terms actions, check suites, check runs, checks, statutes. There are probably more 😅

if leave_comment:
if success:
emoji = "✅"
status = "it's a success"
if failure:
emoji = "❌"
status = "it's a failure or timed out"
else:
emoji = "❌"
message = f"Status check is done, and it's a {result['state']} {emoji} ."
status = "it's a failure"
Comment on lines 109 to +117
Copy link
Contributor Author

@DanielNoord DanielNoord Sep 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ezio-melotti Now that I think about this, this seem to be faulty if ... if logic.

If success is True failure is None and thus we will always go into the third path.

I'll open a PR to address this tomorrow. I'll tag you for a review so we can fix this and remove confusion from the cpython repo.

message = f"Status check is done, and {status} {emoji}."
if not success:
if is_automerge:
participants = await util.get_gh_participants(gh, pr_number)
Expand Down
26 changes: 24 additions & 2 deletions tests/test_check_run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,18 @@ async def test_check_run_completed_other_check_run_pending_with_awaiting_merge_l
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}

gh = FakeGH(getitem=getitem, getiter=getiter)
await check_run.router.dispatch(event, gh)
assert not hasattr(gh, "post_data") # does not leave a comment
assert not hasattr(gh, "put_data") # is not merged
Expand Down Expand Up @@ -326,7 +337,18 @@ async def test_check_run_completed_timed_out_with_awaiting_merge_label_pr_is_not
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}

gh = FakeGH(getitem=getitem, getiter=getiter)
await check_run.router.dispatch(event, gh)
assert len(gh.post_data["body"]) is not None # leaves a comment
assert not hasattr(gh, "put_data") # is not merged
Expand Down
56 changes: 51 additions & 5 deletions tests/test_status_change.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,18 @@ async def test_ci_passed_and_check_run_failure_awaiting_merge_label_pr_is_not_me
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}

gh = FakeGH(getitem=getitem, getiter=getiter)
await status_change.router.dispatch(event, gh)
assert len(gh.post_data["body"]) is not None # leaves a comment
assert not hasattr(gh, "put_data") # is not merged
Expand Down Expand Up @@ -374,7 +385,18 @@ async def test_automerge_with_check_run_failure():
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}

gh = FakeGH(getitem=getitem, getiter=getiter)
await status_change.router.dispatch(event, gh)
assert len(gh.post_data["body"]) is not None # leaves a comment
assert not hasattr(gh, "put_data") # is not merged
Expand Down Expand Up @@ -431,7 +453,18 @@ async def test_ci_passed_and_check_run_pending_awaiting_merge_label_pr_is_not_me
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}

gh = FakeGH(getitem=getitem, getiter=getiter)
await status_change.router.dispatch(event, gh)
assert len(gh.post_data["body"]) is not None # leaves a comment
assert not hasattr(gh, "put_data") # is not merged
Expand Down Expand Up @@ -488,9 +521,22 @@ async def test_ci_passed_and_check_run_timed_out_awaiting_merge_label_pr_is_not_
},
}

gh = FakeGH(getitem=getitem)
getiter = {
"/repos/python/cpython/pulls/5547/commits": [
{
"sha": "f2393593c99dd2d3ab8bfab6fcc5ddee540518a9",
"commit": {
"message": "bpo-32720: Fixed the replacement field grammar documentation. (GH-5544)\n\n`arg_name` and `element_index` are defined as `digit`+ instead of `integer`.\n(cherry picked from commit 7a561afd2c79f63a6008843b83733911d07f0119)\n\nCo-authored-by: Mariatta <[email protected]>"
},
}
]
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This object seem duplicated several time. Can it be defined once and reused? If not, could the "message" be trimmed and the object made more compact so that it takes 3 lines instead of several?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pattern in these tests seems to be redefining. Objects like items could also be re-used.

Do you want me to refactor? Or follow the current style with a shorter message?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could refactor in a separate PR.


gh = FakeGH(getitem=getitem, getiter=getiter)
await status_change.router.dispatch(event, gh)
assert len(gh.post_data["body"]) is not None # leaves a comment
expected_body = ("@miss-islington and @Mariatta: Status check is done, "
"and it's a failure or timed out ❌.")
assert gh.post_data["body"] == expected_body
assert not hasattr(gh, "put_data") # is not merged


Expand Down