Run acceptance tests in Actions with a GitHub App token - #14356
Conversation
8ab3a6e to
e6759b8
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The helper validates groups against the current checkout rather than the selected test revision.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
script/run-acceptance — 🛑 Requirement: Validate command groups against the revision being tested When REF names a… |
What changed in this PR
Adds a manually dispatched, cross-platform acceptance workflow using a GitHub App installation token.
Changes:
- Adds the multi-platform acceptance workflow.
- Adds a dispatch helper with group and OS selection.
- Documents workflow operation and test-authoring guidance.
| File | Description |
|---|---|
.github/workflows/acceptance.yml |
Defines the App-token acceptance workflow. |
script/run-acceptance |
Dispatches selected refs, groups, and platforms. |
acceptance/README.md |
Documents workflow usage and App setup. |
.github/skills/writing-acceptance-tests/SKILL.md |
Clarifies dynamic group discovery. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
e6759b8 to
dd0891c
Compare
dd0891c to
59914ac
Compare
59914ac to
a74882f
Compare
a74882f to
b9f80d8
Compare
b9f80d8 to
4887fc9
Compare
4887fc9 to
fb27f6d
Compare
fb27f6d to
d5fb2b8
Compare
d5fb2b8 to
6790be4
Compare
6790be4 to
11e54f3
Compare
11e54f3 to
15a621b
Compare
15a621b to
87b84ca
Compare
87b84ca to
be33098
Compare
be33098 to
ae10cca
Compare
ae10cca to
46bca95
Compare
46bca95 to
85737a3
Compare
Co-authored-by: Copilot App <[email protected]> Copilot-Session: f67f4129-93de-45b1-a68d-702334aa7fe9
85737a3 to
ca0e191
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Ref validation uses the wrong checkout, and the documented dispatch and environment protections are contradictory.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
script/run-acceptance — 🛑 Requirement: Correct the claimed dispatch-ref behavior The helper always passes trunk to `gh… |
 |
acceptance/README.md — 🛑 Requirement: Reconcile the environment protection description This says the environment has no… |
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
script/run-acceptance — 🛑 Requirement: Validate command groups against the revision being tested When REF names a… View comment |
Suppressed comments (1)
script/run-acceptance:67
- 🛑 Requirement: Validate command groups against the revision being tested
This list comes from the helper's local checkout, not from ref. Running the helper from trunk to test a branch that adds a new group rejects that valid group before dispatch; the reverse accepts a group that the target revision does not contain and only fails in Actions. Please either resolve groups from the selected revision or defer this validation to the acceptance harness after checkout.
command_values=()
while IFS= read -r group; do
command_values+=("$group")
done < <(test_groups)
if ! contains "$command" "${command_values[@]}"; then
| GH_ACCEPTANCE_ORG: gh-acceptance-testing | ||
| GH_ACCEPTANCE_TOKEN: ${{ steps.acceptance-token.outputs.token }} | ||
| GH_ACCEPTANCE_GROUP: ${{ inputs.command }} | ||
| run: go test -tags=acceptance ./acceptance |
There was a problem hiding this comment.
question: do we want an explicit timeout other than go's default (10m)?
There was a problem hiding this comment.
I'm gonna start with saying no because if this takes more than 10 mins I want to know 🤣
Co-authored-by: Copilot App <[email protected]> Copilot-Session: bb46ae80-3bec-4092-bb94-65698edd9833
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Back-to-back runs can exceed the hourly repository limit, and group validation can differ from the dispatched revision.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 3
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/acceptance.yml — 🛑 Requirement: Add rate-limit pacing between full runs A three-platform run creates about 138… |
|
.github/workflows/acceptance.yml — 🛑 Requirement: Preserve every dispatched run With the default concurrency queue,… |
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
script/run-acceptance — 🛑 Requirement: Validate command groups against the revision being tested When REF names a… View comment |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
acceptance/README.md — 🛑 Requirement: Reconcile the environment protection description This says the environment has no… View resolved comment |
|
script/run-acceptance — 🛑 Requirement: Correct the claimed dispatch-ref behavior The helper always passes trunk to `gh… View resolved comment |
Suppressed comments (1)
script/run-acceptance:66
- 🛑 Requirement: Validate command groups against the dispatched revision
The allowed groups are read from this local checkout, but ref may select a different branch or tag. When the target revision adds or removes a group, this helper either rejects a valid target group or dispatches a group that the target harness rejects. Resolve the ref before validation and discover groups from that revision, or verify and document that the local checkout must match the requested ref.
command_values=()
while IFS= read -r group; do
command_values+=("$group")
done < <(test_groups)
if ! contains "$command" "${command_values[@]}"; then
Co-authored-by: Copilot App <[email protected]> Copilot-Session: bb46ae80-3bec-4092-bb94-65698edd9833
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The workflow performs broad live organization mutations with new credentials and has not been exercised end to end.
Review tier: Balanced
Findings: None
Issues resolved since last review (3)
| Severity | Finding |
|---|---|
|
.github/workflows/acceptance.yml — 🛑 Requirement: Preserve every dispatched run With the default concurrency queue,… View resolved comment |
|
.github/workflows/acceptance.yml — 🛑 Requirement: Add rate-limit pacing between full runs A three-platform run creates about 138… View resolved comment |
|
script/run-acceptance — 🛑 Requirement: Validate command groups against the revision being tested When REF names a… View resolved comment |
Description
The live acceptance suite is currently manual to assemble and depends on a maintainer supplying a user OAuth token. This adds a manually dispatched workflow that mints a GitHub App installation token for
gh-acceptance-testingand runs either the complete suite or one selected command group on Linux, macOS, Windows, or all three. A small dispatch helper discovers command groups from the testdata directories and dispatches the workflow from the selected branch or tag incli/cli, without requiring maintainers to expose a user token.This is the top layer of a three-layer native PR stack. It depends on #14354, which depends on #14320. The lower layers retain responsibility for token-capability policy and fixture repository/clone reductions; this layer does not change fixture classification or capability policy. The outcome is that maintainers can run the installation-compatible suite across all supported runner platforms while user-capability tests are skipped by the middle layer.
How did you test this change?
I ran
script/run-acceptance groupsand saw the discovered testdata groups, exercised its help and invalid platform paths, and replacedghwith a recording test double to confirm a Windows PR-group dispatch selects the requested branch with--refand sends the command and operating-system inputs toacceptance.yml. With a recorded in-progress run, I saw a warning before a non-interactive dispatch continued. In a pseudo-terminal, I answerednto the confirmation prompt and saw that no dispatch occurred. I did not dispatch the live workflow because it must first exist on the selected branch or tag.Key points
The workflow uses an installation-wide token because the suite dynamically creates repositories in the dedicated test organization. Each platform job mints its own one-hour token. The helper warns when another acceptance workflow is in flight and asks interactive users to confirm another dispatch because overlapping or closely spaced runs can exhaust the App installation's rate-limit buckets. Non-interactive callers receive the warning and continue.
The workflow is dispatched from the branch or tag being tested, so GitHub limits the selectable revision to refs in
cli/clirather than accepting an arbitrary checkout ref. Before App credentials become available, thegh-acceptance-testingenvironment requires approval fromcli/code-reviewers; self-review and administrator bypass are disabled.Notes for reviewers
Start with
.github/workflows/acceptance.yml, then reviewscript/run-acceptanceand the incremental workflow guidance inacceptance/README.md.Authorship and follow-up
Who wrote this:
Who answers review comments: