Add acceptance tests for `repo` commands #9783

jtmcg · 2024-10-17T19:00:02Z

Closes https://github.com/github/cli/issues/589

Changes

Implements a suite of acceptance tests for the gh repo command, including coverage of the following sub-commands:

archive
clone
create
delete
deploy-key
- add
- delete
- list
edit
fork
list
rename
set-default
sync
unarchive
view

Some of these commands are tested in the same files, such as gh repo fork and gh repo sync in the repo-fork-sync.txtar file, as the testing for one command (gh repo sync, in this example) is either enabled or made easier by the use of the other.

Question/Concern

The way that testing for gh repo deploy-key add is set up now will result in a new deploy key being set up on behalf of whoever runs this test. That results in an email to the user saying a new deploy key was added.

I doubt there's any security risk, here, as the private key has been deleted and will remain unused and the repo gets deleted after the test is running, meaning the public key is now useless but for its format for testing purposes.

Ultimately, however, we'll probably want to change the authentication of these tests to a bot user - especially if/when these become part of our CI pipeline - and that will eliminate the emails. It's probably fine for now, as the only way we're going to run these tests is locally, but I wanted to call it out here.

What am I missing? What concerns with the current approach do y'all have?

Testing

Pull down branch: gh pr checkout 9783
Run the acceptance tests in the repo directory:

GH_ACCEPTANCE_HOST=<HOST> \
GH_ACCEPTANCE_ORG=<ORG> \
GH_ACCEPTANCE_TOKEN=<TOKEN> \
go test -tags=acceptance -run ^TestRepos$ ./acceptance

acceptance/testdata/repos/repo-edit.txtar

acceptance/testdata/repos/repo-create-view.txtar

acceptance/testdata/repos/repo-list-rename.txtar

acceptance/testdata/repos/repo-set-default.txtar

jtmcg · 2024-10-17T21:26:49Z

acceptance/testdata/repos/repo-archive-unarchive.txtar

+# Check that the repo exists and isn't archived
+exec gh repo view $ORG/$SCRIPT_NAME-$RANDOM_STRING --json=isArchived --jq='.isArchived'
+stdout false


Cross referencing a conversation from @andyfeller's PR:

Paraphrasing @williammartin comment:

Can you talk to me a bit about why you feel you need to check a [precondition] before proceeding? How's this moving the needle on your confidence levels?

I think for me it was about instilling confidence that the thing I'm going to assert against to define success of a command was indeed changed or affected. It was the loop I did myself in the terminal to gain confidence that the commands were working as expected.

There is an additional effect in that it protects against unintended changes to a command's default behavior. In my example, if, for some reason, gh repo create started automatically creating archived repos, we'd catch it here. Maybe that's out of scope of my test? 🤔

Thanks for the explanation.

I think that "hey script reader, up front here's the thing this specific test cares about changing and I'm communicating that to you" is probably the thing that resonates with me most. That's why in the other PR I suggested that if we did have these precondition checks that they are the same assertion differing only in whether they are expecting or refuting state.

I think that if we wanted to validate that repos are created in an unarchived form, then that should be tested against repo create directly, so that the confidence in that command is dependent on some other command that may or may not be testing it.

But I want to be clear here, I'm not saying that these should be removed. I am saying that I think it's important that there is a clear and shared pattern to all the testscripts that we can say "make it look like that". That can also mean going back to the scripts that are already being merged and making them use this pre and post pattern.

"hey script reader, up front here's the thing this specific test cares about changing and I'm communicating that to you"

This is definitely my primary motivation for doing it. I hear you on the pattern. That's probably a good chat to have synchronously. Maybe we talk about it in retro?

williammartin

Lots of good stuff in here, reads well to me. As mentioned in the comment, I can see the value of the pre-condition post-condition check when the commands are duals / mirrors of each other.

acceptance/testdata/repos/repo-fork-sync.txtar

acceptance/testdata/repos/repo-archive-unarchive.txtar

acceptance/testdata/repos/repo-clone.txtar

acceptance/testdata/repos/repo-delete.txtar

williammartin · 2024-10-18T14:52:33Z

acceptance/testdata/repos/repo-deploy-key.txtar

+exec gh repo deploy-key list --json=title,id --jq='.[].title="myTitle" | .[].id'
+stdout2env DEPLOY_KEY_ID
+
+# Delete the deploy key
+exec gh repo deploy-key delete $DEPLOY_KEY_ID
+
+# Ensure the deploy key was deleted
+exec gh repo deploy-key list --json=id --jq='.[].id'
+! stdout $DEPLOY_KEY_ID


Very clever incantations 👍

williammartin · 2024-10-18T14:57:20Z

acceptance/testdata/repos/repo-fork-sync.txtar

+defer gh repo delete $ORG/$SCRIPT_NAME-$RANDOM_STRING-fork --yes
+
+# Sleep so that the BE has time to sync
+sleep 5


Question

I was noodling on something with @BagToad yesterday that I want to throw at you. How would you feel if instead of:

# Sleep so that the BE has time to sync sleep 5 # Check that the repo was forked exec gh repo view $ORG/$SCRIPT_NAME-$RANDOM_STRING-fork --json='isFork' --jq='.isFork' stdout true

We had something along the lines of:

# Check that the repo was forked retryableexec gh repo view $ORG/$SCRIPT_NAME-$RANDOM_STRING-fork --json='isFork' --jq='.isFork' eventuallystdout true

Where the command would be repeated on loop every n milliseconds up to m times.

Working example seen here:

f98f45f#diff-0baffab94029cd95fd0c6237468ce85738db4ea07d63397348367930b7ad8833

I would absolutely love that! I had briefly thought about some sort of polling solution in place of sleep, but thought that since sleep got the job done, here, I'd punt it until later. As such, this is right in line with what I would expect, here 👍

Yes totally 100% let's keep sleep. There are some edge cases to figure out around the other one, I just wanted to float it to get your opinion.

acceptance/testdata/repos/repo-fork-sync.txtar

williammartin

Approving with the understanding that you address the last review comment (either by doing it or rejecting it).

➜  cli git:(jtmcg/testscripts-repo) set -o pipefail && GH_ACCEPTANCE_HOST=github.com GH_ACCEPTANCE_ORG=gh-acceptance-testing go test -tags acceptance -json -coverprofile=coverage.out -coverpkg=./... -run ^TestRepos$ github.com/cli/cli/v2/acceptance | tparse --all go test
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │               TEST               │             PACKAGE               │
│─────────┼─────────┼──────────────────────────────────┼───────────────────────────────────│
│  PASS   │   18.09 │ TestRepos/repo-fork-sync         │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    5.99 │ TestRepos/repo-deploy-key        │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    5.93 │ TestRepos/repo-archive-unarchive │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    4.47 │ TestRepos/repo-list-rename       │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    4.23 │ TestRepos/repo-edit              │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    4.09 │ TestRepos/repo-set-default       │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    3.62 │ TestRepos/repo-clone             │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    3.50 │ TestRepos/repo-create-view       │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    3.38 │ TestRepos/repo-delete            │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    0.00 │ TestRepos                        │ github.com/cli/cli/v2/acceptance  │
└──────────────────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │             PACKAGE              │ COVER │ PASS │ FAIL │ SKIP  │
│─────────┼─────────┼──────────────────────────────────┼───────┼──────┼──────┼───────│
│  PASS   │ 19.14s  │ github.com/cli/cli/v2/acceptance │ 12.4% │  10  │  0   │  0    │
└────────────────────────────────────────────────────────────────────────────────────┘

acceptance/acceptance_test.go

Makefile README.md acceptance api bin build cmd context docs git go.mod go.sum internal pkg script share test utils commands with

jtmcg · 2024-10-18T16:46:28Z

Force pushes to sync with trunk and clean up history

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.59.0` -> `v2.62.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.62.0`](https://github.com/cli/cli/releases/tag/v2.62.0): GitHub CLI 2.62.0 [Compare Source](cli/cli@v2.61.0...v2.62.0) #### What's Changed - Update monotonic verification logic and testing by [@malancas](https://github.com/malancas) in cli/cli#9856 - Check extension for latest version when executed by [@andyfeller](https://github.com/andyfeller) in cli/cli#9866 - Shorten extension release checking from 3s to 1s by [@andyfeller](https://github.com/andyfeller) in cli/cli#9914 - Mention GitHub CLI team on discussion issues by [@andyfeller](https://github.com/andyfeller) in cli/cli#9920 **Full Changelog**: cli/cli@v2.61.0...v2.62.0 #### Security - A security vulnerability has been identified in GitHub CLI that could allow remote code execution (RCE) when users connect to a malicious Codespace SSH server and use the `gh codespace ssh` or `gh codespace logs` commands. For more information, see GHSA-p2h2-3vg9-4p87 #### GitHub CLI notifies users about latest extension upgrades Similar to the notification of latest `gh` releases, the `v2.62.0` version of GitHub CLI will notify users about latest extension upgrades when the extension is used: ```shell $ gh ado2gh ... A new release of ado2gh is available: 1.7.0 → 1.8.0 To upgrade, run: gh extension upgrade ado2gh --force https://github.com/github/gh-ado2gh ``` ##### Why does this matter? This removes a common pain point of extension authors as they have had to reverse engineer and implement a similar mechanism within their extensions directly. With this quality of life improvement, there are 2 big benefits: 1. Extension authors will hopefully see increased adoption of newer releases while having lower bar to maintaining their extensions. 2. GitHub CLI users will have greater awareness of new features, bug fixes, and security fixes to the extensions used. ##### What do you need to do? Extension authors should review their extensions and consider removing any custom logic previously implemented to notify users of new releases. ### [`v2.61.0`](https://github.com/cli/cli/releases/tag/v2.61.0): GitHub CLI 2.61.0 [Compare Source](cli/cli@v2.60.1...v2.61.0) #### Ensure users understand consequences before making repository visibility changes In `v2.61.0`, `gh repo edit` command has been enhanced to inform users about [consequences of changing visibility](https://gh.io/setting-repository-visibility) and ensure users are intentional before making irreversible changes: 1. Interactive `gh repo edit` visibility change requires confirmation when changing from `public`, `private`, or `internal` 2. Non-interactive `gh repo edit --visibility` change requires new `--accept-visibility-change-consequences` flag to confirm 3. New content to inform users of consequences - Incorporate [GitHub Docs content](https://gh.io/setting-repository-visibility) into help usage and interactive `gh repo edit` experience - Expanded help usage to call out most concerning consequences - Display repository star and watcher counts to understand impact before confirming #### What's Changed - Add acceptance test for `project` command by [@jtmcg](https://github.com/jtmcg) in cli/cli#9816 - Add comprehensive testscript for `gh ruleset` by [@andyfeller](https://github.com/andyfeller) in cli/cli#9815 - Add comprehensive testscript for gh ext commandset by [@andyfeller](https://github.com/andyfeller) in cli/cli#9810 - Require visibility confirmation in `gh repo edit` by [@andyfeller](https://github.com/andyfeller) in cli/cli#9845 - Clean up skipped online tests for `gh attestation verify` by [@malancas](https://github.com/malancas) in cli/cli#9838 - `gh attestation verify` should only verify provenance attestations by default by [@malancas](https://github.com/malancas) in cli/cli#9825 - Set `dnf5` commands as default by [@its-miroma](https://github.com/its-miroma) in cli/cli#9844 - Fix verbiage for deleting workflow runs by [@akx](https://github.com/akx) in cli/cli#9876 - Bump github.com/creack/pty from 1.1.23 to 1.1.24 by [@dependabot](https://github.com/dependabot) in cli/cli#9862 - `gh attestation verify` policy enforcement refactor by [@malancas](https://github.com/malancas) in cli/cli#9848 - Simplify Sigstore verification result handling in `gh attestation verify` by [@malancas](https://github.com/malancas) in cli/cli#9877 - Print empty array for `gh cache list` when `--json` is provided by [@williammartin](https://github.com/williammartin) in cli/cli#9883 - Bump actions/attest-build-provenance from 1.4.3 to 1.4.4 by [@dependabot](https://github.com/dependabot) in cli/cli#9884 - Create the automatic key when specified with -i by [@cmbrose](https://github.com/cmbrose) in cli/cli#9881 - fix: `gh pr create -w` ignore template flag by [@nilvng](https://github.com/nilvng) in cli/cli#9863 #### New Contributors - [@akx](https://github.com/akx) made their first contribution in cli/cli#9876 - [@nilvng](https://github.com/nilvng) made their first contribution in cli/cli#9863 **Full Changelog**: cli/cli@v2.60.1...v2.61.0 ### [`v2.60.1`](https://github.com/cli/cli/releases/tag/v2.60.1): GitHub CLI 2.60.1 [Compare Source](cli/cli@v2.60.0...v2.60.1) This is a small patch release to fix installing `gh` via `go install` which was broken with v2.60.0. #### What's Changed - Update testscript to use hard fork by [@williammartin](https://github.com/williammartin) in cli/cli#9821 **Full Changelog**: cli/cli@v2.60.0...v2.60.1 ### [`v2.60.0`](https://github.com/cli/cli/releases/tag/v2.60.0): GitHub CLI 2.60.0 [Compare Source](cli/cli@v2.59.0...v2.60.0) #### What's Changed - Add ArchivedAt field by [@tsukasaI](https://github.com/tsukasaI) in cli/cli#9790 - Include startedAt, completedAt in run steps data by [@andyfeller](https://github.com/andyfeller) in cli/cli#9774 - Adjust environment help for host and tokens by [@williammartin](https://github.com/williammartin) in cli/cli#9809 - Add handling of empty titles for Issues and MRs by [@jtmcg](https://github.com/jtmcg) in cli/cli#9701 - `LiveSigstoreVerifier.Verify` should error if no attestations are present by [@phillmv](https://github.com/phillmv) in cli/cli#9742 - `gh at verify` retries fetching attestations if it receives a 5xx by [@phillmv](https://github.com/phillmv) in cli/cli#9797 - Prevent local extension installations with invalid names and conflicts with core commands and other extensions by [@BagToad](https://github.com/BagToad) in cli/cli#9794 - Rewrite a sentence in CONTRIBUTING.md by [@muzimuzhi](https://github.com/muzimuzhi) in cli/cli#9772 - Use new GitHub preview terms in `working-with-us.md` by [@BagToad](https://github.com/BagToad) in cli/cli#9800 - Use new GitHub previews terminology in attestation commands' help docs by [@BagToad](https://github.com/BagToad) in cli/cli#9799 - Clarify in README that `gh` is supported on GitHub Enterprise Cloud by [@BagToad](https://github.com/BagToad) in cli/cli#9805 - build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.5 to 1.4.6 by [@dependabot](https://github.com/dependabot) in cli/cli#9752 ##### Acceptance Test Changes - Add acceptance tests for `workflow`, `run`, and `cache` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9766 - Add basic `api` acceptance tests by [@BagToad](https://github.com/BagToad) in cli/cli#9770 - Add acceptance tests for `release` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9771 - Add acceptance tests for `org` and `ssh-key` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9812 - Add acceptance tests for `gh auth` commands by [@jtmcg](https://github.com/jtmcg) in cli/cli#9787 - Add acceptance tests for `repo` commands by [@jtmcg](https://github.com/jtmcg) in cli/cli#9783 - Add acceptance tests for `search` command by [@BagToad](https://github.com/BagToad) in cli/cli#9786 - Add acceptance tests for `variable` commands by [@andyfeller](https://github.com/andyfeller) in cli/cli#978 - Add testscripts for gpg-key and label commands by [@williammartin](https://github.com/williammartin) in cli/cli#9811 - Use forked testscript for token redaction by [@williammartin](https://github.com/williammartin) in cli/cli#9804 - Add acceptance tests for `secret` commands by [@andyfeller](https://github.com/andyfeller) in cli/cli#9782 - Note token redaction in Acceptance test README by [@williammartin](https://github.com/williammartin) in cli/cli#9813 #### New Contributors - [@tsukasaI](https://github.com/tsukasaI) made their first contribution in cli/cli#9790 **Full Changelog**: cli/cli@v2.59.0...v2.60.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

jtmcg commented Oct 17, 2024

View reviewed changes

acceptance/testdata/repos/repo-edit.txtar Outdated Show resolved Hide resolved

jtmcg commented Oct 17, 2024

View reviewed changes

acceptance/testdata/repos/repo-create-view.txtar Outdated Show resolved Hide resolved

jtmcg commented Oct 17, 2024

View reviewed changes

acceptance/testdata/repos/repo-list-rename.txtar Outdated Show resolved Hide resolved

jtmcg commented Oct 17, 2024

View reviewed changes

acceptance/testdata/repos/repo-list-rename.txtar Outdated Show resolved Hide resolved

jtmcg commented Oct 17, 2024

View reviewed changes

acceptance/testdata/repos/repo-set-default.txtar Outdated Show resolved Hide resolved

jtmcg force-pushed the jtmcg/testscripts-repo branch from 6050789 to a3fee07 Compare October 17, 2024 20:40

jtmcg marked this pull request as ready for review October 17, 2024 21:02

jtmcg requested a review from a team as a code owner October 17, 2024 21:02

jtmcg requested a review from andyfeller October 17, 2024 21:02

cliAutomation added the external pull request originating outside of the CLI core team label Oct 17, 2024

jtmcg removed the external pull request originating outside of the CLI core team label Oct 17, 2024

jtmcg commented Oct 17, 2024

View reviewed changes

williammartin requested changes Oct 18, 2024

View reviewed changes

jtmcg requested a review from williammartin October 18, 2024 15:53

williammartin approved these changes Oct 18, 2024

View reviewed changes

acceptance/acceptance_test.go Outdated Show resolved Hide resolved

jtmcg added 14 commits October 18, 2024 09:31

Added test function for repos and repo-create test

1842867

Added acceptance test for repo-delete

83cbdc4

Add acceptance test for repo-clone

60e20ee

Acceptance testing for repo-archive and repo-unarchive

0c632d7

Add acceptance tests for repo-list and repo-rename

fb6f773

Add acceptance test for repo-edit

c572edb

Add acceptance test for repo-set-default

7508131

Add acceptance tests for repo-fork and repo-sync

96fa853

Add acceptance tests for repo deploy-key add/list/delete

95ed11d

Cleanup some inconsistencies and improve collapse some functionality

e7549c9

Remove unnecessary gh auth setup-git steps

a5a7753

Wrap boolean strings in '' so it is clear they are strings

ee0d9ab

Replace LICENSE

25f2956

Makefile README.md acceptance api bin build cmd context docs git go.mod go.sum internal pkg script share test utils commands with

Remove unnecessary flags from repo-delete testscript

6c3ad50

jtmcg force-pushed the jtmcg/testscripts-repo branch from 6604f06 to 8ecab0d Compare October 18, 2024 16:32

jtmcg added 2 commits October 18, 2024 09:37

Rename acceptance test directory from repos to repo

3f4a2ae

Wrap true in '' in repo-fork-sync

ba324d8

jtmcg force-pushed the jtmcg/testscripts-repo branch from 7c64b18 to ba324d8 Compare October 18, 2024 16:45

jtmcg merged commit 92ff87c into trunk Oct 18, 2024

jtmcg deleted the jtmcg/testscripts-repo branch October 18, 2024 16:55

jtmcg mentioned this pull request Oct 25, 2024

Add comprehensive testscript for gh ruleset #9815

Merged

Uh oh!

Add acceptance tests for repo commands #9783

Add acceptance tests for repo commands #9783

Uh oh!

Conversation

jtmcg commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Question/Concern

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jtmcg Oct 17, 2024

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

jtmcg Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

williammartin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williammartin Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

jtmcg Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

williammartin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jtmcg commented Oct 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add acceptance tests for `repo` commands #9783

Add acceptance tests for `repo` commands #9783

jtmcg commented Oct 17, 2024 •

edited

Loading