Add acceptance tests for `secret` commands #9782

andyfeller · 2024-10-17T14:53:55Z

This PR expands upon the GitHub CLI acceptance tests to cover gh secret commands for 1) repository, 2) repository environment, and 3) organization secrets.

set -o pipefail && GH_ACCEPTANCE_TOKEN=$(gh auth token) GH_ACCEPTANCE_HOST=github.com GH_ACCEPTANCE_ORG=tinyfists go test -tags acceptance -json -run ^TestSecrets$ github.com/cli/cli/v2/acceptance  | tparse --all go test

resulting in:

┌─────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │            TEST             │             PACKAGE               │
│─────────┼─────────┼─────────────────────────────┼───────────────────────────────────│
│  PASS   │    4.82 │ TestSecrets/secret-repo-env │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    4.39 │ TestSecrets/secret-repo     │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    1.80 │ TestSecrets/secret-org      │ github.com/cli/cli/v2/acceptance  │
│  PASS   │    0.00 │ TestSecrets                 │ github.com/cli/cli/v2/acceptance  │
└─────────────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │             PACKAGE              │ COVER │ PASS │ FAIL │ SKIP  │
│─────────┼─────────┼──────────────────────────────────┼───────┼──────┼──────┼───────│
│  PASS   │  5.35s  │ github.com/cli/cli/v2/acceptance │  --   │  4   │  0   │  0    │
└────────────────────────────────────────────────────────────────────────────────────┘

Additional notes

GitHub Actions uppercases all secret names
Organization secrets require admin:org scope on token
Repository environments are only available to paid GitHub plans, require running secret-repo-env against a different organization than other testscripts

These are initial testscripts for confidence around core `gh secret` subcommands around 1) repository, 2) repository environment, 3) organization secrets. Each testscript exercises the 3 core subcommands of `gh secret`: 1) setting, 2) listing, and 3) deleting each type of secret. Note should be made that repository environments do not exist for free accounts such as `gh-acceptance-testing`.

Environments are only supported for GitHub Actions with certain plans; `gh-acceptance-testing` at this time does not have this support.

This commit refactors 2 new custom commands to work more like the native env command, allowing for multiple environment variables to be set. Additionally, the documentation on acceptance tests have been expanded to list out custom commands.

cliAutomation · 2024-10-17T14:54:07Z

Hi! Thanks for the pull request. Please ensure that this change is linked to an issue by mentioning an issue number in the description of the pull request. If this pull request would close the issue, please put the word 'Fixes' before the issue number somewhere in the pull request body. If this is a tiny change like fixing a typo, feel free to ignore this message.

williammartin

Firstly, please don't take my many comments as a criticism, there are only 1 or possibly 2 requested changes here.

Really, the reason there are so many comments is because you've managed to pick a command with a lot of interesting quirks and because you've brought a different perspective and taken some things in a different direction than the scripts @BagToad or I wrote so far. We just need to work through these to make sure we end up with a consistent approach to tests, even if that means we go back to adjust the already written ones.

Thanks!

acceptance/testdata/secret/secret-org.txtar

acceptance/testdata/secret/secret-repo-env.txtar

williammartin · 2024-10-17T19:00:36Z

acceptance/testdata/secret/secret-repo-env.txtar

+exec gh repo clone $ORG/$SCRIPT_NAME-$RANDOM_STRING
+cd $SCRIPT_NAME-$RANDOM_STRING
+
+# Create a repository environment, will fail if organization does not have environment support


Question

Do you think this test should run at all if the org doesn't have environment support? We could cond the whole test script so that it skips if there isn't support.

That is a really good question! We could skip this test, but I somewhat would prefer we have an environment where we don't have to worry about that and ensuring that core commands work as expected.

Yeh so Kynan suggested we just make this work for our org, and that seems like an obvious first step. I'm not sure how that works on ghe.com, need to investigate. Beyond that I can imagine that if anyone else were to run these tests we might want to add cond support.

This is not something I would prioritise until other people running them was an outcome we wanted to achieve, just looking for your general opinions.

Unfortunately I think the only way this could happen is attempting to pull plan information from the API about the organization or user account being targeted from

Get an organization

Get the authenticated user

acceptance/README.md

acceptance/testdata/secret/secret-org.txtar

acceptance/README.md

This is a bit of a refactor based on the work done in `gh workflow` as a better approach to verify secrets created are what we expect. Changes made: 1. Removed `env2lower` as it wasn't being used in testscripts 2. Added `replace` custom command to deal with testing organization workflow secrets 3. Refactored secret testscripts to create and run workflow that tests the value of the secret provided 4. Minor reordering of test `acceptance` test functions as appending to the end is confusing and adds conflicts 5. Removed stdout TTY assertions

acceptance/testdata/secret/secret-org.txtar

williammartin

Pretty much there but one thing I do think needs to be resolved in the env var expansion for secret-org.

acceptance/testdata/secret/secret-org.txtar

acceptance/acceptance_test.go

acceptance/testdata/secret/secret-repo.txtar

williammartin · 2024-10-21T14:03:30Z

acceptance/acceptance_test.go

 	}))
 }

-func TestPullRequests(t *testing.T) {


It's a shame about this diff because it's going to make it kinda janky to track back the git history. It's not the end of the world though. Maybe once we're happy with this PR we can squash it all down and only have the addition of TestSecrets in this part.

Aside from being able to skim the list of tests, the conflict with these tests being added at the end of the ever growing list felt like it could benefit from being alphabetized 🤷

Ah, from the diff it just looked like things had been bounced around by accident. Sure, is fine with me.

For now, I'm just going to undo my reorganizing changes around the test functions. I don't like it but want to get this done despite value in being able to scan the test functions alphabetically.

That's fine. I would accept a follow up PR to move them around given your reasoning. Makes sense to me.

williammartin

LGTM once the conflict is resolved. We may want to follow up with some kind of conditional behaviour for the extra org permissions but these are not currently auto-running anywhere so we have the knowledge of what should and shouldn't work.

Thanks a lot for this, I know it wasn't the smoothest sailing.

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.59.0` -> `v2.62.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.62.0`](https://github.com/cli/cli/releases/tag/v2.62.0): GitHub CLI 2.62.0 [Compare Source](cli/cli@v2.61.0...v2.62.0) #### What's Changed - Update monotonic verification logic and testing by [@malancas](https://github.com/malancas) in cli/cli#9856 - Check extension for latest version when executed by [@andyfeller](https://github.com/andyfeller) in cli/cli#9866 - Shorten extension release checking from 3s to 1s by [@andyfeller](https://github.com/andyfeller) in cli/cli#9914 - Mention GitHub CLI team on discussion issues by [@andyfeller](https://github.com/andyfeller) in cli/cli#9920 **Full Changelog**: cli/cli@v2.61.0...v2.62.0 #### Security - A security vulnerability has been identified in GitHub CLI that could allow remote code execution (RCE) when users connect to a malicious Codespace SSH server and use the `gh codespace ssh` or `gh codespace logs` commands. For more information, see GHSA-p2h2-3vg9-4p87 #### GitHub CLI notifies users about latest extension upgrades Similar to the notification of latest `gh` releases, the `v2.62.0` version of GitHub CLI will notify users about latest extension upgrades when the extension is used: ```shell $ gh ado2gh ... A new release of ado2gh is available: 1.7.0 → 1.8.0 To upgrade, run: gh extension upgrade ado2gh --force https://github.com/github/gh-ado2gh ``` ##### Why does this matter? This removes a common pain point of extension authors as they have had to reverse engineer and implement a similar mechanism within their extensions directly. With this quality of life improvement, there are 2 big benefits: 1. Extension authors will hopefully see increased adoption of newer releases while having lower bar to maintaining their extensions. 2. GitHub CLI users will have greater awareness of new features, bug fixes, and security fixes to the extensions used. ##### What do you need to do? Extension authors should review their extensions and consider removing any custom logic previously implemented to notify users of new releases. ### [`v2.61.0`](https://github.com/cli/cli/releases/tag/v2.61.0): GitHub CLI 2.61.0 [Compare Source](cli/cli@v2.60.1...v2.61.0) #### Ensure users understand consequences before making repository visibility changes In `v2.61.0`, `gh repo edit` command has been enhanced to inform users about [consequences of changing visibility](https://gh.io/setting-repository-visibility) and ensure users are intentional before making irreversible changes: 1. Interactive `gh repo edit` visibility change requires confirmation when changing from `public`, `private`, or `internal` 2. Non-interactive `gh repo edit --visibility` change requires new `--accept-visibility-change-consequences` flag to confirm 3. New content to inform users of consequences - Incorporate [GitHub Docs content](https://gh.io/setting-repository-visibility) into help usage and interactive `gh repo edit` experience - Expanded help usage to call out most concerning consequences - Display repository star and watcher counts to understand impact before confirming #### What's Changed - Add acceptance test for `project` command by [@jtmcg](https://github.com/jtmcg) in cli/cli#9816 - Add comprehensive testscript for `gh ruleset` by [@andyfeller](https://github.com/andyfeller) in cli/cli#9815 - Add comprehensive testscript for gh ext commandset by [@andyfeller](https://github.com/andyfeller) in cli/cli#9810 - Require visibility confirmation in `gh repo edit` by [@andyfeller](https://github.com/andyfeller) in cli/cli#9845 - Clean up skipped online tests for `gh attestation verify` by [@malancas](https://github.com/malancas) in cli/cli#9838 - `gh attestation verify` should only verify provenance attestations by default by [@malancas](https://github.com/malancas) in cli/cli#9825 - Set `dnf5` commands as default by [@its-miroma](https://github.com/its-miroma) in cli/cli#9844 - Fix verbiage for deleting workflow runs by [@akx](https://github.com/akx) in cli/cli#9876 - Bump github.com/creack/pty from 1.1.23 to 1.1.24 by [@dependabot](https://github.com/dependabot) in cli/cli#9862 - `gh attestation verify` policy enforcement refactor by [@malancas](https://github.com/malancas) in cli/cli#9848 - Simplify Sigstore verification result handling in `gh attestation verify` by [@malancas](https://github.com/malancas) in cli/cli#9877 - Print empty array for `gh cache list` when `--json` is provided by [@williammartin](https://github.com/williammartin) in cli/cli#9883 - Bump actions/attest-build-provenance from 1.4.3 to 1.4.4 by [@dependabot](https://github.com/dependabot) in cli/cli#9884 - Create the automatic key when specified with -i by [@cmbrose](https://github.com/cmbrose) in cli/cli#9881 - fix: `gh pr create -w` ignore template flag by [@nilvng](https://github.com/nilvng) in cli/cli#9863 #### New Contributors - [@akx](https://github.com/akx) made their first contribution in cli/cli#9876 - [@nilvng](https://github.com/nilvng) made their first contribution in cli/cli#9863 **Full Changelog**: cli/cli@v2.60.1...v2.61.0 ### [`v2.60.1`](https://github.com/cli/cli/releases/tag/v2.60.1): GitHub CLI 2.60.1 [Compare Source](cli/cli@v2.60.0...v2.60.1) This is a small patch release to fix installing `gh` via `go install` which was broken with v2.60.0. #### What's Changed - Update testscript to use hard fork by [@williammartin](https://github.com/williammartin) in cli/cli#9821 **Full Changelog**: cli/cli@v2.60.0...v2.60.1 ### [`v2.60.0`](https://github.com/cli/cli/releases/tag/v2.60.0): GitHub CLI 2.60.0 [Compare Source](cli/cli@v2.59.0...v2.60.0) #### What's Changed - Add ArchivedAt field by [@tsukasaI](https://github.com/tsukasaI) in cli/cli#9790 - Include startedAt, completedAt in run steps data by [@andyfeller](https://github.com/andyfeller) in cli/cli#9774 - Adjust environment help for host and tokens by [@williammartin](https://github.com/williammartin) in cli/cli#9809 - Add handling of empty titles for Issues and MRs by [@jtmcg](https://github.com/jtmcg) in cli/cli#9701 - `LiveSigstoreVerifier.Verify` should error if no attestations are present by [@phillmv](https://github.com/phillmv) in cli/cli#9742 - `gh at verify` retries fetching attestations if it receives a 5xx by [@phillmv](https://github.com/phillmv) in cli/cli#9797 - Prevent local extension installations with invalid names and conflicts with core commands and other extensions by [@BagToad](https://github.com/BagToad) in cli/cli#9794 - Rewrite a sentence in CONTRIBUTING.md by [@muzimuzhi](https://github.com/muzimuzhi) in cli/cli#9772 - Use new GitHub preview terms in `working-with-us.md` by [@BagToad](https://github.com/BagToad) in cli/cli#9800 - Use new GitHub previews terminology in attestation commands' help docs by [@BagToad](https://github.com/BagToad) in cli/cli#9799 - Clarify in README that `gh` is supported on GitHub Enterprise Cloud by [@BagToad](https://github.com/BagToad) in cli/cli#9805 - build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.5 to 1.4.6 by [@dependabot](https://github.com/dependabot) in cli/cli#9752 ##### Acceptance Test Changes - Add acceptance tests for `workflow`, `run`, and `cache` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9766 - Add basic `api` acceptance tests by [@BagToad](https://github.com/BagToad) in cli/cli#9770 - Add acceptance tests for `release` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9771 - Add acceptance tests for `org` and `ssh-key` commands by [@BagToad](https://github.com/BagToad) in cli/cli#9812 - Add acceptance tests for `gh auth` commands by [@jtmcg](https://github.com/jtmcg) in cli/cli#9787 - Add acceptance tests for `repo` commands by [@jtmcg](https://github.com/jtmcg) in cli/cli#9783 - Add acceptance tests for `search` command by [@BagToad](https://github.com/BagToad) in cli/cli#9786 - Add acceptance tests for `variable` commands by [@andyfeller](https://github.com/andyfeller) in cli/cli#978 - Add testscripts for gpg-key and label commands by [@williammartin](https://github.com/williammartin) in cli/cli#9811 - Use forked testscript for token redaction by [@williammartin](https://github.com/williammartin) in cli/cli#9804 - Add acceptance tests for `secret` commands by [@andyfeller](https://github.com/andyfeller) in cli/cli#9782 - Note token redaction in Acceptance test README by [@williammartin](https://github.com/williammartin) in cli/cli#9813 #### New Contributors - [@tsukasaI](https://github.com/tsukasaI) made their first contribution in cli/cli#9790 **Full Changelog**: cli/cli@v2.59.0...v2.60.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

andyfeller added 3 commits October 17, 2024 10:09

Update secret note about potential failure

5bbe69b

Environments are only supported for GitHub Actions with certain plans; `gh-acceptance-testing` at this time does not have this support.

Refactor env2upper, env2lower; add docs

7c534e6

This commit refactors 2 new custom commands to work more like the native env command, allowing for multiple environment variables to be set. Additionally, the documentation on acceptance tests have been expanded to list out custom commands.

andyfeller requested a review from a team as a code owner October 17, 2024 14:53

andyfeller requested a review from jtmcg October 17, 2024 14:53

cliAutomation added the external pull request originating outside of the CLI core team label Oct 17, 2024

andyfeller changed the title ~~Andyfeller/testscripts secret~~ Add acceptance tests for secret commands Oct 17, 2024

Fix typo in custom command doc

e91052d

williammartin requested changes Oct 17, 2024

View reviewed changes

Minor polish / consistency

e335113

andyfeller mentioned this pull request Oct 17, 2024

Add acceptance tests for variable commands #9784

Merged

jtmcg mentioned this pull request Oct 17, 2024

Add acceptance tests for repo commands #9783

Merged

andyfeller added 2 commits October 18, 2024 09:46

Merge branch 'trunk' into andyfeller/testscripts-secret

1571a11

andyfeller commented Oct 18, 2024

View reviewed changes

acceptance/testdata/secret/secret-org.txtar Show resolved Hide resolved

williammartin requested changes Oct 21, 2024

View reviewed changes

Address @williammartin PR feedback

acf62da

williammartin approved these changes Oct 21, 2024

View reviewed changes

Merge branch 'trunk' into andyfeller/testscripts-secret

b3c0c39

andyfeller merged commit 1886d1b into trunk Oct 21, 2024

andyfeller deleted the andyfeller/testscripts-secret branch October 21, 2024 19:33

This comment was marked as spam.

Sign in to view

Uh oh!

Add acceptance tests for secret commands #9782

Add acceptance tests for secret commands #9782

Uh oh!

Conversation

andyfeller commented Oct 17, 2024

Additional notes

Uh oh!

cliAutomation commented Oct 17, 2024

Uh oh!

williammartin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williammartin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

williammartin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Add acceptance tests for `secret` commands #9782

Add acceptance tests for `secret` commands #9782

williammartin left a comment •

edited

Loading