Add acceptance tests for release commands
#9771
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi! Thanks for the pull request. Please ensure that this change is linked to an issue by mentioning an issue number in the description of the pull request. If this pull request would close the issue, please put the word 'Fixes' before the issue number somewhere in the pull request body. If this is a tiny change like fixing a typo, feel free to ignore this message. |
jtmcg
reviewed
Oct 16, 2024
|
|
||
| # Upload an asset to the release | ||
| exec gh release upload v1.2.3 ../asset.txt | ||
| stdout '' |
There was a problem hiding this comment.
🤔 have you considered checking stderr instead of stdout, here? I took a quick look at uploadRun() and it seems the only time that stdout won't be empty is for the TTY use case. Thus, I think if this errors, this test may still pass.
There was a problem hiding this comment.
Or does exec gh release upload v1.2.3 ../asset.txt catch errors for us already? If so, do we need this check?
There was a problem hiding this comment.
Ah, exec definitely is an assertion!
This exec statement, though, is more than just a way of running programs; it’s also an assertion. The program we execute must succeed: that is, its exit status must be zero. Otherwise, the exec assertion fails.
There was a problem hiding this comment.
So what is the stdout '' checking for?
There was a problem hiding this comment.
@BagToad and I discussed these and agreed to remove them because they were asserting a bit more than was necessary.
There was a problem hiding this comment.
Ah, cool. I thought I might be missing something 👍
ac90b42 to
c8c1523
Compare
c8c1523 to
24fc2db
Compare
williammartin
approved these changes
Oct 17, 2024
There was a problem hiding this comment.
LGTM thanks
➜ cli git:(kw/github-cli-586-add-acceptance-tests-for-release-commands) set -o pipefail && GH_ACCEPTANCE_HOST=github.com GH_ACCEPTANCE_ORG=gh-acceptance-testing go test -tags acceptance -json -coverprofile=coverage.out -coverpkg=./... -run ^TestReleases$ github.com/cli/cli/v2/acceptance | tparse --all go test
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ STATUS │ ELAPSED │ TEST │ PACKAGE │
│─────────┼─────────┼──────────────────────────────────────┼───────────────────────────────────│
│ PASS │ 7.38 │ TestReleases/release-upload-download │ github.com/cli/cli/v2/acceptance │
│ PASS │ 5.39 │ TestReleases/release-list │ github.com/cli/cli/v2/acceptance │
│ PASS │ 5.07 │ TestReleases/release-create │ github.com/cli/cli/v2/acceptance │
│ PASS │ 5.07 │ TestReleases/release-view │ github.com/cli/cli/v2/acceptance │
│ PASS │ 0.00 │ TestReleases │ github.com/cli/cli/v2/acceptance │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────────────────────┐
│ STATUS │ ELAPSED │ PACKAGE │ COVER │ PASS │ FAIL │ SKIP │
│─────────┼─────────┼──────────────────────────────────┼───────┼──────┼──────┼───────│
│ PASS │ 8.34s │ github.com/cli/cli/v2/acceptance │ 11.0% │ 5 │ 0 │ 0 │
└────────────────────────────────────────────────────────────────────────────────────┘
| exec gh release create v1.2.3 --notes 'awesome release' --latest | ||
|
|
||
| # Upload an asset to the release | ||
| exec gh release upload v1.2.3 ../asset.txt |
There was a problem hiding this comment.
This currently fails when targeting ghe.com. It is being tracked by the owning API team.
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Nov 21, 2024
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.59.0` -> `v2.62.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.62.0`](https://github.com/cli/cli/releases/tag/v2.62.0): GitHub CLI 2.62.0 [Compare Source](cli/cli@v2.61.0...v2.62.0) #### What's Changed - Update monotonic verification logic and testing by [@​malancas](https://github.com/malancas) in cli/cli#9856 - Check extension for latest version when executed by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9866 - Shorten extension release checking from 3s to 1s by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9914 - Mention GitHub CLI team on discussion issues by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9920 **Full Changelog**: cli/cli@v2.61.0...v2.62.0 #### Security - A security vulnerability has been identified in GitHub CLI that could allow remote code execution (RCE) when users connect to a malicious Codespace SSH server and use the `gh codespace ssh` or `gh codespace logs` commands. For more information, see GHSA-p2h2-3vg9-4p87 #### GitHub CLI notifies users about latest extension upgrades Similar to the notification of latest `gh` releases, the `v2.62.0` version of GitHub CLI will notify users about latest extension upgrades when the extension is used: ```shell $ gh ado2gh ... A new release of ado2gh is available: 1.7.0 → 1.8.0 To upgrade, run: gh extension upgrade ado2gh --force https://github.com/github/gh-ado2gh ``` ##### Why does this matter? This removes a common pain point of extension authors as they have had to reverse engineer and implement a similar mechanism within their extensions directly. With this quality of life improvement, there are 2 big benefits: 1. Extension authors will hopefully see increased adoption of newer releases while having lower bar to maintaining their extensions. 2. GitHub CLI users will have greater awareness of new features, bug fixes, and security fixes to the extensions used. ##### What do you need to do? Extension authors should review their extensions and consider removing any custom logic previously implemented to notify users of new releases. ### [`v2.61.0`](https://github.com/cli/cli/releases/tag/v2.61.0): GitHub CLI 2.61.0 [Compare Source](cli/cli@v2.60.1...v2.61.0) #### Ensure users understand consequences before making repository visibility changes In `v2.61.0`, `gh repo edit` command has been enhanced to inform users about [consequences of changing visibility](https://gh.io/setting-repository-visibility) and ensure users are intentional before making irreversible changes: 1. Interactive `gh repo edit` visibility change requires confirmation when changing from `public`, `private`, or `internal` 2. Non-interactive `gh repo edit --visibility` change requires new `--accept-visibility-change-consequences` flag to confirm 3. New content to inform users of consequences - Incorporate [GitHub Docs content](https://gh.io/setting-repository-visibility) into help usage and interactive `gh repo edit` experience - Expanded help usage to call out most concerning consequences - Display repository star and watcher counts to understand impact before confirming #### What's Changed - Add acceptance test for `project` command by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9816 - Add comprehensive testscript for `gh ruleset` by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9815 - Add comprehensive testscript for gh ext commandset by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9810 - Require visibility confirmation in `gh repo edit` by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9845 - Clean up skipped online tests for `gh attestation verify` by [@​malancas](https://github.com/malancas) in cli/cli#9838 - `gh attestation verify` should only verify provenance attestations by default by [@​malancas](https://github.com/malancas) in cli/cli#9825 - Set `dnf5` commands as default by [@​its-miroma](https://github.com/its-miroma) in cli/cli#9844 - Fix verbiage for deleting workflow runs by [@​akx](https://github.com/akx) in cli/cli#9876 - Bump github.com/creack/pty from 1.1.23 to 1.1.24 by [@​dependabot](https://github.com/dependabot) in cli/cli#9862 - `gh attestation verify` policy enforcement refactor by [@​malancas](https://github.com/malancas) in cli/cli#9848 - Simplify Sigstore verification result handling in `gh attestation verify` by [@​malancas](https://github.com/malancas) in cli/cli#9877 - Print empty array for `gh cache list` when `--json` is provided by [@​williammartin](https://github.com/williammartin) in cli/cli#9883 - Bump actions/attest-build-provenance from 1.4.3 to 1.4.4 by [@​dependabot](https://github.com/dependabot) in cli/cli#9884 - Create the automatic key when specified with -i by [@​cmbrose](https://github.com/cmbrose) in cli/cli#9881 - fix: `gh pr create -w` ignore template flag by [@​nilvng](https://github.com/nilvng) in cli/cli#9863 #### New Contributors - [@​akx](https://github.com/akx) made their first contribution in cli/cli#9876 - [@​nilvng](https://github.com/nilvng) made their first contribution in cli/cli#9863 **Full Changelog**: cli/cli@v2.60.1...v2.61.0 ### [`v2.60.1`](https://github.com/cli/cli/releases/tag/v2.60.1): GitHub CLI 2.60.1 [Compare Source](cli/cli@v2.60.0...v2.60.1) This is a small patch release to fix installing `gh` via `go install` which was broken with v2.60.0. #### What's Changed - Update testscript to use hard fork by [@​williammartin](https://github.com/williammartin) in cli/cli#9821 **Full Changelog**: cli/cli@v2.60.0...v2.60.1 ### [`v2.60.0`](https://github.com/cli/cli/releases/tag/v2.60.0): GitHub CLI 2.60.0 [Compare Source](cli/cli@v2.59.0...v2.60.0) #### What's Changed - Add ArchivedAt field by [@​tsukasaI](https://github.com/tsukasaI) in cli/cli#9790 - Include startedAt, completedAt in run steps data by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9774 - Adjust environment help for host and tokens by [@​williammartin](https://github.com/williammartin) in cli/cli#9809 - Add handling of empty titles for Issues and MRs by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9701 - `LiveSigstoreVerifier.Verify` should error if no attestations are present by [@​phillmv](https://github.com/phillmv) in cli/cli#9742 - `gh at verify` retries fetching attestations if it receives a 5xx by [@​phillmv](https://github.com/phillmv) in cli/cli#9797 - Prevent local extension installations with invalid names and conflicts with core commands and other extensions by [@​BagToad](https://github.com/BagToad) in cli/cli#9794 - Rewrite a sentence in CONTRIBUTING.md by [@​muzimuzhi](https://github.com/muzimuzhi) in cli/cli#9772 - Use new GitHub preview terms in `working-with-us.md` by [@​BagToad](https://github.com/BagToad) in cli/cli#9800 - Use new GitHub previews terminology in attestation commands' help docs by [@​BagToad](https://github.com/BagToad) in cli/cli#9799 - Clarify in README that `gh` is supported on GitHub Enterprise Cloud by [@​BagToad](https://github.com/BagToad) in cli/cli#9805 - build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.5 to 1.4.6 by [@​dependabot](https://github.com/dependabot) in cli/cli#9752 ##### Acceptance Test Changes - Add acceptance tests for `workflow`, `run`, and `cache` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9766 - Add basic `api` acceptance tests by [@​BagToad](https://github.com/BagToad) in cli/cli#9770 - Add acceptance tests for `release` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9771 - Add acceptance tests for `org` and `ssh-key` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9812 - Add acceptance tests for `gh auth` commands by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9787 - Add acceptance tests for `repo` commands by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9783 - Add acceptance tests for `search` command by [@​BagToad](https://github.com/BagToad) in cli/cli#9786 - Add acceptance tests for `variable` commands by [@​andyfeller](https://github.com/andyfeller) in cli/cli#978 - Add testscripts for gpg-key and label commands by [@​williammartin](https://github.com/williammartin) in cli/cli#9811 - Use forked testscript for token redaction by [@​williammartin](https://github.com/williammartin) in cli/cli#9804 - Add acceptance tests for `secret` commands by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9782 - Note token redaction in Acceptance test README by [@​williammartin](https://github.com/williammartin) in cli/cli#9813 #### New Contributors - [@​tsukasaI](https://github.com/tsukasaI) made their first contribution in cli/cli#9790 **Full Changelog**: cli/cli@v2.59.0...v2.60.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds basic acceptance tests for the
releasecommand:You can run these with: