Add comprehensive testscript for gh ruleset
#9815
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi! Thanks for the pull request. Please ensure that this change is linked to an issue by mentioning an issue number in the description of the pull request. If this pull request would close the issue, please put the word 'Fixes' before the issue number somewhere in the pull request body. If this is a tiny change like fixing a typo, feel free to ignore this message. |
jtmcg
reviewed
Oct 24, 2024
jtmcg
reviewed
Oct 24, 2024
| @@ -0,0 +1,62 @@ | |||
| # Setup environment variables used for testscript | |||
There was a problem hiding this comment.
I think we should follow the same convention for this file's name as we have for the other files: ruleset-list-check
There was a problem hiding this comment.
I'm open to discuss as all of the other major primitive testscripts were more thoroughly designed into discrete testscripts for specific capabilities.
This testscript is an encompasses testing various behaviors of the whole commandset; there is no single focus. Ideally, we'd have actual separate testscripts that are specific to subcommands.
But let's go look at the other testscripts:
Fine-grain scenario testscripts
issue: https://github.com/cli/cli/tree/trunk/acceptance/testdata/issuepr: https://github.com/cli/cli/tree/trunk/acceptance/testdata/prrelease: https://github.com/cli/cli/tree/trunk/acceptance/testdata/releaserepo: https://github.com/cli/cli/tree/trunk/acceptance/testdata/reposecret: https://github.com/cli/cli/tree/trunk/acceptance/testdata/secretvariable: https://github.com/cli/cli/tree/trunk/acceptance/testdata/variableworkflow,run: https://github.com/cli/cli/tree/trunk/acceptance/testdata/workflow
Coarse-grain comprehensive testscripts
gpg-key: https://github.com/cli/cli/blob/trunk/acceptance/testdata/gpg-key/gpg-key.txtarlabel: https://github.com/cli/cli/blob/trunk/acceptance/testdata/label/label.txtarssh-key: https://github.com/cli/cli/blob/trunk/acceptance/testdata/ssh-key/ssh-key.txtar
Being more coarse grained, I should rename this to extension.txtar being it isn't a collection of fine-grained testscripts.
There was a problem hiding this comment.
🤔 I'm not sure I agree with these course-grain names...
I would call gpg-key and ssh-key "comprehensive" testscripts, as they contains all the subcommands in the gpg-key and ssh-key command trees (add, delete, list)
I wouldn't call label or ruleset comprehensive, though:
- label is missing
cloneanddelete - ruleset (this PR) is missing
view
Either way, I had to click into each of these and scan the code to verify which subcommands were covered. To me, it's tidier to be explicit and consistent, regardless of whether or not we're comprehensively covered.
There is the marginal benefit, too, that if we add another subcommand to any of these command sets we'll probably do this renaming again later
There was a problem hiding this comment.
I guess I don't think we should make the distinction between "course-grained" and "fine-grained" test scripts. They're just test scripts.
There was a problem hiding this comment.
To close this out, @andyfeller and I discussed on Zoom and agree its fine to leave as-is for now
jtmcg
reviewed
Oct 24, 2024
williammartin
approved these changes
Oct 28, 2024
There was a problem hiding this comment.
Nice work on the API request.
Just noting that this doesn't work out of the box for me on dotcom, erroring with:
Upgrade to GitHub Pro or make this repository public to enable this feature.
But I'm content that this has tackled the primary purpose of the test, thanks!
|
|
||
| # Verify repository ruleset associated with branch | ||
| exec gh ruleset check | ||
| stdout '- pull_request:.+dismiss_stale_reviews_on_push: false.+require_code_owner_review: true.+require_last_push_approval: false.+required_approving_review_count: 1.+required_review_thread_resolution: false' |
There was a problem hiding this comment.
Wow the ruleset check output leaves a bit to be desired 😅
There was a problem hiding this comment.
Yeah ... wasn't enthused about that ... also the whole GitHub plan piece you mentioned above.
Similar to `gpg-key`, `label`, `ssh-key`, this coarse grained testscript should be named after the commandset given it isn't a collection of targeted scenarios.
eead88a to
6dc86e5
Compare
|
@andyfeller I rebased this on |
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Nov 21, 2024
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.59.0` -> `v2.62.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.62.0`](https://github.com/cli/cli/releases/tag/v2.62.0): GitHub CLI 2.62.0 [Compare Source](cli/cli@v2.61.0...v2.62.0) #### What's Changed - Update monotonic verification logic and testing by [@​malancas](https://github.com/malancas) in cli/cli#9856 - Check extension for latest version when executed by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9866 - Shorten extension release checking from 3s to 1s by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9914 - Mention GitHub CLI team on discussion issues by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9920 **Full Changelog**: cli/cli@v2.61.0...v2.62.0 #### Security - A security vulnerability has been identified in GitHub CLI that could allow remote code execution (RCE) when users connect to a malicious Codespace SSH server and use the `gh codespace ssh` or `gh codespace logs` commands. For more information, see GHSA-p2h2-3vg9-4p87 #### GitHub CLI notifies users about latest extension upgrades Similar to the notification of latest `gh` releases, the `v2.62.0` version of GitHub CLI will notify users about latest extension upgrades when the extension is used: ```shell $ gh ado2gh ... A new release of ado2gh is available: 1.7.0 → 1.8.0 To upgrade, run: gh extension upgrade ado2gh --force https://github.com/github/gh-ado2gh ``` ##### Why does this matter? This removes a common pain point of extension authors as they have had to reverse engineer and implement a similar mechanism within their extensions directly. With this quality of life improvement, there are 2 big benefits: 1. Extension authors will hopefully see increased adoption of newer releases while having lower bar to maintaining their extensions. 2. GitHub CLI users will have greater awareness of new features, bug fixes, and security fixes to the extensions used. ##### What do you need to do? Extension authors should review their extensions and consider removing any custom logic previously implemented to notify users of new releases. ### [`v2.61.0`](https://github.com/cli/cli/releases/tag/v2.61.0): GitHub CLI 2.61.0 [Compare Source](cli/cli@v2.60.1...v2.61.0) #### Ensure users understand consequences before making repository visibility changes In `v2.61.0`, `gh repo edit` command has been enhanced to inform users about [consequences of changing visibility](https://gh.io/setting-repository-visibility) and ensure users are intentional before making irreversible changes: 1. Interactive `gh repo edit` visibility change requires confirmation when changing from `public`, `private`, or `internal` 2. Non-interactive `gh repo edit --visibility` change requires new `--accept-visibility-change-consequences` flag to confirm 3. New content to inform users of consequences - Incorporate [GitHub Docs content](https://gh.io/setting-repository-visibility) into help usage and interactive `gh repo edit` experience - Expanded help usage to call out most concerning consequences - Display repository star and watcher counts to understand impact before confirming #### What's Changed - Add acceptance test for `project` command by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9816 - Add comprehensive testscript for `gh ruleset` by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9815 - Add comprehensive testscript for gh ext commandset by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9810 - Require visibility confirmation in `gh repo edit` by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9845 - Clean up skipped online tests for `gh attestation verify` by [@​malancas](https://github.com/malancas) in cli/cli#9838 - `gh attestation verify` should only verify provenance attestations by default by [@​malancas](https://github.com/malancas) in cli/cli#9825 - Set `dnf5` commands as default by [@​its-miroma](https://github.com/its-miroma) in cli/cli#9844 - Fix verbiage for deleting workflow runs by [@​akx](https://github.com/akx) in cli/cli#9876 - Bump github.com/creack/pty from 1.1.23 to 1.1.24 by [@​dependabot](https://github.com/dependabot) in cli/cli#9862 - `gh attestation verify` policy enforcement refactor by [@​malancas](https://github.com/malancas) in cli/cli#9848 - Simplify Sigstore verification result handling in `gh attestation verify` by [@​malancas](https://github.com/malancas) in cli/cli#9877 - Print empty array for `gh cache list` when `--json` is provided by [@​williammartin](https://github.com/williammartin) in cli/cli#9883 - Bump actions/attest-build-provenance from 1.4.3 to 1.4.4 by [@​dependabot](https://github.com/dependabot) in cli/cli#9884 - Create the automatic key when specified with -i by [@​cmbrose](https://github.com/cmbrose) in cli/cli#9881 - fix: `gh pr create -w` ignore template flag by [@​nilvng](https://github.com/nilvng) in cli/cli#9863 #### New Contributors - [@​akx](https://github.com/akx) made their first contribution in cli/cli#9876 - [@​nilvng](https://github.com/nilvng) made their first contribution in cli/cli#9863 **Full Changelog**: cli/cli@v2.60.1...v2.61.0 ### [`v2.60.1`](https://github.com/cli/cli/releases/tag/v2.60.1): GitHub CLI 2.60.1 [Compare Source](cli/cli@v2.60.0...v2.60.1) This is a small patch release to fix installing `gh` via `go install` which was broken with v2.60.0. #### What's Changed - Update testscript to use hard fork by [@​williammartin](https://github.com/williammartin) in cli/cli#9821 **Full Changelog**: cli/cli@v2.60.0...v2.60.1 ### [`v2.60.0`](https://github.com/cli/cli/releases/tag/v2.60.0): GitHub CLI 2.60.0 [Compare Source](cli/cli@v2.59.0...v2.60.0) #### What's Changed - Add ArchivedAt field by [@​tsukasaI](https://github.com/tsukasaI) in cli/cli#9790 - Include startedAt, completedAt in run steps data by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9774 - Adjust environment help for host and tokens by [@​williammartin](https://github.com/williammartin) in cli/cli#9809 - Add handling of empty titles for Issues and MRs by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9701 - `LiveSigstoreVerifier.Verify` should error if no attestations are present by [@​phillmv](https://github.com/phillmv) in cli/cli#9742 - `gh at verify` retries fetching attestations if it receives a 5xx by [@​phillmv](https://github.com/phillmv) in cli/cli#9797 - Prevent local extension installations with invalid names and conflicts with core commands and other extensions by [@​BagToad](https://github.com/BagToad) in cli/cli#9794 - Rewrite a sentence in CONTRIBUTING.md by [@​muzimuzhi](https://github.com/muzimuzhi) in cli/cli#9772 - Use new GitHub preview terms in `working-with-us.md` by [@​BagToad](https://github.com/BagToad) in cli/cli#9800 - Use new GitHub previews terminology in attestation commands' help docs by [@​BagToad](https://github.com/BagToad) in cli/cli#9799 - Clarify in README that `gh` is supported on GitHub Enterprise Cloud by [@​BagToad](https://github.com/BagToad) in cli/cli#9805 - build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.5 to 1.4.6 by [@​dependabot](https://github.com/dependabot) in cli/cli#9752 ##### Acceptance Test Changes - Add acceptance tests for `workflow`, `run`, and `cache` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9766 - Add basic `api` acceptance tests by [@​BagToad](https://github.com/BagToad) in cli/cli#9770 - Add acceptance tests for `release` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9771 - Add acceptance tests for `org` and `ssh-key` commands by [@​BagToad](https://github.com/BagToad) in cli/cli#9812 - Add acceptance tests for `gh auth` commands by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9787 - Add acceptance tests for `repo` commands by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9783 - Add acceptance tests for `search` command by [@​BagToad](https://github.com/BagToad) in cli/cli#9786 - Add acceptance tests for `variable` commands by [@​andyfeller](https://github.com/andyfeller) in cli/cli#978 - Add testscripts for gpg-key and label commands by [@​williammartin](https://github.com/williammartin) in cli/cli#9811 - Use forked testscript for token redaction by [@​williammartin](https://github.com/williammartin) in cli/cli#9804 - Add acceptance tests for `secret` commands by [@​andyfeller](https://github.com/andyfeller) in cli/cli#9782 - Note token redaction in Acceptance test README by [@​williammartin](https://github.com/williammartin) in cli/cli#9813 #### New Contributors - [@​tsukasaI](https://github.com/tsukasaI) made their first contribution in cli/cli#9790 **Full Changelog**: cli/cli@v2.59.0...v2.60.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a comprehensive testscript around
gh rulesetcommandset.Due to the complexities of the create repository ruleset endpoint fields, this needed to build a JSON file modeling the input values in order to test the subcommands.
resulting in: