Introduce testscript acceptance tests generally, and for the PR command specifically #9745
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0e1e364 to
f9b2499
Compare
Accuweaty24
added a commit
to Accuweaty24/cli
that referenced
this pull request
Oct 11, 2024
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. @probablycorey @matschaffer @digitalfu cli#9745 cli#9721 cli#9728 cli#9746 №accuweaty24 #
jtmcg
reviewed
Oct 11, 2024
|
|
||
| #### Scope Validation | ||
|
|
||
| TODO: I believe tests should early exit if the correct scopes aren't in place to execute the entire lifecycle. It's extremely annoying if a `defer` fails to clean up resources because there's no `delete_repo` scope for example. However, I'm not sure yet whether this scope checking should be in the Go tests or in the scripts themselves. It seems very cool to understand required scopes for a script just by looking at the script itself. |
williammartin
commented
Oct 14, 2024
| > [!WARNING] | ||
| > Verbose mode dumps the `testscript` environment variables, including the `GH_TOKEN`, so be careful. | ||
|
|
||
| By default `testscript` removes the directory in which it was running the script, and if you've been a conscientious engineer, you should be cleaning up resources using the `defer` statement. However, this can be an impediment to debugging. As such you can set `GH_ACCEPTANCE_PRESERVE_WORK_DIR=true` and `GH_ACCEPTANCE_SKIP_DEFER=true` to skip these cleanup steps. |
There was a problem hiding this comment.
I went back and forth a bit on having a single env var that could have different values but I decided until we get more usage and understand what the enumeration of values might be, it was better to be explicit.
williammartin
commented
Oct 14, 2024
| @@ -0,0 +1,25 @@ | |||
| # Use gh as a credential helper | |||
There was a problem hiding this comment.
I recognise that this is almost identical to pr-create-basic.txtar. I'm being quite intentional about keeping them separate. Just because they look the same now doesn't mean they will always look the same.
andyfeller
requested changes
Oct 14, 2024
Co-authored-by: Andy Feller <[email protected]>
Co-authored-by: Andy Feller <[email protected]>
Co-authored-by: Andy Feller <[email protected]>
andyfeller
approved these changes
Oct 14, 2024
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Oct 17, 2024
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.58.0` -> `v2.59.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.59.0`](https://github.com/cli/cli/releases/tag/v2.59.0): GitHub CLI 2.59.0 [Compare Source](cli/cli@v2.58.0...v2.59.0) #### What's Changed - Allow community submitted design work by [@​BagToad](https://github.com/BagToad) in cli/cli#9683 - Improve `SECURITY.md` with expectations for privately reported vulnerabilities by [@​BagToad](https://github.com/BagToad) in cli/cli#9687 - Emit a log message when extension installation falls back to a `darwin-amd64` binary on an Apple Silicon macOS device by [@​timrogers](https://github.com/timrogers) in cli/cli#9650 - Print the login URL even when opening a browser by [@​ulfjack](https://github.com/ulfjack) in cli/cli#7091 - configurable maxwidth for markdown WithWrap() by [@​smemsh](https://github.com/smemsh) in cli/cli#9626 - Handle errors when parsing hostname in auth flow by [@​BagToad](https://github.com/BagToad) in cli/cli#9729 - Add `repo license list/view` and `repo gitignore list/view` by [@​BagToad](https://github.com/BagToad) in cli/cli#9721 - Introduce testscript acceptance tests generally, and for the MR command specifically by [@​williammartin](https://github.com/williammartin) in cli/cli#9745 - Support `GH_ACCEPTANCE_SCRIPT` env var to target a single script by [@​williammartin](https://github.com/williammartin) in cli/cli#9756 - Ensure Acceptance defer failures are debuggable by [@​williammartin](https://github.com/williammartin) in cli/cli#9754 - Add acceptance task to makefile by [@​williammartin](https://github.com/williammartin) in cli/cli#9748 - Add Acceptance tests for `issue` command by [@​williammartin](https://github.com/williammartin) in cli/cli#9757 - Update IsEnterprise and IsTenancy for orthogonality using go-gh by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9755 - Supporting filtering on `gist list` by [@​heaths](https://github.com/heaths) in cli/cli#9728 #### New Contributors - [@​ulfjack](https://github.com/ulfjack) made their first contribution in cli/cli#7091 - [@​smemsh](https://github.com/smemsh) made their first contribution in cli/cli#9626 **Full Changelog**: cli/cli@v2.58.0...v2.59.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The goal of this work is to have a set of automated tests that we can use to point
ghat a real GitHub host in order to have some sense of blackbox behavioural validation. This PR introduces the use oftestscriptusing thegh prcommand as a motivating example.It is a non-goal right now to have these running as part of our CI suite.
Reviewer Notes
The README contains most information that I won't repeat in this description.
One big change here is that in order to avoid building
ghand to enable code coverage, I had to move theRealMainthat used to be in thecmd/gh/main.gointo its own package calledghcmd(mainpackages can't be imported), and to export the function. I can't really see much of an issue with this, it's actually pretty idiomatic.