Add CLI E2E tests for live GitHub and GitLab forge workflows#13709
Add CLI E2E tests for live GitHub and GitLab forge workflows#13709mtsgrd wants to merge 1 commit into
Conversation
First CLI-level end-to-end test suite that exercises `but` against real
forge APIs on a twice-daily schedule via GitHub Actions.
Test coverage (e2e/cli/):
- Create single PR / MR
- Create draft PR / MR
- Toggle draft ↔ ready state
- Create stacked PRs / MRs with correct base branch targets
- Force-push after amend keeps PR / MR open
Infrastructure:
- `but config forge auth --provider <github|gitlab> --token <PAT>`
Non-interactive auth for CI — stores token via but-api without
prompts. Also available via BUT_FORGE_TOKEN env var.
- `but-secret`: activate git-credentials backend when
E2E_TEST_APP_DATA_DIR is set (CI containers lack a system keychain).
- Test helper creates isolated app-data dir + git credential-store
per run, clones a sandbox repo, and cleans up branches/PRs after.
CI (.github/workflows/e2e-cli-continuous.yml):
schedule: 0 6,18 * * * (twice daily)
build CLI → run vitest → upload JUnit XML to Buildkite Test Engine
|
@krlvi in order to get live test runs twice daily using the 1. Create sandbox repos
Both should be empty repos that tests can freely push to and open PRs/MRs against. 2. Create bot accounts / tokens
3. Add secrets and variables in GitHub ActionsGo to the repo's Settings > Secrets and variables > Actions: Secrets:
Variables:
|
|
im not sure about the e2e setup here with typescript. Id like to defer to @davidpdrsn @slarse @Byron + others involved in the CLI - How do we want the CLI e2e setup to look like? |
|
Thanks for reading me in. I took a look and tried to be open-minded about the E2E CLI. At first I thought if it's just about managing the sandbox repo using the GH API, TS probably isn't too bad. Then I thought that a lot of that Whether or not existing plumbing crates should see changes to support E2E with forges, I think the implementation language should be familiar to those doing the Knowing myself, I tend to not care much about TS because I find it too 'fluffy', which would reduce my interest in interacting with it and making it better. Also, I'd always feel I couldn't review TS properly even if an agent does it, and I don't even want to get into TS anymore either. While admitting bias, it seems objectively better to do this in Rust even if the code will certainly look more complex due to typing. But… it's worth it? |
slarse
left a comment
There was a problem hiding this comment.
Minor comment on passing token via command line argument.
Other than that, I agree with @Byron. There should be a strong reason for writing tests in a different language, and I don't see any here. This seems to largely recreate but-testsupport with some additional and missing features.
That being said, this is a great initiative and I'd love to assist in getting this kind of testing in. I just don't think this is the way to do it.
| /// Personal Access Token to store. When provided, authentication is non-interactive. | ||
| #[clap(long, env = "BUT_FORGE_TOKEN", hide_env_values = true)] | ||
| token: Option<String>, |
There was a problem hiding this comment.
Taking a token as a CLI argument is not good practice, this is a prime avenue for unwitting devs to accidentally leak secrets. We don't need it for CI so I think it shouldn't be there.
I'd suggest only reading from the environment variable, or potentially taking it via stdin, to be used something like:
$ but config forge auth --provider github --token-stdin < token.txt
First CLI-level end-to-end test suite that exercises
butagainst realforge APIs on a twice-daily schedule via GitHub Actions.
Test coverage (e2e/cli/):
Infrastructure:
but config forge auth --provider <github|gitlab> --token <PAT>Non-interactive auth for CI — stores token via but-api without
prompts. Also available via BUT_FORGE_TOKEN env var.
but-secret: activate git-credentials backend whenE2E_TEST_APP_DATA_DIR is set (CI containers lack a system keychain).
per run, clones a sandbox repo, and cleans up branches/PRs after.
CI (.github/workflows/e2e-cli-continuous.yml):
schedule: 0 6,18 * * * (twice daily)
build CLI → run vitest → upload JUnit XML to Buildkite Test Engine