feat: Initial E2E test framework for v2 #288
Conversation
Codecov Report
@@ Coverage Diff @@
## main #288 +/- ##
==========================================
- Coverage 67.94% 67.75% -0.19%
==========================================
Files 150 150
Lines 8482 8482
Branches 72 72
==========================================
- Hits 5763 5747 -16
- Misses 2143 2156 +13
- Partials 576 579 +3
Continue to review full report at Codecov.
|
|
Thanks @vapurrmaid ! It's basically what we have in I like the idea of having at least login verified build-over-build as we're standing stuff up |
| if: (success() || failure()) && github.actor != 'dependabot[bot]' && runner.os == 'Linux' | ||
| env: | ||
| DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }} | ||
| DD_CATEGORY: e2e |
There was a problem hiding this comment.
@kylecarbs - I added a new DataDog trait for test runs ("category") - intending to be one of unit | integration | e2e
.github/workflows/coder.yaml
Outdated
| os: | ||
| - ubuntu-latest | ||
| - macos-latest | ||
| - windows-2022 |
There was a problem hiding this comment.
This is so cool! That means we can even test things like cross-os keybinds in the UI 😍
There was a problem hiding this comment.
For sure! Ideally - we'd have a matrix of windows | macos | ubuntu x relevant browsers (edge | chrome | firefox | safari). Getting macos + safari would be a huge value add, since we saw some bugs there in v1 (ie https://github.com/coder/m/pull/11093 & https://github.com/coder/m/pull/9508)
There was a problem hiding this comment.
Unfortunately, I had to pull out Windows for now because the build is failing - some of our build scripts aren't cross-plat ready - ie, the yarn_install.sh script which fails with:
./scripts/yarn_install.sh
Windows Subsystem for Linux has no installed distributions.
Distributions can be installed by visiting the Microsoft Store:
https://github.com/coder/coder/runs/5368058980?check_suite_focus=true#step:8:6
There was a problem hiding this comment.
Opened a couple issues to break down next steps:
- Windows build + run E2E: Build coderd + run E2E tests on Windows #384
- Run w/ other browsers: E2E | Run E2E tests against other browsers #385
| import { SignInPage } from "../pom" | ||
|
|
||
| test("Login takes user to /projects", async ({ page, baseURL }) => { | ||
| await page.goto(baseURL + "/", { waitUntil: "networkidle" }) |
There was a problem hiding this comment.
Interesting that unpacking the baseURL works here since it didn't in v1 - good sign! I believe you can avoid doing the string concatenation here though if you define url in the POM.
There was a problem hiding this comment.
Interesting that unpacking the baseURL works here since it didn't in v1 - good sign!
Yep! The v1 E2E tests use a special (external) address to point to the server under test (picked up from RUNTIME_CONFIG: https://github.com/coder/m/blob/833f23892c4802bfcdf9b39c86f415a1dc70a3b2/product/coder/e2e/configuration/runtime.ts#L18)
With these tests - using the webServer automatically populates the baseURL which is convenient.
There was a problem hiding this comment.
I believe you can avoid doing the string concatenation here though if you define url in the POM.
Good point - we have some more utilities in the POM layer in v1, I brought them over (ie, BasePOM) here in this change: 7c568de
It's a little bit awkward - since I'm passing both the baseURL and page into the POM's - let me know if you were thinking of a different approach!
There was a problem hiding this comment.
Implemented @vapurrmaid 's suggestions which help improve this in: fae6b3f - I'll merge what we have so the test is in place for the NextJS -> Webpack changes, but happy to iterate on it more
There was a problem hiding this comment.
@bryphe-coder I'm definitely in favor of this merging sooner than later for iteration, it's in a really good starting spot!
Co-authored-by: G r e y <[email protected]>
…r into bryphe/feat/initial-e2e-tests
|
Thanks for this one @bryphe-coder 🎉 ! |
|
@vapurrmaid - I like the proposal, IMO it's simpler than the current implementation I have. Thanks for the suggestion!
|
| export const username = "admin" | ||
| export const password = "password" | ||
| export const organization = "acme-crop" | ||
| export const email = "[email protected]" |
There was a problem hiding this comment.
@bryphe-coder thanks for implementing this! 🎉
I def didn't mean for it to block the work, but it's a nice touch 🎊
There was a problem hiding this comment.
Thanks for the great feedback & review @vapurrmaid !
This brings an initial E2E test (really, an integration test - it's only running the server locally, as opposed to against a deployment - but it'd be easy to point playwright to a deployment).
Demo gif:
This test exercises a minimal flow for login:
coderdbinary to start a server on 3000It will be useful to have to ensure that #360 doesn't introduce a regression in the login flow
Future E2E tests that would be useful:
Some remaining work left:
coderddirectly instead ofdevelop.shtest_resultsvstest-resultspaths