More robust playwright tests#13457
Conversation
|
Thank you for the pull request, @jjspace! ✅ We can confirm we have a CLA on file for you. |
b23a739 to
af59516
Compare
|
I noticed during the release that the sandcastle specs don't ignore development only sandcastles which may not exist. This can happen during release since |
donmccurdy
left a comment
There was a problem hiding this comment.
After running npm run build-sandcastle I tried:
npm run test-e2e -- -g "3d-tiles-1.1-cdb-yemen|3d-tiles-gaussian-splatting|3d-tiles-gaussian-splats-with-lod|callback-position-property|callback-property|custom-primitive-dev|czml-path|manually-controlled-animation|material-with-custom-glsl|relative-paths|moon|multi-part-czml|particle-system|time-dynamic-wheels"Result:
Of the failures, about half exceeded the 100,000ms timeout. The other half had small (3–12%) pixel differences. I don't necessarily feel that any of this should block the PR if it's an improvement over current status, but wasn't sure if it was expected.
I also noticed that workers are disabled in playwright.config.js without. My results on these tests were similar (but faster) with workers turned back on — do we know of any reason that couldn't be on by default?
| await cesiumPage.page.clock.runFor(1000); | ||
| await cesiumPage.page.clock.runFor(1000); | ||
| await cesiumPage.page.clock.runFor(1000); |
There was a problem hiding this comment.
Is this different from runFor(3000)?
There was a problem hiding this comment.
The await should allow any completed promises to resolve. I believe simply sleeping the thread does not do so.
There was a problem hiding this comment.
The playwright runFor function is a little weird. It claims to "fast forward" any timing based functions. It seems like it should work with the viewer to process data but in my testing it's still a little flaky and doesn't actually advance time. Maybe it would work with a single 3000 instead of multiple, multiple were there before so I just kept it as I was tweaking things.
@donmccurdy As long as the system is set up correctly I wouldn't expect any tests to hit the max timeout. Can you check the troubleshooting steps from the testing guide to make sure you're running them with the gpu? |
Description
While prepping for the release and running the e2e tests I got nerd sniped again by wishing they were more robust and reproducible. This PR includes a handful of changes, mostly to the sandcastle specs, that made the snapshots much more consistent on my machine. This is a slight follow up to #13381
--use-gl=vulkanas I believe most, if not all, of the team said this works cross platform and was annoying to continually do on linux where it was actually neededThese changes are still not perfect but in my testing only a small handful of specs actually fail or need re-run with these changes. So much so that I'm planning to open another PR right after this to actually commit snapshots to the repo as a reference point. No more "generate against the previous release" during the release! 🎉
Issue number and link
Testing plan
npm run test-e2e -- --uiand test running a number of specs of different typesAuthor checklist
CONTRIBUTORS.mdCHANGES.mdwith a short summary of my changeAI acknowledgment
If yes, I used the following Tools(s) and/or Service(s):
If yes, I used the following Model(s):