test: node_modules installs for system-tests, other improvements #18574

flotwig · 2021-10-20T19:31:20Z

Closes Support system testing environment #16207

User facing changelog

Additional details

System tests are now scaffolded to /tmp, not ./packages/server/.projects.
- Now system tests must explicitly have their dependencies in package.json.
- The exception is a few common node modules (lodash,, ...)
- yarn install runs automatically locally and in CI for these projects.
Only the selected project is scaffolded, not all test projects.
e2e.exec now accepts the project name as project, not the project path.
See changes to system-tests/README.md for details.

How has the user experience changed?

PR Tasks

Have tests been added/updated?
[na] Has the original issue or this PR been tagged with a release in ZenHub?
[na] Has a PR for user-facing changes been opened in cypress-documentation?
[na] Have API changes been updated in the type definitions?
[na] Have new configuration options been added to the cypress.schema.json?

cypress-bot · 2021-10-20T19:31:23Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

cypress · 2021-10-20T20:01:43Z

Test summary

18708 • 1 • 202 • 0 • 3

Run details


Project	cypress
Status	Failed
Commit	`1de92bf`
Started	Nov 24, 2021 7:13 PM
Ended	Nov 24, 2021 7:24 PM
Duration	11:17 💡
OS	Linux Debian - 10.10
Browser	Multiple

View run in Cypress Dashboard ➡️

Failures

	cypress/integration/settings_spec.js		1
1	Settings > file preference panel > loads preferred editor, available editors and shows spinner

Flakiness

	commands/net_stubbing_spec.ts	2
1	network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
2	network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
	cypress/proxy-logging-spec.ts	1
1	Proxy Logging > request logging > xhr log has response body/status code

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

- only scaffold selected project - accept project name, not path - symlink some common node packages

…ement

(and thus no yarn.lock)

see https://stackoverflow.com/a/49455609/3474615 for details on why this is needed to fix retries_spec

…ement

chrisbreiding

This is great - nice work!

emilyrohrbough · 2021-11-23T19:47:47Z

circle.yml

          name: Restore all node_modules to proper workspace folders
          command: node scripts/circle-cache.js --action unpack

+  restore_cached_system_tests_deps:


Can you add a describe to this job describe how this cache differentiates from the existing node_modules cache, which includes system-tests/node_modules?

Might be helpful to specify these are specifically the project's node_modules within system-tests.

Sure. Do you know if the description is actually used anywhere? It's part of the circle.yml spec but IDK where it ends up, I think it may be the equivalent of a code comment: https://circleci.com/docs/2.0/configuration-reference/#commands-requires-version-21

emilyrohrbough · 2021-11-23T19:51:30Z

system-tests/package.json

  "main": "index.js",
  "scripts": {
+    "cache:key": "bash -c 'cat ./projects/**/{package.json,yarn.lock}'",
+    "projects:yarn:install": "node ./scripts/projects-yarn-install.js",


When running locally, this will need to be ran manually to setup the projects. Any reasons this shouldn't be ran as a post-install script?

We could leverage scripts/run-if-not-ci.sh to ensure this doesn't install / cache modules for non-system-test jobs in circleci.

Part of the idea here is that we can defer the install cost locally until you run the tests for the first time. You may never have a need to run error_ui_spec, so there's no need to install the node_modules for it until you run the test. Then, they'll be cached forevermore.

Putting it in postinstall would make install slower to get rid of the one-time cost when running the test for the first time locally, which is undesirable.

emilyrohrbough · 2021-11-23T19:52:14Z

circle.yml

+          name: Restore system tests node_modules cache
+          keys:
+            - v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-node-modules-cache-{{ checksum "system_tests_cache_key" }}
+            - v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-node-modules-cache-


Why are two keys needed?

Suggested change

- v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-node-modules-cache-

- v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-projects-node-modules-cache-

https://circleci.com/docs/2.0/caching/#restoring-cache

CircleCI restores caches in the order of keys listed in the restore_cache step. Each cache key is namespaced to the project, and retrieval is prefix-matched. The cache will be restored from the first matching key. If there are multiple matches, the most recently generated cache will be used.

So it will either restore the exact correct cache, or the latest created cache.

emilyrohrbough · 2021-11-23T19:53:28Z

circle.yml

+    steps:
+      - run:
+          name: Generate Circle Cache key for system tests
+          command: yarn workspace @tooling/system-tests cache:key > system_tests_cache_key


Can we use/update the existing scripts/circle-cache.js script to generate this cache key? The package.json script is straight-forward but it would align how we are generating these keys in circleci.

Discussed this with @tgriesser originally, but decided to split this out because the system-tests/projects cache key is way simpler to generate than the circle-cache.js key. It's a one-liner, versus the somewhat complicated logic in circle-cache to generate a key. This way we can keep circle-cache.js's specific purpose as clear as possible.

I didn't re-use any of the existing cache stuff, because I didn't want to make it extra confusing/add even more logic to that process.

emilyrohrbough · 2021-11-23T20:01:57Z

circle.yml

          - build
+          - system-tests-node-modules-install


Since system-tests-node-modules-install requires build, lets slim this requires list to system-tests-node-modules-install to make the circle ci graph a bit cleaner:

oo i didn't even realize it did this, nice catch

yarn was wrapping the stdout with timestamps and stuff, making the hash non-deterministic and breaking caching. now i see why `node_modules_install` just calls the script directly too.

…ement

* develop: test: node_modules installs for system-tests, other improvements (#18574) chore(deps): update dependency semantic-release to v17.2.3 [security] (#19022) chore: remove flaky ci jobs for main builds (#19071) chore(contributing): clarify PULL_REQUEST_TEMPLATE (#19068) fix: the shadow root container element is ignored when clicking an element in it. (#18908) 'Fix' flaky redirect test (#19042) release 9.1.0 [skip ci] fix: Allow 'this' to be used in overridden commands (#18899) fix(react): link to rerender example (#19020) chore(deps): update dependency aws-sdk to v2.814.0 [security] (#18948) fix: test config overrides leak for .only execution (#18961) feat: Set CYPRESS=true as env var in child processes where Cypress runs user code in Node (#18981) fix: Restore broken gif (#18987) chore: release @cypress/vite-dev-server-v2.2.1

) Co-authored-by: Emily Rohrbough <[email protected]>

* 10.0-release: feat(graphql): ability to update/query for appData (#19082) fix system test fix failing tests due to merge resolve conflicts test: node_modules installs for system-tests, other improvements (#18574) update yarn.lock chore(deps): update dependency semantic-release to v17.2.3 [security] (#19022) chore: remove flaky ci jobs for main builds (#19071) chore(contributing): clarify PULL_REQUEST_TEMPLATE (#19068) fix: the shadow root container element is ignored when clicking an element in it. (#18908) 'Fix' flaky redirect test (#19042) release 9.1.0 [skip ci] fix: Allow 'this' to be used in overridden commands (#18899) fix(react): link to rerender example (#19020) chore(deps): update dependency aws-sdk to v2.814.0 [security] (#18948) fix: test config overrides leak for .only execution (#18961) feat: Set CYPRESS=true as env var in child processes where Cypress runs user code in Node (#18981) fix: Restore broken gif (#18987) chore: release @cypress/vite-dev-server-v2.2.1

flotwig force-pushed the system-tests-refinement branch from 47c6d89 to 8603a13 Compare October 20, 2021 19:40

- scaffold to tmp dir

962c306

- only scaffold selected project - accept project name, not path - symlink some common node packages

flotwig force-pushed the system-tests-refinement branch from 8603a13 to 962c306 Compare October 20, 2021 21:08

flotwig added 25 commits October 21, 2021 14:36

fix smoke test

00c6736

Merge remote-tracking branch 'origin/develop' into system-tests-refin…

52a43bf

…ement

remove video_compression_spec bluebird usage

84f5e8f

async scaffoldCommonNodeModules

e6e1c64

fixtures.js -> .ts

efcec2a

revert change to num of video tests

3aebe0f

add node modules scaffolding

11f228e

update projects for error_ui tests with package.jsons

70378e1

tokenize relative path in monorepo yarn.lock

6f322cf

fix opts.static

f882d27

fix opts.static

4e70311

fix case where package.json has no dependencies

2a6204b

(and thus no yarn.lock)

fix record_spec

29c7ee9

fix controllers_spec

099a406

fix: resolve to package.json

deabd09

see https://stackoverflow.com/a/49455609/3474615 for details on why this is needed to fix retries_spec

fix retries_spec

1886b70

remove stubPackage

85632c7

ci caching

7a6f56a

do update earlier

00a9826

Merge remote-tracking branch 'origin/develop' into system-tests-refin…

5e08542

…ement

use file paths for workspace packages

19c17f5

Merge remote-tracking branch 'origin/develop' into system-tests-refin…

6223a15

…ement

fix record_spec

e2c141e

fix replaceAll

bf29714

remove unneeded logging

2d04800

flotwig requested review from jennifer-shehane, chrisbreiding and tgriesser and removed request for a team and jennifer-shehane November 22, 2021 15:21

chrisbreiding previously approved these changes Nov 23, 2021

View reviewed changes

emilyrohrbough reviewed Nov 23, 2021

View reviewed changes

address review feedback

b6aaa99

flotwig dismissed chrisbreiding’s stale review via b6aaa99 November 23, 2021 20:34

flotwig requested review from emilyrohrbough and chrisbreiding November 23, 2021 20:35

This was referenced Nov 24, 2021

fix: bump resource class for tests that require node_modules install #19079

Merged

Convert npm/**/examples/** tests to system-tests #19088

Closed

chrisbreiding previously approved these changes Nov 24, 2021

View reviewed changes

flotwig added 2 commits November 24, 2021 09:55

fix CI cache key generation

17a4f4c

yarn was wrapping the stdout with timestamps and stuff, making the hash non-deterministic and breaking caching. now i see why `node_modules_install` just calls the script directly too.

Merge remote-tracking branch 'origin/develop' into system-tests-refin…

86a2d87

…ement

flotwig dismissed chrisbreiding’s stale review via 86a2d87 November 24, 2021 14:56

fix

cf6f6d4

emilyrohrbough approved these changes Nov 24, 2021

View reviewed changes

Merge branch 'develop' into system-tests-refinement

1de92bf

flotwig merged commit 71d92e0 into develop Nov 24, 2021

flotwig deleted the system-tests-refinement branch November 24, 2021 20:10

cypress-bot bot mentioned this pull request Nov 24, 2021

Support system testing environment #16207

Closed

10 tasks

lmiller1990 pushed a commit that referenced this pull request Nov 29, 2021

test: node_modules installs for system-tests, other improvements (#18574

28a9123

) Co-authored-by: Emily Rohrbough <[email protected]>

lmiller1990 mentioned this pull request Nov 29, 2021

chore: merge develop into 10.0-release and resolve conflicts #19134

Merged

chrisbreiding mentioned this pull request Dec 13, 2021

chore: Fix server unit tests running on mac by using actual tmp dir #19350

Merged

5 tasks

flotwig mentioned this pull request Mar 17, 2022

test(system-tests): support npm for test projects #20664

Merged

1 task

	- v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-node-modules-cache-
	- v{{ .Environment.CACHE_VERSION }}-{{ arch }}-system-tests-projects-node-modules-cache-

test: node_modules installs for system-tests, other improvements #18574

test: node_modules installs for system-tests, other improvements #18574

Uh oh!

Conversation

flotwig commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User facing changelog

Additional details

How has the user experience changed?

PR Tasks

Uh oh!

cypress-bot bot commented Oct 20, 2021

Uh oh!

cypress bot commented Oct 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test summary

Run details

Failures

Flakiness

Uh oh!

chrisbreiding left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flotwig Nov 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

flotwig commented Oct 20, 2021 •

edited

Loading

cypress bot commented Oct 20, 2021 •

edited

Loading

flotwig Nov 23, 2021 •

edited

Loading