When You Can’t Test Everything – Testing Strategy To The Rescue

I am exploring end-to-end (E2E UI) tests in this post, but there are many levels for automated testing, which include integration and end-to-end tests. Often there is the situation that you cannot, should not, and must not test everything. Especially during the early phases of web app development it is difficult to test since tests tend to break in every way continuously. It is actually quite natural that the test coverage improves as the code base grows and matures, and finally the coverage reaches set target level – maybe around 50-100%. But when you begin writing tests, how do you decide which tests must be written first? What is their value? Why them, why now?

In E2E you care about functional coverage:

  • Critical paths – Can a user log in, place an order, reset a password?
  • High-value features – Revenue-impacting flows get full E2E coverage.
  • Risky areas – Flows prone to regressions, e.g., checkout, auth, payment.
  • Data loading – Load web app with relevant and correct data to test functionality

A metric for coverage unit could be a user journey, which means that a user typically needs to follow a path in order to complete a task or reach a goal. In terms of Agile development these could be user stories or tasks. For example,

  • Login with valid + invalid credentials
  • Add item to cart → checkout → payment
  • Search → filter → select result
  • Update user profile

Test data should NOT be stored as database scripts like DDLs, but rather any test state must be constructed using application methods (for example this is possible with Spring context). In a way there should be either UI or API approach to be able to load needed test data into the system, which should be also part of the E2E tests.

What should your testing strategy be like? Well, you can group test needs into four different groups, and then prioritize. Testing strategy must be (in this order): 1, 4, 2, 3.

  1. correct system behavior – happy path (desired outcome for user is reached). This includes critical and high-value paths: all critical paths must work with 100% certainty. Also, data needs to be loaded in the system to be able to test the system!
  2. correct system behavior – unhappy path (system behaves correctly under different cases where e.g. input is wrong)
  3. incorrect system behavior – does not happen in happy path (system does not expose incorrect behavior that would contradict correct behavior)
  4. incorrect system behavior – does not happen in unhappy path (system does not break when incorrect behavior happens): This includes critical path in negative sense!
Basic testing strategy Happy user pathUnhappy user path
Correct system behavior[1] user does everything correctly, and receives defined and expected value, which means this delivers most of the actual value![2] user does errors, which forces system into erroneous states from which it must guide user back to “optimal” track in order to deliver value
Incorrect system behavior[3] System behaves incorrectly but safely[4] System behaves incorrectly and unsafely:severe, business-breaking errors, or serious privacy or security issues. For instance, payments have incorrect amount or wrong recipient.
Basic testing strategy
Performance test strategyStable performanceUnstable performance
Correct system behaviorCapacity testing (system can reach specified capacity)Load testing (system can sustain specified capacity)
Incorrect system behaviorResilience testing (system’s ability to fail gracefully, and then recover from failures and disruptions)Stress testing (system’s ability to tolerate spikes and short-term load that exceeds specified limits)
System performance testing strategy

E2E UI tests should use Page-Object Model (POM), which allows Separation of Concerns “Tests” describe user flows in business terms e.g. login with valid credentials, add item to cart. “Page objects” describe implementation details of each page: element locators, interaction methods. POM typically requires that all (react) blocks have test identifier. When adding “test identifiers” (test-id) you should prefer roles, and labels over IDs because they mirror real user interaction, encourage accessibility, make tests robust to style/refactor changes, catch meaningful UI regressions, use IDs/test IDs only when there’s no user-facing attribute to query. Use IDs/test IDs only when there’s no user-facing attribute to query. In end-to-end tests, data-test-id is common because text/roles may vary with i18n, design, etc. but use them ONLY when necessary! Also, use Stable Identifiers for Dynamic Lists.

P.S. It is important to realize something that I have written about previously: you frontend and backend MUST support re-creating “current” state from scratch by executing call to APIs. What this means is that you are not allowed to load aggregated stored state, but the app must support a way to generate the state by calling functions. So it was a common practice to load a state to database using ddl scripts, but now you should not do it. For example, before I run my E2E test for my Java Spring Boot application (uses PostgreSQL), I do run schema updates using Flyway, and finally I call mostly private API endpoints (exposed only for testing) to set up a state. This allows me to be sure that I can fully trust my E2E tests later on!