Load testing

Load testing evaluates how the system performs under different levels of traffic. It helps ensure that the system remains stable during high-traffic events such as sales campaigns or seasonal peaks. When you change any part of your commerce solution, such as integrations or extensions, performance tests show how those changes affect response times. Early testing helps you detect issues sooner and reduce later rework costs. We recommend running load tests regularly.

Test phases

The typical phases of a load test are:

1. Definition phase:
   • Formulate scenarios by mapping user journeys.
   • Determine levels of traffic and load profiles.
   • Define primary objectives, including key performance indicators (KPIs).
2. Test script creation phase:
   • Design test scripts using the defined scenarios.
   • Run small-scale tests to ensure the quality of the scripts.
3. Load test performance phase:
   • Carry out different types of load tests based on the established scenarios.
   • Create a final report summarizing the results of the tests.
4. Cleanup phase:
   • Remove data generated by the tests once they have been completed.

Load test scenarios should focus on testing crucial workflows that mimic typical user behavior, with an emphasis on impactful and realistic scenarios for the business. For example, simulate a checkout journey: browsing, adding items to a cart, and placing an order.

Tests should examine the system's behavior under regular load conditions, validate its performance during high-traffic periods, and assess its performance under sustained load over an extended period.

Use the following table as a template to document expected hourly activity:

Types of load tests

Baseline load tests

Apply baseline load tests to simulate routine traffic scenarios and to validate major architectural changes regularly. Plan for future growth based on the results from load test analyses. Keep historic artifacts (load driver reports featuring performance data, resource usage, and observations) for comparison over time.

Peak load tests

Endurance tests

Choose the right environment

Obligations and responsibilities

• In the context of load testing, specific data such as Product catalog details may be required. Do not incorporate creating this data directly into your load tests. Generate the necessary data incrementally in advance to avoid undue stress on the system. Consider keeping supporting data for consecutive tests. For instance, if you run a daily load test for order creation and use the same Product data set, it's practical to keep that data set.
• Notify your commercetools representative at least three business days in advance to schedule any load tests.
• Generate traffic from nearby regions to minimize network overhead.
• Start your load testing conservatively. Perform initial testing at lower loads to validate responses and confirm the functionality of your load test scripts.
• Gradually increase load levels during testing. Directly starting at maximum load is unrealistic behavior and not permitted.
• Include periods of think time in performance tests to mimic human behavior.
• Monitor resource utilization (such as CPU, memory, and network usage) on the load driver machines that simulate the traffic.
• Remove and permanently delete data generated by the tests within one week from when the tests have been completed. Automated cleanup is recommended.

Fair-use test allowances and extra load testing add-on

Fair-use allowance

The following tests, aligned with industry standards, are considered fair use and are included as part of the commercetools offering by default:

• Automated load tests of development changes on the test Project (for example, regression CI/CD).
• Baseline load tests: Up to four production-level baseline load tests per calendar year across any endpoints (such as Carts, Orders, and Products). Each baseline test is limited to 1× expected peak load and to 25% of production purchased orders.
• One peak load test per calendar year, with traffic levels not exceeding the expected peak load (1×).
• Tests must run only on test Projects.

Remember: all data generated by the tests must be fully removed and deleted within [1 week] once the tests have been completed.

Extra Load Testing add-on

For information about further possibilities, such as the Extra Load Testing add-on, please contact your commercetools representative. The Extra Load Testing package extends the fair-use allowance as follows:

• Up to ten additional tests per calendar year, at up to 2× expected peak load, with a combined total duration not exceeding 15 hours annually.
• Aggregate annual test order volume across all load tests must not exceed the total number of production purchased orders in a calendar year.
• Any single test is capped at 50% of the total number of production purchased orders.
• Tests must run only on test Projects.
• Provide at least three business days advance notice before any test.
• All test data must be cleaned up and permanently deleted within one month after each test. Automated cleanup is recommended.

Exclusion periods

Load tests are not allowed in the following periods of the year:

• Two weeks before Black Friday to the Tuesday after Black Friday
• Between December 20 and December 26