Overview

Our nightly XL e2e job terminates after exactly 6 hours of runtime due to GitHub runner limitations around job execution time. For example:

Logs here

If we look at the GitHub Actions documentation, it states the following about runner execution time:

Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

Source: https://docs.github.com/en/actions/administering-github-actions/usage-limits-billing-and-administration

"Root Cause"

Essentially, the total runtime of our XL e2e job has recently increased beyond 6 hours due to us needing to test more features, etc.. Because we need more than 6 hours to complete the job, we need to find a way to allow the XL e2e job to run for more than 6 hours;

Possible Solutions

Here are some possible solutions, but there may be other ideas that could work better:

1. Self-host our runners so we can keep them running >6 hours
2. Use a GitHub runner trigger the nightly XL E2E job, shut down the GitHub runner after the XL E2E job starts, then spin up a new GitHub runner every 30 mins to check the status of the job. (For reference: We never launch the XL e2e job during a PR anyway, so if someone needed to debug the XL e2e job, they could technically edit the workflow logic to keep the GitHub runner alive.)