I like the event-based approach, since that will scale better to large number of workspaces. Even with < 1000 workspaces, we were seeing high DB load from the default 1 minute scan interval (multiplied by 3 server instances in a HA configuraion).

If you keep the existing timer-based approach, I'd recommend spacing out the per-workspace checks over time instead of running them in a tight loop whenever the timer fires. Having adjusted the timer interval to 10 minutes, this results in clear load spikes on the database:

I like the event-based approach, since that will scale better to large number of workspaces.

My reservation here is that event-based systems can be tricky, and scheduling is also tricky 😅. So I'm definitely biased in favour of a simpler polling-based system.

As an alternative, I was thinking about the possibility of storing the next autostart time alongside the cron schedule in the database. That would allow us to only iterate over the workspaces that we know are due to start in the next poll interval, as opposed to all of the workspaces that could potentially start ever.

If you keep the existing timer-based approach, I'd recommend spacing out the per-workspace checks over time instead of running them in a tight loop whenever the timer fires.

Yeah, that's also a good call. Decoupling the workspace transition query logic from the workspace transition action logic would probably also help here.

A quick win is we could probably cache GetTemplateByID and GetTemplateVersionByID. A lot of workspaces are fetching the same data.

To give an update on this:

We've successfully managed to reduce the false-positive rate of GetWorkspacesEligibleForTransition.

As we can see from this graph of our dogfood instance, we have massively reduced the number of "last transition not valid" logs. These logs occur when GetWorkspacesEligibleForTransition returns workspaces that are not eligible for transition.

Unfortunately there is still work to do on this as we still return false-positives for the "is eligible for autostart" transition.

The benefit of reducing the false-positive rate of GetWorkspacesEligibleForTransition is that we now make less calls to the database. This is because for each workspace returned from this query, we proceed to make another 7 database calls.

One of the 7 DB calls that we make is to GetWorkspaceByID, and we can see that this change has meant that this query now has a smaller impact on the database as we make far fewer calls to it.

We've also seen a reduction in the data transfer from the database, this is for two reasons. One reason is a reduction in the number of workspaces returned. The second is because we no longer return the entire workspace row, instead only the ID and Name (as these were the only values we were using).

The next step of work is to stop returning false-positives for the "is eligible for autostart" transition. This should hopefully stop GetWorkspacesEligibleForTransition returning false-positives. Once we've achieved that, we will evaluate if any further steps need to be made (i.e. investigate a different approach, such as an event-based system), or if the load from this becomes insignificant.

Whilst writing some tests for #15594, we've discovered the tests require mocking time in provisionerdserver. This already somewhat exists but is inaccessible to the autostart/autostop tests.

Before making further progress on #15594, a change will need to be made to provisionerdserver to use a quartz.Clock, as well as a change to coderd to pass its clock to provisionerdserver.

Update:

On our dogfood instance we've managed to reduce the number of "last transition not valid" logs to 0.

Here is the per/minute count of "last transition not valid" logs for the past 30 days, notice the massive drop off at the first fix (discussed on the previous update post). Then we have a drop off again at the tail end of the graph, which can be noticed by the lack of data.

Here is that same graph but zoomed into the past 2 days.

This reduction has been achieved by tackling the final false positive situation: "is eligible for autostart". We previously only stored the cron schedule in the database, which meant we couldn't figure out if a workspace was eligible for autostart in our GetWorkspacesEligibleForTransition query. This meant the query returned any workspace that would be eligible for the autostart at any point in the future.

To rectify this we added a next_start_at column to the workspaces table so we can know when the next time a workspace should start is. This means our query will now only return workspaces it knows should be valid for an autostart.

To summarise: We've managed to stop the query from returning false-positives which should massively reduce the amount of database calls our lifecycle executor should make (each false-positive would trigger a further 7 database calls). This means our lifecycle executor should only be making 1 database call per tick (which defaults to 1 minute) unless any workspaces should need to transition, instead of up to 1 + $NUMBER_OF_WORKSPACES * 7 per tick.

If we discover this polling method still causes an unnecessary amount of database load, we can investigate if we need to consider a larger refactor.