Steps to reproduce
- Run server 0.20.3.
- Create an instance divided into blocks.
- Run a job on one of the blocks.
- Run server 0.20.4 (or, until 0.20.4 is released, run from master with
DSTACK_USE_LATEST_FROM_BRANCH=1).
- Try running a new job on the same instance.
Actual behaviour
Until the job from step 3 is stopped, no other jobs can run on the instance.
smooth-mayfly-2 provisioning completed (failed)
Exited (none)
Incorrect Usage: flag provided but not defined: -home-dir
Expected behaviour
New jobs can run on the instance without stopping the job from step 3.
dstack version
master
Server logs
Additional information
After step 4, the runner and shim binaries are updated on the instance. However, shim cannot be restarted while a job is running, which on step 5 leads to the old shim binary trying to start the new runner binary, which has an incompatible CLI.