Describe the solution you'd like

Today, shutdown worker RPC call when a server is gone hangs because it retries a lot. We should reduce this retry (maybe only twice). This doesn't need to affect all potential uses of the shutdown worker call, can just be applied at the shutdown_worker callsite in impl Worker.