Use new constraints field for cluster assignment
#462
Conversation
|
I added @mewil, as he knows the constraint stuff better than I! |
| env_vars=self.env_vars, | ||
| priority=priority, | ||
| ) | ||
| .with_constraint("cluster", [clusters] if isinstance(clusters, str) else clusters) |
| experiment, | ||
| strict=True, | ||
| quiet=True, | ||
| timeout=6.0, | ||
| poll_interval=5.0, | ||
| ) | ||
| time.sleep(2.0) | ||
| time.sleep(5.0) |
There was a problem hiding this comment.
Eases back the volume of requests FYI @codeviking
There was a problem hiding this comment.
So this will make a request every 5 seconds, instead of every 2, per job -- is that right?
This still feels like a lot. Most of our jobs are quite long. Plus, it often takes at least a minute or two for Beaker to start the job by shoveling around datasets, containers and the like.
So I'd propose ratcheting this way back. What if we only asked for a job's status twice per minute?
There was a problem hiding this comment.
+1 to doing fewer requests for long-running jobs. Here's a potentially simple heuristic:
If time observing < 5 minutes: poll every 5 seconds
If time observing >= 5 minutes: poll ever 1 minute
If time observing >= 30 minutes: poll every 5 minutes
What do you think?
There was a problem hiding this comment.
I think we can just keep it simple and always poll twice per minute
No description provided.