Before this gets too far: can you address the concerns with this approach I raised over in #160 (comment) ?

I'll take a stab at responding as I'd really like to get this feature out as soon as possible :)

Cloning a volume via your cloud provider's API, then mounting it inside K8S is FAR more complicated than doing a simple copy via exec API. My understanding is that runner copies only the job "spec" (for the lack of better word) and maybe nodejs binary to what used to be a shared volume. Although maybe node is copied from the init container actually, I don't have the full picture of Nikola's implementation yet. In any case the size of this is relatively small and I don't see why it shouldn't be reliable. Doing a whole PV clone for <100MB of files seems like a huge overkill. Potentially heavy operations like repo cloning actually happen in workflow pod and wouldn't be copied using kube exec API.

Most importantly runner container hooks are written to be pretty generic and not prefer one cloud provider over the other.
Be careful what you wish for, even if they decided to implement something like you are suggesting. GCP/GKE would likely be the last one to get support for this. Both AWS and and Azure are bigger and I'm sure GH has more customers on those two clouds than GCP.

Hey @zarko-a! Yeah thanks for braining this out with me

my main concern was
"I have significant doubts that this will be a stable approach. At scale we observe even trivial use cases for the exec api (like exec into a pod and check for existence of a file on a cron) to fail for all sorts of reasons."

To expand on that:

• Anecdotally the exec API is super flakey. We experience lots of random connection issues and timeouts when we issued execs 100s of times per day as a part of our deploy workflows. This is anecdotal on Kube 1.25-1.27 though. We removed exec api stuff from the hot path around the time 1.27 was released.
  • I'm happy to burn some $$$ if we want to stress test this. Like Spin up hundreds of pod pairs and use this code to copy files between the pods.
• Logically the exec API is dependent on control plane uptime, which is not 100%
  • For instance in GKE land the control plane has a 99.5% and 99.95% monthly uptime SLA for zonal and regional clusters respectively. Intentional control plane upgrades and other things like that could also cause api downtime which would fail worker setup.

👉 So at minimum I would expect this implementation to expect these execs to fail or be interrupted. Heavily integrate backoff retries or something like that.

GCP/GKE would likely be the last one to get support for this. Both AWS and and Azure are bigger and I'm sure GH has more customers on those two clouds than GCP.

Well yeah if there was an ADR out for cloud specific providers my team would for sure contribute a GCP one in short order

Adding my 5 cents here. As we have altered hooks code on our end and implemented copy via k8s api, similar to how it was originally proposed in #160, we had to implement retry mechanism and the uptime is still not 100%, as sometimes copy fails even after a number of retries with or without sleep/wait incorporated between the attempts.

We do see the retry to kick in in roughly 20% of the executions.

The copy of the nodejs distro still takes a decent amount of time and the overall lag for workflow container kickstart varies between a 40s to 3min on our set up.

Hey @anlesk,

Thank you for your feedback! That is one of the reasons I wanted to use an init container to copy runner assets, so we can only copy the temporary directory. The _temp should be much smaller in size, resulting in a lower number of errors, but we will still add retries to ensure it works.
Initially, we would copy the workdir, which has actions downloaded. This one would be slightly larger, but node assets should be much larger than that.

Quick update:

Retries are added. We are trying to minimize the number of files being copied by leveraging the init container as much as possible. It is challenging with the container action, where the workspace should be mounted, and we don't know in advance what the workspace will look like at the time the container step is invoked.

But for most use-cases, a single workspace copy at the time the prepare-job is called should be okay, and each run-step basically copies the temp dir only to the container, and back to the runner. This approach may lead to fewer issues.

I was discussing this issue with someone and they mentioned it sounds similar to this problem AWS is looking to solve for their mountpoint-s3-csi-driver.
https://github.com/awslabs/mountpoint-s3-csi-driver/blob/0753c0635a38b68a4433683bef53b769ad2c7b40/docs/HEADROOM_FOR_MPPOD.md

This solution wouldn't get rid of the runner and workflow pods be on the same node requirement, but it does provide a potential solution for setting requests and limits on workflow pods and ensuring there is enough space on a node for these pods. I think everyone would prefer the flexibility of workflow pods going to whatever node has room for them, but if the issues people are bringing up with exec being too flaky ends up holding true, or the spin up time of these pods is heavily impacted by the copy solution, maybe this "headroom" pod solution is a decent alternative.

Hey @ukhl,

There is no way we can afford to build on top of a cloud-specific solution. However, the intention of this repo is to provide a solution that should work in most cases and would allow you to customize/modify the implementation that suits your needs.

The copy is needed because there are many instances where read-write-many volumes don't exist in user environments. If we had the luxury of relying on read-write-many volumes, it would avoid all the headaches of scheduling the workflow pod on the same node as the runner. We would simply rely on the shared filesystem maintained by a driver, allowing workflow pods to be scheduled on nodes with enough capacity to handle it.

Since the copy using exec is supported by Kubernetes itself, and many people have faced issues using the volume, we are trying our best to remove this dependency. I would personally love to avoid it, since we now have to make sure the file permissions are properly set, that there are binaries available on arbitrary job containers in order to execute the action, user volume mounts are applied to the correct places, etc... But we can't. That is why using the exec API with retries, transferring the least amount of files possible, does seem to be the best approach.

We truly appreciate the feedback, though! It is amazing to see this many people coming up with such helpful and thoughtful feedback and suggestions!

Hi @nikola-jokic I think you might have jumped to conclusions based on the repository I linked to. The doc I linked to isn't cloud proprietary. It's a method of getting multiple pods to schedule together via pod affinity.

• https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
• https://karpenter.sh/docs/concepts/scheduling/#pod-affinityanti-affinity

The idea here would be to schedule the runner pod and a dummy workflow pod at the same time. This pod would then be replaced with a real workflow pod if/when required.

This approach would be less efficient on resources, but only if you don't use workflow/step pods that often. It would solve the problem of trying to schedule a workflow/step pod on a node and there not being enough resources for it.