Codecov Report

Merging #5026 (f1b5f58) into master (fa01253) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head f1b5f58 differs from pull request most recent head 6e8d370. Consider uploading reports for the commit 6e8d370 to get more accurate results

@@           Coverage Diff           @@
##           master    #5026   +/-   ##
=======================================
  Coverage   41.73%   41.74%           
=======================================
  Files         108      108           
  Lines       10157    10158    +1     
=======================================
+ Hits         4239     4240    +1     
  Misses       5470     5470           
  Partials      448      448           

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/retest
/override ci/prow/e2e-gcp

@saschagrunert: Overrode contexts on behalf of saschagrunert: ci/prow/e2e-gcp

kata-jenkins failure is:

07:21:15 #   Warning  FailedCreatePodSandBox  4s (x7 over 89s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = CreateContainer failed: unrecognised machinetype: pc: unknown
07:21:15 # pod "handlers" deleted
07:21:15 Failed at 69: bats "${K8S_TEST_ENTRY}"

which seems unrelated

e2e-gcp failure is:

fail [k8s.io/[email protected]/test/e2e/storage/ubernetes_lite_volumes.go:163]: Unexpected error:     <*errors.errorString \| 0xc00263f220>: {         s: "PersistentVolumeClaims [pvc-1] not all in phase Bound within 5m0s",     }     PersistentVolumeClaims [pvc-1] not all in phase Bound within 5m0s

Also seems like a flake.

Updated with tests in network.bats

/retest

@dcbw: The following test failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

Ongoing sandbox requests cannot be (or are not) canceled by kubelet, leading to a situation where short-lived pods (especially Kubernetes e2e tests for stateful sets) cause overlapping sandbox requests

weird this sounds like a kubelet bug. I would expect cri-o to fail to create a duplicate pod while the first request is ongoing, and synchronously wait on cni plugin, thus preventing duplicate calls.

this change is fine to me, but I fear we're putting a bandaid on a bigger wound
/lgtm

/override ci/prow/e2e-gcp

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/e2e-gcp

weird this sounds like a kubelet bug. I would expect cri-o to fail to create a duplicate pod while the first request is ongoing, and synchronously wait on cni plugin, thus preventing duplicate calls.

this change is fine to me, but I fear we're putting a bandaid on a bigger wound

@haircommander the scenarios are something like this:

Scenario 1: pod recreated during sandbox wait

1. pod created, kubelet notices, asks CRI to create the sandbox
2. CRI creates sandbox, execs CNI plugin
3. CNI plugin gets pod from apiserver, starts setting up networking
4. something deletes and recreates the pod
5. CNI plugin waiting for networking to converge

Now in this scenario, the plugin could create a pod watch for delete events. But that's not race-proof since the pod could be deleted + recreated between steps 2 and 3 (see scenario 2) and the sandbox would be for an old pod (and be subsequently torn down by kubelet at some point in the future). Pod UID allows the plugin to notice the pod instance it gets in (3) or (5) is different and exit early.

Scenario 2: pod deleted during sandbox init

1. pod created, kubelet notices, asks CRI to create the sandbox
2. CRI creates sandbox, execs CNI plugin
3. something deletes and recreates the pod
4. CNI plugin gets pod from apiserver, starts setting up networking
5. CNI plugin still waiting for networking

in this scenario, the CNI plugin gets the new pod instance, which is still wrong for this sandbox setup. Pod UUID immediately tells the plugin that its pod is gone and it can exit early.

In all cases, kubelet will just tear the sandbox down anyway, and what we're trying to prevent is waiting longer than necessary before noticing that this sandbox is useless.

I also tried a variant of this that asks the CRI for the sandbox metadata during the call, but that fails because ListPodSandbox only lists completed sandbox setups, not in-progress ones :(

I suppose the real fix would be to allow a sandbox delete + CNI DEL while an existing add was going on, or a CANCEL operation that kubelet could execute to tell the CRI and plugins to stop the request. But that's a much longer arc to make happen (but it should happen, and at least on the CNI side we are working on that via gRPC).

/cherry-pick release-1.22
/cherry-pick release-1.21

@dcbw: new pull request could not be created: failed to create pull request against cri-o/cri-o#release-1.22 from head openshift-cherrypick-robot:cherry-pick-5026-to-release-1.22: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between cri-o:release-1.22 and openshift-cherrypick-robot:cherry-pick-5026-to-release-1.22"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

/cherry-pick release-1.21

@dcbw: #5026 failed to apply on top of branch "release-1.21":

Applying: vendor: bump ocicni to 4ea5fb8752cfe
Using index info to reconstruct a base tree...
M	go.mod
M	go.sum
M	vendor/modules.txt
Falling back to patching base and 3-way merge...
Auto-merging vendor/modules.txt
Auto-merging go.sum
CONFLICT (content): Merge conflict in go.sum
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 vendor: bump ocicni to 4ea5fb8752cfe
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

thanks for the explaination @dcbw , makes sense!