Signed-off-by: Peter Hunt <[email protected]>

Before when a client's request for a RunPodSandbox or ContainerCreate timed out, CRI-O would clean up the resource.

However, these requests usually fail when the node is under load. In these cases, it would be better to hold onto the progress,
not get rid of it.

This commit uses the previously created ResourceCache to cache the progress of a container creation and sandbox run.
When a duplicate name is detected, before erroring, the server checks in the ResourceCache to see if we've already
successfully created that resource. If so, we return it as if we'd just created it.

It also moves the SetCreated call to after the resource is deemed as not having timed out.

Hopefully, this reduces the load on already overloaded nodes.

Signed-off-by: Peter Hunt <[email protected]>

Even if we use the resource cache as is, the user is still bombarded with messages saying the name is reserved.

This is bad UX, and we're capable of improving it.

Add watcher idiom to resource cache, allowing a handler routine of RunPodSandbox or CreateContainer to
wait for a resource to be available.

Something that is key here is if the resource becomes available while we're watching for it,
*we still need to error on this request*
This is because we could get the resource from the cache, remove it (thus meaning it won't be cleaned up),
and the kubelet's request could time out, and it could try again. This would cause us to leak a resource.

This way, if we get into this situation, there needs to be three requests:
first that times out
second that discovers the resource is ready, but still errors
third that actually retrives that resource and returns it.

This will result in many fewer "name is reserved" errors (one every 2 seconds to one every 4 minutes)

Signed-off-by: Peter Hunt <[email protected]>

Now that we plan on caching the results of a pod sandbox creation, we shouldn't short circut the
network creation. In a perfect world, we'd give the CNI plugin unbounded time, which would allow
us to reuse even the longest of CNI creation time. However, this leads to the chance that the
CNI plugin runs forever, which is not ideal.

Instead, give the sandbox network creation 5 minutes (a minute more than the full request),
to improve the odds we have a completed sandbox that can be reused, rather than thrown away.

Signed-off-by: Peter Hunt <[email protected]>