Repeating kubelet errors with crio: Failed to create existing container #3259
Description
Description
I have a baremetal Kubernetes 1.17.2 cluster, using crio with the ZFS graph driver for storage. I have one host using crun as the default runtime, and one using runc as the default. I have both crio and kubelet configured to use systemd as the cgroup driver.
Steps to reproduce the issue:
- Use CRI-O on ZFS with Kubernetes
- Watch kubelet logs
Describe the results you received:
On both machines, I see errors and warnings like these in my logs every minute (or more, for the crio warnings):
crio[59456]: time="2020-02-15 13:44:32.065927387-07:00" level=warning msg="unable to get stats for container fc7efb695cfb307b48421225a47d19f172f6555d5a09a7a90821c9edc06a98f2: unable to load cgroup at kubepods-besteffort-pod0fb386ed7a9f3e4462d5182de8ccaa2b.slice: cgroup deleted" id=b01c9014-5969-4452-81b5-3d5a3f43230b
kubelet[59015]: E0215 13:44:37.944357 59015 manager.go:1086] Failed to create existing container: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fb386ed7a9f3e4462d5182de8ccaa2b.slice/crio-31b710e9b2340e9f18fbd3a14b19a44e76a4192e668239e5cfd9597681eed9bc.scope: Error finding container 31b710e9b2340e9f18fbd3a14b19a44e76a4192e668239e5cfd9597681eed9bc: Status 404 returned error &{%!s(*http.body=&{0xc0017ae640 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x74f5f0) %!s(func() error=0x74f580)}
Describe the results you expected:
I expect to see no errors or warnings in my kubelet logs with crio, the same way I don't see any warnings with docker or containerd.
Additional information you deem important (e.g. issue happens only occasionally):
I see these logs every 60 seconds.
Output of crio --version:
crio version 1.17.0
commit: "3afd3331a57e85bc56358f4e13802e7c25cd6c2d"
Additional environment details (AWS, VirtualBox, physical, etc.):
- baremetal
- debian bullseye
- crun
4a9b272b98768549da1277ec073c66c3ef51fd5b - runc
1.0.0~rc6+dfsg1