Error log: Failed to create existing container #5490
Description
Description
I checked the similar issues #3259 and #4465 , but fail to fix with the workaround.
My situation is that the error messages were triggered after restarting crio & kubelet.
Though the error messages can disappear after reboot, is there any solution to this problem without rebooting?
Steps to reproduce the issue:
systemctl stop kubeletsystemctl restart criosystemctl start kubeletjournalctl -xeu kubelet -f
Describe the results you received:
I executed tests with different cgroup drivers, and I received errors and warnings below in kubelet logs every minute.
#kubelet & crio cgroup driver use systemd:
kubelet[16645]: E1119 12:07:05.393398 16645 manager.go:1123] Failed to create existing container: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e900d5d_3559_4bc2_9b52_761eb5ed3c3f.slice/crio-217e7ff7799d62d02282afd1a0f2a8bbba85e62c8614ad4bf0aa9c029f2a0661.scope: Error finding container 217e7ff7799d62d02282afd1a0f2a8bbba85e62c8614ad4bf0aa9c029f2a0661: Status 404 returned error &{%!s(*http.body=&{0xc001297aa0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x55fd88bf3020) %!s(func() error=0x55fd88bf2fa0)}
#kubelet & crio cgroup driver use cgroupfs:
kubelet[7608]: E1203 12:30:49.996168 7608 manager.go:1123] Failed to create existing container: /kubepods/burstable/pod5f65e3d8d489e8fea295a7cd01aff842/crio-a3e66d2af18c551b3a281b5369de361a19ed1ea1ed508563fbdf94366cb321eb: Error finding container a3e66d2af18c551b3a281b5369de361a19ed1ea1ed508563fbdf94366cb321eb: Status 404 returned error &{%!s(*http.body=&{0xc000b6cfd8 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x55fe06c23f40) %!s(func() error=0x55fe06c23ec0)}
Describe the results you expected:
In kubelet logs, no errors or warnings of "Failed to create existing container".
Additional information you deem important (e.g. issue happens only occasionally):
I received these logs every 60 seconds after restarting crio and kubelet.
Output of crio --version:
crio version 1.22.1
Version: 1.22.1
GitCommit: 63ca93845d5fe05cdca826367afcb601ece8d7ad
GitTreeState: clean
BuildDate: 2021-11-11T20:24:17Z
GoVersion: go1.16.8
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic
BuildTags: exclude_graphdriver_devicemapper, seccomp
SeccompEnabled: true
AppArmorEnabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
OS: Centos 7
Kubernetes v1.22.1
crio configuration:
...
[crio.runtime]
conmon_cgroup = "system.slice"
cgroup_manager = "systemd"
...
Kubelet configuration:
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 5s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
kubeReserved:
cpu: 500m
kubeletCgroups: /systemd/system.slice
logging: {}
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s