crio is crashed during container restore #8083
Description
What happened?
- I'm following this procedure to test container checkpoint/restore with Kubernetes 1.30:
https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone - After I created checkpoint of container started by Kubernetes successfully, I deleted Kubernetes cluster, and try to restore a container via crio.
- I used following json to create pod sandbox:
crictl runp pod-config.json
{
"metadata": {
"name": "clone-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"linux": {
}
}- Then I used following json to restore container from checkpoint:
crictl create ef80f5a8c32b7 container-config.json pod-config.json
{
"metadata": {
"name": "test"
},
"image":{
"image": "/home/tonyaw/k8s/kubelet/checkpoints/checkpoint-tonyaw-deploy-78fdd8bbf5-5p2zf_default-test-2024-04-29T20:43:15+08:00.tar"
}
}- crictl is failed, and crio is crashed:
[root@test-host kubelet]# crictl create 96480f6885a14 container-config.json pod-config.json
E0429 20:50:58.512622 44825 remote_runtime.go:319] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Dead
lineExceeded desc = context deadline exceeded" podSandboxID="96480f6885a14"
crio journal log and backtrace:
crio-crash-log.txt
What did you expect to happen?
container is restored successfully.
How can we reproduce it (as minimally and precisely as possible)?
Follow procedure https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone to restore a container with Kubernetes 1.30.
The issue shall happen when you restore the container.
Anything else we need to know?
No response
CRI-O and Kubernetes version
Details
$ crio --version
WARN[0000] Failed to decode the keys ["storage.options.override_kernel_check"] from "/etc/containers/storage.conf"
crio version 1.30.0
Version: 1.30.0
GitCommit: a3de11a8aa3070c8b040054461ce82314c95baf2
GitCommitDate: 2024-04-26T16:51:03Z
GitTreeState: clean
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.22.0
Compiler: gc
Platform: linux/amd64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
exclude_graphdriver_devicemapper
seccomp
apparmor
selinux
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "30",
"gitVersion": "v1.30.0",
"gitCommit": "7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a",
"gitTreeState": "clean",
"buildDate": "2024-04-17T17:36:05Z",
"goVersion": "go1.22.2",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3"
}
The connection to the server 10.67.39.251:6443 was refused - did you specify the right host or port?I deleted Kubernetes cluster, as crictl can't create pod successfully if Kubernetes cluster is up with calico.
OS version
Details
# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
$ uname -a
Linux foss-ssc-11 5.1.11-1.el7.elrepo.x86_64 #1 SMP Mon Jun 17 15:51:08 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux