cri-o cri checkpoint feature #6749

abdelghanimeliani · 2023-03-21T10:24:57Z

abdelghanimeliani
Mar 21, 2023

hello everyone , in readed this blog post about the new alpha kubelet checkpoint api feature and i wanted to test it but i found that the new image generated from the checkpoint is smaller then first image and when i tied to pull it again i got an error "NO command specified" , mor details are in this issue kubernetes/kubernetes#116744 (comment) .
do you think that the problem is in the cri-o CheckpointContainer implementation or in my envirement maybe ?

Answered by adrianreber

Apr 6, 2023

One thing we did to make this work was disable copy on write on btrfs with chattr +C. Also see https://criu.org/Filesystems_pecularities

View full answer

haircommander · 2023-03-21T14:29:05Z

haircommander
Mar 21, 2023
Maintainer

@adrianreber WDYT

0 replies

adrianreber · 2023-03-21T14:35:39Z

adrianreber
Mar 21, 2023

@abdelghanimeliani please try with the latest version of CRI-O from git.

I think this is related to a fix latest CRI-O git has. A workaround could be to start CRI-O with --drop-infra-ctr=false.

Or start the container with shareProcessNamespace: true. Based on your Kubernetes issue you are using CRI-O 1.25.0 and that had problems restoring containers with pods without infrastructure containers.

0 replies

abdelghanimeliani · 2023-03-22T18:31:16Z

abdelghanimeliani
Mar 22, 2023
Author

@adrianreber , for some reason , im not able to build the package from the source code , is this fix included in version 1.26.1 or 1.26.2 ?

4 replies

adrianreber Mar 22, 2023

Not sure. There is some message about the required go version. Do you have 1.19?

abdelghanimeliani Mar 22, 2023
Author

yes i have 1.20

abdelghanimeliani Mar 22, 2023
Author

i will use 1.19 maybe that's the issue

adrianreber Mar 23, 2023

From the error message it does not sound like you are using 1.20.

abdelghanimeliani · 2023-03-23T12:50:02Z

abdelghanimeliani
Mar 23, 2023
Author

@adrianreber im using the last version of crio from git and go1.19 , i tried to run the pod with shareProcessNamespace: true and to to start CRI-O with --drop-infra-ctr=false , the same problem reported in kubernetes repo is happening

29 replies

adrianreber Mar 23, 2023

You just need to create it.

abdelghanimeliani Mar 23, 2023
Author

@adrianreber , okey here the logs

Error (criu/util.c:641): exited, status=4
Error (criu/util.c:641): exited, status=4
Warn (criu/proc_parse.c:930): Write 4294967295 to /proc/self/loginuid failed: Operation not permitted
Warn (criu/net.c:3427): Unable to get tun network namespace
Warn (criu/sk-unix.c:224): unix: Unable to open a socket file: Operation not permitted
Error (criu/net.c:3711): Unable create a network namespace: Operation not permitted
Warn (criu/net.c:3767): NSID isn't reported for network links
Warn (criu/net.c:3427): Unable to get socket network namespace
Error (criu/net.c:1246): Unexpected error: -1(Operation not permitted)
Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
Error (criu/kerndat.c:1000): Fail to mount tmfps to /tmp/.criu.move_mount_set_group.0g361d: Operation not permitted
Error (criu/kerndat.c:1653): kerndat_has_move_mount_set_group failed when initializing kerndat.
Error (criu/crtools.c:260): Could not initialize kernel features detection.

adrianreber Mar 23, 2023

What kind of system are you using? That looks strange. Is CRI-O running as root?

abdelghanimeliani Mar 23, 2023
Author

@adrianreber im using the "ubuntu/jammy64" vagrant box , and yes it is runing as root , before i was using bionic box but now im using the scripts from this blog post https://martinheinz.dev/blog/85

adrianreber Mar 23, 2023

The CRIU messages are really unusual. It looks like it is not running as root. Can you do a criu check --all -v4.

abdelghanimeliani · 2023-03-23T16:59:32Z

abdelghanimeliani
Mar 23, 2023
Author

@adrianreber
(00.000006) Version: 3.17.1 (gitid 0)
(00.000048) Running on kubemaster Linux 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64
(00.000067) File /run/criu.kdat does not exist
(00.004913) sockets: Probing sock diag modules
(00.005062) sockets: Done probing
(00.007312) Error (criu/util.c:641): exited, status=4
(00.009476) Error (criu/util.c:641): exited, status=4
(00.009561) Pagemap provides flags only
(00.015156) Found anon-shmem device at 1
(00.015217) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.015233) Reset 40278's dirty tracking
(00.015254) Dirty tracking support is OFF
(00.015260) Zero page detection failed, optimization turns off.
(00.015304) Found task size of 7ffffffff000
(00.015338) Warn (criu/proc_parse.c:930): Write 4294967295 to /proc/self/loginuid failed: Operation not permitted
(00.021464) Warn (criu/net.c:3427): Unable to get tun network namespace
(00.021497) Warn (criu/sk-unix.c:224): unix: Unable to open a socket file: Operation not permitted
(00.021791) Error (criu/net.c:3711): Unable create a network namespace: Operation not permitted
(00.022319) Warn (criu/net.c:3767): NSID isn't reported for network links
(00.022407) uffd: Lazy pages are not available: Operation not permitted
(00.022411) Lazy pages are not permitted
(00.023236) vdso: Parsing at 7ffdd77fd000 7ffdd77ff000
(00.023240) vdso: PT_LOAD p_vaddr: 0
(00.023243) vdso: DT_HASH: 120
(00.023245) vdso: DT_STRTAB: 300
(00.023248) vdso: DT_SYMTAB: 1c8
(00.023250) vdso: DT_STRSZ: 8b
(00.023252) vdso: DT_SYMENT: 18
(00.023254) vdso: nbucket 3 nchain d bucket 7ffdd77fd128 chain 7ffdd77fd134
(00.023258) vdso: rt [vdso] 7ffdd77fd000-7ffdd77ff000 [vvar] 7ffdd77f9000-7ffdd77fd000
(00.023767) vdso: Parsing at 7fe7c2062000 7fe7c2064000
(00.023776) vdso: PT_LOAD p_vaddr: 0
(00.023779) vdso: DT_HASH: b4
(00.023781) vdso: DT_STRTAB: 1f0
(00.023783) vdso: DT_SYMTAB: 140
(00.023786) vdso: DT_STRSZ: c0
(00.023788) vdso: DT_SYMENT: 10
(00.023790) vdso: nbucket 3 nchain b bucket 7fe7c20620bc chain 7fe7c20620c8
(00.023793) vdso: compat [vdso] 7ffdd7801000-7ffdd7803000 [vvar] 7ffdd77fd000-7ffdd7801000
(00.024302) Warn (criu/net.c:3427): Unable to get socket network namespace
(00.024343) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz
(00.024354) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 960
(00.024359) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.024363) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.024462) Error (criu/net.c:1246): Unexpected error: -1(Operation not permitted)
(00.024522) Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
(00.024604) Error (criu/kerndat.c:1000): Fail to mount tmfps to /tmp/.criu.move_mount_set_group.jaVOGx: Operation not permitted
(00.024638) Error (criu/kerndat.c:1653): kerndat_has_move_mount_set_group failed when initializing kerndat.
(00.024694) Found mmap_min_addr 0x10000
(00.024704) files stat: fs/nr_open 1048576
(00.024706) Error (criu/crtools.c:260): Could not initialize kernel features detection.

15 replies

abdelghanimeliani Mar 29, 2023
Author

@adrianreber

what about this

Warn  (criu/crtools.c:303): Using -t with criu restore is obsoleted
Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)

adrianreber Mar 29, 2023

That also looks like CRIU has been started from the command-line and not from CRI-O.

abdelghanimeliani Mar 30, 2023
Author

@adrianreber i tested it today whit fedora37 , i got the same error msg="criu failed: type NOTIFY errno 0\nlog file: /run/containers/storage/overlay-containers/counter/userdata/restore.log" , the kernel version is 6.0.7-301.fc37.x86_64 and everything is running as root , can you tell me what system you are using and how did you setup your envirement ?

adrianreber Mar 30, 2023

Fedora 37 and compiled CRI-O from latest git

adrianreber Mar 30, 2023

Can you try to reach out to me on slack or something like that.

adrianreber · 2023-04-06T07:00:15Z

adrianreber
Apr 6, 2023

One thing we did to make this work was disable copy on write on btrfs with chattr +C. Also see https://criu.org/Filesystems_pecularities

0 replies

jmLee4 · 2025-01-24T08:59:45Z

jmLee4
Jan 24, 2025

I encountered the same kubelet error. Why am I getting this issue even though I'm using XFS? I cannot solve it using chattr +C

1 reply

adrianreber Jan 24, 2025

chattr doesn't change anything on XFS. No need to do that.

What error do you get? This discussion is already pretty long, so it is hard to know what your problem is.

jmLee4 · 2025-01-24T11:17:48Z

jmLee4
Jan 24, 2025

I encountered the same issue as the author initially mentioned in kubernetes/kubernetes#116744. In short, after sending a request to kubelet to generate checkpoint files and using buildah to build them into an image, I encountered the same 'Error: no command specified' error when starting the Pod. I noticed that the image built by buildah is also very small in size, just like mentioned in the current issue.

Looking through the previous content, I learned that disabling CoW on BTRFS could solve the problem, but my environment is relatively old (CentOS 7, XFS, which is not a CoW filesystem itself), so I'm not clear which step caused this.

Here is my environment information, thank you in advance for your reply:

CRIU Version: 3.17.1:

criu check --all -v4

...
(00.021335) netlink: Collect netlink sock 0x574e
(00.021346) netlink: Collect netlink sock 0x33dd
(00.021355) netlink: Collect netlink sock 0x2396
(00.021369) netlink: Collect netlink sock 0x29
(00.021377) netlink: Collect netlink sock 0x18fd4fc9
(00.021385) netlink: Collect netlink sock 0x23
(00.021404) Warn  (criu/cr-check.c:1231): clone3() with set_tid not supported
(00.021416) Error (criu/cr-check.c:1273): Time namespaces are not supported
(00.021429) Error (criu/cr-check.c:1283): IFLA_NEW_IFINDEX isn't supported
(00.021438) Warn  (criu/cr-check.c:1300): Pidfd store requires pidfd_open syscall which is not supported
(00.021522) Warn  (criu/cr-check.c:1334): Nftables based locking requires libnftables and set concatenations support
(00.021537) Warn  (criu/cr-check.c:804): ptrace(PTRACE_GET_RSEQ_CONFIGURATION) isn't supported. C/R of processes which are using rseq() won't work.
(00.021668) Warn  (criu/cr-check.c:1160): compat_cr is not supported. Requires kernel >= v4.12
Looks good but some kernel features are missing
which, depending on your process tree, may cause
dump or restore failure.

Other env info

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux host6 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ containerd --version
containerd github.com/containerd/containerd/v2 v2.0.2 c507a0257ea6462fbd6f5ba4f5c74facb04021f4

7 replies

adrianreber Jan 24, 2025

If you are using containerd this cannot work. containerd does not support container restore through the CRI.

jmLee4 Jan 24, 2025

Is it because this PR hasn't been merged yet? In other words, for the ContainerCheckpoint feature, does K8S only support sending requests to kubelet to generate Checkpoint files, but doesn't support directly using the image after importing from these Checkpoint files?

adrianreber Jan 24, 2025

Yes, the PR you mentioned is needed. The feature in Kubernetes is officially called Forensic Container Checkpointing and is only about checkpointing containers.

jmLee4 Jan 24, 2025

Thank you so much! I plan to try switching from containerd to CRI-O, but I'd like to ask if I will still encounter the same issue mentioned in the current issue after switching to CRI-O. I'm concerned about encountering this problem on XFS but being unable to resolve it using the BTRFS solution.

adrianreber Jan 24, 2025

Like mentioned a couple of times, this is not file-system related.

cri-o cri checkpoint feature #6749

Uh oh!

Replies: 8 comments · 56 replies

Uh oh!

haircommander Mar 21, 2023 Maintainer

Uh oh!

Uh oh!

abdelghanimeliani Mar 22, 2023 Author

Uh oh!

Uh oh!

Uh oh!

abdelghanimeliani Mar 22, 2023 Author

Uh oh!

abdelghanimeliani Mar 22, 2023 Author

Uh oh!

Uh oh!

abdelghanimeliani Mar 23, 2023 Author

Uh oh!

Uh oh!

abdelghanimeliani Mar 23, 2023 Author

Uh oh!

Uh oh!

abdelghanimeliani Mar 23, 2023 Author

Uh oh!

Uh oh!

abdelghanimeliani Mar 23, 2023 Author

Uh oh!

abdelghanimeliani Mar 29, 2023 Author

Uh oh!

Uh oh!

abdelghanimeliani Mar 30, 2023 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 8 comments 56 replies

haircommander
Mar 21, 2023
Maintainer

abdelghanimeliani
Mar 22, 2023
Author

abdelghanimeliani Mar 22, 2023
Author

abdelghanimeliani Mar 22, 2023
Author

abdelghanimeliani
Mar 23, 2023
Author

abdelghanimeliani Mar 23, 2023
Author

abdelghanimeliani Mar 23, 2023
Author

abdelghanimeliani
Mar 23, 2023
Author

abdelghanimeliani Mar 29, 2023
Author

abdelghanimeliani Mar 30, 2023
Author