Node configuration validation for fs.may_detach_mounts sysctl failed: fs.may_detach_mounts sysctl: expected 1, got 0 on CRI-O Version 1.20.0 #4558
Description
Description
Update my cluster to 1.20.1 sample:
$ kubectl version --short
Client Version: v1.20.1
Server Version: v1.20.1And I also updated my CRI-O socket to the latest version:
$ crio -v
INFO[0000] Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(dirty)
crio version 1.20.0
Version: 1.20.0
GitCommit: d388528dbed26b93c5bc1c89623607a1e597aa57
GitTreeState: dirty
BuildDate: 1980-01-01T00:00:00Z
GoVersion: go1.15.3
Compiler: gc
Platform: linux/amd64
Linkmode: staticAs part of my testing procedure of the cluster I restart one of the master nodes in HA cluster with 3 master nodes and 3 workers.
After restart I can see that the socket is not restarting:
$ sudo systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/usr/local/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-02-09 10:28:46 CET; 31s ago
Docs: https://github.com/cri-o/cri-o
Process: 1391 ExecStart=/usr/local/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
Main PID: 1391 (code=exited, status=1/FAILURE)
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09T10:28:46+01:00" level=info msg="Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(dirty)"
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09 10:28:46.938998374+01:00" level=info msg="Node configuration value for hugetlb cgroup is true"
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09 10:28:46.939026585+01:00" level=info msg="Node configuration value for pid cgroup is true"
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09 10:28:46.939295532+01:00" level=info msg="Node configuration value for memoryswap cgroup is true"
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09 10:28:46.943370828+01:00" level=info msg="Node configuration value for systemd CollectMode is false"
Feb 09 10:28:46 node-name crio[1391]: time="2021-02-09 10:28:46.943910044+01:00" level=fatal msg="Node configuration validation for fs.may_detach_mounts sysctl failed: fs.may_detach_mounts sysctl: expected 1...ving containers"
Feb 09 10:28:46 node-name systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Feb 09 10:28:46 node-name systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).
Feb 09 10:28:46 node-name systemd[1]: Unit crio.service entered failed state.
Feb 09 10:28:46 node-name systemd[1]: crio.service failed.Steps to reproduce the issue:
- Launch a cluster with the same cri-o version and also the same k8s version (k8s: 1.20.1 / cri-o: 1.20.0).
- Restart the master node without destroying the master. For example (
sudo shutdown -r now) - Once the node is up again simply check status of cri-o socket:
systemctl status crio
Describe the results you received:
The socket if failing to restart with error message:
Node configuration validation for fs.may_detach_mounts sysctl failed: fs.may_detach_mounts sysctl: expected 1, got 0; this may result in \"device or resource busy\" errors while stopping or removing containers"Describe the results you expected:
Expected result is the socket to be up and running after restarting the node.
Additional information you deem important (e.g. issue happens only occasionally):
Output of crio --version:
INFO[0000] Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(dirty)
crio version 1.20.0
Version: 1.20.0
GitCommit: d388528dbed26b93c5bc1c89623607a1e597aa57
GitTreeState: dirty
BuildDate: 1980-01-01T00:00:00Z
GoVersion: go1.15.3
Compiler: gc
Platform: linux/amd64
Linkmode: staticAdditional environment details (AWS, VirtualBox, physical, etc.):
The cluster is running on on premises on RH7 OS:
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)