Upgrade from 1.21.8 to 1.22.5 fails (due to active PSP?) #107229
Description
What happened?
I am currently running a Kubernetes cluster on version 1.21.8 which has Pod Security Policies in place. The cluster has been set up manually using kubeadm on bare metal.
After "kubeadm upgrade apply 1.22.5", the new API server did not start up. Due to the kubelet logs, I believe this might be due to incorrectly configured PodSecurityPolicies, which however work perfectly fine in the current cluster. A more expert user might be able to fix this, but I am not.
Any advice on how to still perform the upgrade is greatly appreciated. Since it is a live cluster, I would prefer to upgrade instead of dismantling and reinstalling.
(probably) Relevant messages from journalctl -u kubelet:
Dec 27 10:25:15 beholder01 kubelet[1258223]: E1227 10:25:15.275456 1258223 kubelet.go:1683] "Failed creating a mirror pod for" err="pods \"kube-apiserver-beholder01\" is forbidden: PodSecurityPolicy: unable to admit pod: [pod.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: Forbidden: seccomp may not be set pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/kube-apiserver]: Forbidden: seccomp may not be set]" pod="kube-system/kube-apiserver-beholder01"
Dec 27 10:25:15 beholder01 kubelet[1258223]: I1227 10:25:15.275580 1258223 scope.go:111] "RemoveContainer" containerID="726355ec47d3a4c179f048253b8071d386a08a7feab908efd4e68d3884058f56"
Dec 27 10:25:15 beholder01 kubelet[1258223]: E1227 10:25:15.276835 1258223 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-beholder01_kube-system(00f07179d3d1bf50fac213d5777f0b24)\"" pod="kube-system/kube-apiserver-beholder01" podUID=00f07179d3d1bf50fac213d5777f0b24Relevant output of kubeadm upgrade apply:
[upgrade/staticpods]` Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-12-27-10-24-07/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-beholder01 hash: 753c1321c78e051dbb5d618d4ada8f4c
...
[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: timed out waiting for the conditionWhat did you expect to happen?
Since Pod Security Policies are supposed to be still working in 1.22.5 (although deprecated), I would expect the upgrade to work normally, thus I filed as a bug instead of support request.
How can we reproduce it (as minimally and precisely as possible)?
I believe that upgrading with existing Pod Security Policies has been tested, so it has probably something to do with my specific cluster setup and is not easily reproducible.
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:46:22Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux beholder01 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux