Timeout waiting for systemd to create kubepods-burstable-***.slice #120103
Description
What happened?
When I frequently increase and decrease the replica count (with python api)to adjust the number of pods over a period of time:
The cluster cost some more memory but with the same count of services(and pods), and at this point, continuing to adjust replicas will result in a significantly longer time (possibly even more than 600 seconds, normal is 1.5-3s) required for creating new pods.
As title: Timeout waiting for systemd to create kubepods-burstable-***.slice
And also: Timed out while waiting for systemd to remove kubepods-burstable-***.slice
What did you expect to happen?
normally start new pod without timeout
How can we reproduce it (as minimally and precisely as possible)?
frequent replica adjustments (roughly over 500 times within 15 hours, up to 40 one by one and reduce to 1 one by one) for a same service, use python api
Anything else we need to know?
I continuously adjust the replica count using a Python program. Under normal circumstances, the startup time of pods is shown in the left graph (where the left side represents the pod count, and the right side represents the startup time of new pods). However, after an extended period of frequent replica adjustments (roughly over 500 times within 15 hours), the duration is shown as depicted in the right graph.
Even Kubernetes' API timeouts in the end.
Kubernetes version
$ kubectl version
# paste output hereWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:16:51Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.11", GitCommit:"8cfcba0b15c343a8dc48567a74c29ec4844e0b9e", GitTreeState:"clean", BuildDate:"2023-06-14T09:49:38Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux iccl 5.15.0-79-generic #86~20.04.2-Ubuntu SMP Mon Jul 17 23:27:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here