I'm running Kubernetes RKE2 v1.33.1+rke2r1 and MinIO Operator 7.1.1. I have a simple test tenant with 1 server and 2 disks using the DirectPV CSI driver. I've got the resources requests and limits configured for 24Gb of memory and 12 CPUs. I'm using this to simply test our configuration by storing ~10Tb of data. I'm able to write data (several million smallish files across 256 buckets) to the server at speeds that more than meet our criteria, which is no surprise. However, after restarting the Kubernetes node the MinIO server will not start. I've rebuilt the Kubernetes node and reinstalled MinIO and repeated the same experiment and the results are the same.

Looking at the kubelet log i see the message over and over:

err=failed to StartContainer for minio with CreateContainerError: failed to reserve container name minio_minio-pool ...

I left it run over the weekend to see if it would start, and it still hasn't started. When I do a describe on the pod it says Ready: false with a Restart Count: 84. I don't believe Kubernetes is trying to change file permissions because I see the operator already sets fsGroupChangePolicy to onRootMismatch.

Is there any reason a loaded MinIO server will not start on Kubernetes? It starts fine until I load millions of files. I'm out of ideas because I can't find any specific errors from containerd or the kubelet as to why it's not starting.