Description
For heavily mult-threaded workloads like databases, the default of 1024 for pids-limit is too low. This parameter controls not just the number of processes but also the amount of threads (since a thread is technically a process with shared memory). When installing CRI-O via the PPA, it comes pre-packaged with a crio.conf that defaults to 1024. Also CRI-O defaults internally to 1024 according to crio --help | grep pids-limit.

I'm not sure what a good default is, but I set it to 64000 in my environment as that is what MongoDB recommended for nproc.

Steps to reproduce the issue:

1. Start CRI-O with defaults
2. Run the following test pod which kicks of 2048 threads:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: pid-limit
spec:
  containers:
  - image: docker.io/ssheehy/pid-limit
    name: pid-limit
EOF

Describe the results you received:
Pod crashes repeatedly since it can't create threads past 1024

Describe the results you expected:
Pod continues to run and can create > 1024 threads.

Additional information you deem important (e.g. issue happens only occasionally):
Discussed in slack with @runcom and @mrunalp

Output of crio --version:

crio version 1.11.9-dev
commit: ""

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 18.04
ulimit -u is 47931
Kubernetes v1.11.4