pinns failure diagnostics is not adequate #4147
Description
Description
Pinns failure diagnostics is not adequate. Here's an excerpt from logs:
# time="2020-08-31 16:48:27.990804565-07:00" level=warning msg="Failed to find memory.max for slice: \"pod_123-456.slice\"" file="cgmgr/systemd.go:117"
# time="2020-08-31 16:48:27.990879645-07:00" level=debug msg="Calling pinns with [-d /var/run -f 034cf2f1-b18e-4853-95b0-05b8421c67b4 -n -i -u]" file="sandbox/namespaces_linux.go:127"
# time="2020-08-31 16:48:27.993261284-07:00" level=info msg="RunSandbox: cleaning up namespaces after failing to run sandbox 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c" file="server/sandbox_run_linux.go:681" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.993295622-07:00" level=info msg="RunSandbox: deleting pod ID 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c from idIndex" file="server/sandbox_run_linux.go:648" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.993328819-07:00" level=info msg="RunSandbox: removing pod sandbox 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c" file="server/sandbox_run_linux.go:635" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.993358646-07:00" level=info msg="RunSandbox: deleting container ID from idIndex for sandbox 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c" file="server/sandbox_run_linux.go:536" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.993385125-07:00" level=info msg="RunSandbox: unmounting shmPath for sandbox 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c" file="server/sandbox_run_linux.go:508" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.993547334-07:00" level=info msg="RunSandbox: removing pod sandbox from storage: 45ff147e5b3db805d255594f0f597eef84e194c86458b230c03a56b795d0f98c" file="server/sandbox_run_linux.go:361" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.994827466-07:00" level=info msg="RunSandbox: releasing container name: k8s_POD_podsandbox1_redhat.test.crio_redhat-test-crio_1" file="server/sandbox_run_linux.go:321" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.994869752-07:00" level=info msg="RunSandbox: releasing pod sandbox name: k8s_podsandbox1_redhat.test.crio_redhat-test-crio_1" file="server/sandbox_run_linux.go:301" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31 16:48:27.994900542-07:00" level=debug msg="Response error: failed to cleanup [/var/run/netns/034cf2f1-b18e-4853-95b0-05b8421c67b4 /var/run/ipcns/034cf2f1-b18e-4853-95b0-05b8421c67b4 /var/run/utsns/034cf2f1-b18e-4853-95b0-05b8421c67b4] after pinns failure exit status 1" file="go-grpc-middleware/chain.go:25" id=d6035872-bc9d-4d98-b3b4-b1e036c7546d name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
# time="2020-08-31T16:48:27-07:00" level=fatal msg="run pod sandbox failed: rpc error: code = Unknown desc = failed to cleanup [/var/run/netns/034cf2f1-b18e-4853-95b0-05b8421c67b4 /var/run/ipcns/034cf2f1-b18e-4853-95b0-05b8421c67b4 /var/run/utsns/034cf2f1-b18e-4853-95b0-05b8421c67b4] after pinns failure exit status 1"
There are a few problems in here.
-
There is no clear mention that pinns has failed (it is overshadowed by "failed to unmount" message).
-
Stderr from pinns is not shown at all.
-
umount errors are most probably bogus (as the paths were not mounted).