Could it be a fix for #15373?

Happy to see the additional logs and artifacts ended up being useful.

/lgtm

It is not clear to me if the go runtime suspend the signal (makes it non blocking) before the signal is send to channel and so there is no race.

I think the new loop already eliminates any possible race since it reaps all the waiting child processes in one go, even if only one SIGCHLD was sent.
/approve

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vladikr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/hold

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-k8s-1.31-windows2016
/test pull-kubevirt-e2e-kind-1.33-vgpu
/test pull-kubevirt-e2e-kind-sriov
/test pull-kubevirt-e2e-k8s-1.33-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.32-sig-network
/test pull-kubevirt-e2e-k8s-1.32-sig-storage
/test pull-kubevirt-e2e-k8s-1.32-sig-compute
/test pull-kubevirt-e2e-k8s-1.32-sig-operator
/test pull-kubevirt-e2e-k8s-1.33-sig-compute-serial
/test pull-kubevirt-e2e-k8s-1.33-sig-network
/test pull-kubevirt-e2e-k8s-1.33-sig-storage
/test pull-kubevirt-e2e-k8s-1.33-sig-compute
/test pull-kubevirt-e2e-k8s-1.33-sig-operator

It is not clear to me if the go runtime suspend the signal (makes it non blocking) before the signal is send to channel and so there is no race.

I'm not completely sure, but I think the race happens when multiple child processes finish before the first signal is received. Since signals are just flags and aren't queued, we only get one signal, even if more than one child has exited. That could explain the issue, though it's a rare case.

It is not clear to me if the go runtime suspend the signal (makes it non blocking) before the signal is send to channel and so there is no race.

I'm not completely sure, but I think the race happens when multiple child processes finish before the first signal is received. Since signals are just flags and aren't queued, we only get one signal, even if more than one child has exited. That could explain the issue, though it's a rare case.

Yes, this is exactly what is happening but the think I describe is the abstraction that Go does. It matters if the signal is processed first and the sig is send to channel or the sig can be send to channel before the signal was processed. In the latter case we can end the loop, new child dies and we miss it.

Anyway I did small test

diff --git a/pkg/virt-launcher/monitor_test.go b/pkg/virt-launcher/monitor_test.go
index e2879b4963..97cfd3ecdb 100644
--- a/pkg/virt-launcher/monitor_test.go
+++ b/pkg/virt-launcher/monitor_test.go
@@ -21,7 +21,9 @@ package virtlauncher
 
 import (
 	"flag"
+	"os"
 	"os/exec"
+	"os/signal"
 	"path/filepath"
 	"strings"
 	"syscall"
@@ -29,6 +31,7 @@ import (
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	"kubevirt.io/client-go/log"
 
 	"github.com/google/uuid"
 )
@@ -108,8 +111,58 @@ var _ = Describe("VirtLauncher", func() {
 	AfterEach(func() {
 		if processStarted {
 			stopProcess()
+			_ = cmd.Wait()
 		}
-		_ = cmd.Wait()
+
+	})
+
+	FIt("t", func() {
+		start := func() *exec.Cmd {
+			cmd := exec.Command(fakeQEMUBinary, "--uuid", uuid.New().String(), "--pidfile", filepath.Join(pidDir, "fakens_fakevmi.pid"))
+			err := cmd.Start()
+			ExpectWithOffset(1, err).ToNot(HaveOccurred(), "command failed to start")
+			return cmd
+		}
+
+		reap := make(chan bool, 10)
+		sigs := make(chan os.Signal, 10)
+		signal.Notify(sigs, syscall.SIGCHLD)
+		go func() {
+			for sig := range sigs {
+				switch sig {
+				case syscall.SIGCHLD:
+					for {
+						var wstatus syscall.WaitStatus
+						wpid, err := syscall.Wait4(-1, &wstatus, syscall.WNOHANG, nil)
+						if err != nil {
+							log.Log.Reason(err).Errorf("Failed to reap process %d", wpid)
+						}
+						if wpid == 0 {
+							log.Log.Infof("No more processes to be reaped")
+							break
+						}
+						reap <- true
+						log.Log.Infof("Reaped pid %d with status %d", wpid, int(wstatus))
+					}
+
+				default:
+					Panic()
+				}
+			}
+		}()
+
+		cmds := []*exec.Cmd{}
+		for range 10 {
+			cmds = append(cmds, start())
+		}
+
+		for i := range 10 {
+			go func(i int) {
+				Expect(cmds[i].Process.Kill()).To(Succeed())
+			}(i)
+		}
+		Eventually(reap).Should(HaveLen(10))
+
 	})
 
 	Describe("VirtLauncher", func() {

Without the loop, we missed 1-7 signals. With the loop I couldn't reproduce, so I am pretty sure this helps but still not confident that this is completely fixing the issue.

big thanks @xpivarc

@xpivarc I had the same issue, but it was a bit more serious. #15711

@fossedihelm @vladikr PTAL

/lgtm
Big thanks
@vladikr up to you to unhold :)

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-k8s-1.31-windows2016
/test pull-kubevirt-e2e-kind-1.33-vgpu
/test pull-kubevirt-e2e-kind-sriov
/test pull-kubevirt-e2e-k8s-1.33-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.32-sig-network
/test pull-kubevirt-e2e-k8s-1.32-sig-storage
/test pull-kubevirt-e2e-k8s-1.32-sig-compute
/test pull-kubevirt-e2e-k8s-1.32-sig-operator
/test pull-kubevirt-e2e-k8s-1.33-sig-network
/test pull-kubevirt-e2e-k8s-1.33-sig-storage
/test pull-kubevirt-e2e-k8s-1.33-sig-compute
/test pull-kubevirt-e2e-k8s-1.33-sig-operator

/unhold

/retest-required

@xpivarc: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

/cherry-pick release-1.6 release-1.5 release-1.4 release-1.3 release-1.2 release-1.1 release-1.0

@xpivarc: new pull request created: #15816