[ML] Better messaging regarding OOM process termination (#2841)

edsavage · web-flow · commit 8a713efa7778 · 2025-03-27T10:19:47.000+13:00
This PR provides a more detailed message when a process is terminated with SIGKILL. On Linux, the OOM (Out Of Memory) system handler will kill processes, according to heuristics, when the OS runs low on memory. Our native processes (apart from controller) are configured so that they would be chosen first to be terminated in such a situation. The OOM handler terminates processes with a SIGKILL (signal 9). SIGKILL is not able to be handled by processes and will result in immediate termination, not allowing for any logging of the situation. However, the parent process - controller - can detect and report on the death of its children. Relates elastic/ml-team#1158
diff --git a/docs/CHANGELOG.asciidoc b/docs/CHANGELOG.asciidoc
@@ -42,6 +42,12 @@
 
 * Update Linux build images to Rocky Linux 8 with gcc 13.3. (See {ml-pull}2773[#2773].)
 
+== {es} version 8.19.0
+
+=== Enhancements
+
+* Better messaging regarding OOM process termination. (See {ml-pull}2841[#2841].)
+
 == {es} version 8.18.0
 
 === Enhancements
diff --git a/lib/core/CDetachedProcessSpawner.cc b/lib/core/CDetachedProcessSpawner.cc
@@ -185,13 +185,22 @@ class CTrackerThread : public CThread {
                         // at a lower level
                         LOG_INFO(<< "Child process with PID " << pid
                                  << " was terminated by signal " << signal);
-                    } else {
+                    } else if (signal == SIGKILL) {
                         // This should never happen if the system is working
                         // normally - possible reasons are the Linux OOM
-                        // killer, manual intervention and bugs that cause
-                        // access violations
+                        // killer or manual intervention. The latter is highly unlikely
+                        // if running in the cloud.
+                        LOG_ERROR(<< "Child process with PID " << pid << " was terminated by signal 9 (SIGKILL)."
+                                  << " This is likely due to the OOM killer."
+                                  << " Please check system logs for more details.");
+                    } else {
+                        // This should never happen if the system is working
+                        // normally - possible reasons are bugs that cause
+                        // access violations or manual intervention. The latter is highly unlikely
+                        // if running in the cloud.
                         LOG_ERROR(<< "Child process with PID " << pid
-                                  << " was terminated by signal " << signal);
+                                  << " was terminated by signal " << signal
+                                  << " Please check system logs for more details.");
                     }
                 } else {
                     int exitCode = WEXITSTATUS(status);
