-
Notifications
You must be signed in to change notification settings - Fork 163
Description
Describe the problem
We are observing a native (off-heap) memory leak in a JVM application embedding Python via JEP.
Our setup uses one long-lived SharedInterpreter per thread. These interpreters are intentionally kept open for the entire lifetime of the thread to minimize latency, since initializing a new interpreter is relatively expensive. During initialization, we import all our Python modules and load the necessary classes and utilities into the JEP context.
We do not pollute the global scope, every invocation calls a method in an isolated scope. We assume that every variable is cleaned up after we exit the method.
Over time, we see the container’s RSS memory usage steadily increasing, while all JVM-side metrics remain stable:
- JVM heap (old + new gen) — flat
- Loaded classes, Direct buffer usage, metaspace — flat
What we have tried:
- Triggering Python garbage collection manually (gc.collect() in Python) — no effect
- Periodically closing and recreating interpreters after N invocations
Before building a minimal reproduction, we would appreciate any debugging hints or recommended profiling tools for investigating deeper.
Environment (please complete the following information):
- OS Platform, Distribution, and Version: Linux
- Python Distribution and Version: 3.13
- Java Distribution and Version: 21 eclipse temurin
- Jep Version: 4.3-dev
- Python packages used (e.g. numpy, pandas, tensorflow): msgspec