Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Native memory leak with long-lived SharedInterpreter instances in JEP #618

@ahabel-wob

Description

@ahabel-wob

Describe the problem
We are observing a native (off-heap) memory leak in a JVM application embedding Python via JEP.

Our setup uses one long-lived SharedInterpreter per thread. These interpreters are intentionally kept open for the entire lifetime of the thread to minimize latency, since initializing a new interpreter is relatively expensive. During initialization, we import all our Python modules and load the necessary classes and utilities into the JEP context.

We do not pollute the global scope, every invocation calls a method in an isolated scope. We assume that every variable is cleaned up after we exit the method.

Over time, we see the container’s RSS memory usage steadily increasing, while all JVM-side metrics remain stable:

  • JVM heap (old + new gen) — flat
  • Loaded classes, Direct buffer usage, metaspace — flat

What we have tried:

  • Triggering Python garbage collection manually (gc.collect() in Python) — no effect
  • Periodically closing and recreating interpreters after N invocations

Before building a minimal reproduction, we would appreciate any debugging hints or recommended profiling tools for investigating deeper.

Environment (please complete the following information):

  • OS Platform, Distribution, and Version: Linux
  • Python Distribution and Version: 3.13
  • Java Distribution and Version: 21 eclipse temurin
  • Jep Version: 4.3-dev
  • Python packages used (e.g. numpy, pandas, tensorflow): msgspec

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions