Description
Hi John,
We are currently investigating a memory leak, it can be caused by lua-vips (or libvips 8.7 RC1), another dependency, a leak in OpenResty or a combination of these factors.
To exclude things, I started checking lua-vips for possible memory leaks. I've found a number of things:
-
While checking for possible memory leaks, I used valgrind on our test environment from the
openresty-valgrind
RPM package provided by OpenResty. With this lua-script:local vips = require "vips"
and this command:
valgrind --leak-check=full /usr/local/openresty-valgrind/luajit/bin/luajit-2.1.0-beta3 vips-init.lua &> output.txt
it generates this log: https://gist.github.com/kleisauke/57558977a2e31be0c809424078885196
Note: this is without any GLib / libvips suppression file. So it may be a false-positive. -
In addition to valgrind we've made some stacktraces to potential memory leaks with memleax on the production server. See the
memleax.txt
attachment. We usedmemleax -e 400
to report all memory allocations that haven't been freed after 400 seconds. -
While attempting to fix this (see: kleisauke@bb4971f), I found something odd:
Test image: https://images2.alphacoders.com/651/651450.jpgpyvips:
python3.6 soak-test.py /home/651450.jpg memory: high-water mark 80.83 MB
lua-vips master:
luajit-2.1.0-beta3 soak-test.lua -- /home/651450.jpg memory: high-water mark 1.02 GB
This may be the cause of the GC collector of LuaJIT, but it seems odd that the high-water mark is much higher than that of pyvips / python. I thought this was due to caching, but the soak test from Python is also disabling that: https://github.com/jcupitt/pyvips/blob/master/examples/soak-test.py#L9
I will further debug this (in combination with OpenResty) in the next few days, but I thought I'd let you know in advance. Any help would be greatly appreciated.