zeek containers need to be limited in max number of open files or memory grows very large #747
Description
I was trying to debug large memory usage in Zeek from Malcolm's zeek-live container:
$ ps axf | grep "[/]opt/zeek/bin/zeek "
61308 ? Sl 0:03 | \_ /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local /opt/zeek/share/zeek/site/extractor.zeek /opt/zeek/share/zeek/site/extractor_override.interesting.zeek zeekctl base/frameworks/cluster zeekctl/auto
61744 ? Sl 0:14 | \_ /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p manager local /opt/zeek/share/zeek/site/extractor.zeek /opt/zeek/share/zeek/site/extractor_override.interesting.zeek zeekctl base/frameworks/cluster zeekctl/auto
62145 ? Sl 0:03 | \_ /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p proxy-1 local /opt/zeek/share/zeek/site/extractor.zeek /opt/zeek/share/zeek/site/extractor_override.interesting.zeek zeekctl base/frameworks/cluster zeekctl/auto
62533 ? Sl 0:04 \_ /opt/zeek/bin/zeek -i af_packet::enp0s25 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-1 local /opt/zeek/share/zeek/site/extractor.zeek /opt/zeek/share/zeek/site/extractor_override.interesting.zeek zeekctl base/frameworks/cluster zeekctl/auto
$ for PID in $(ps axf | grep "[/]opt/zeek/bin/zeek " | cols 1); do top -p "$PID" -b -n 1 2>&1 | tail -n 1; done
61308 tlacuac+ 20 0 18.1g 4.2g 53048 S 0.0 6.7 0:03.31 zeek
61744 tlacuac+ 20 0 17.8g 4.8g 53156 S 0.0 7.7 0:14.64 zeek
62145 tlacuac+ 20 0 17.1g 4.2g 52764 S 0.0 6.7 0:03.04 zeek
62533 tlacuac+ 20 0 17.3g 4.4g 118968 S 0.0 7.0 0:04.35 zeek
$ howmuchmem zeek
17.69 GiB
@awelzel on the Zeek team brought this issue (mheily/libkqueue#153) to my attention. Running ulimit -n inside the zeek-live container was returning 1073741816 (I'm still not sure where that number was coming from). As described in the comments on that bug, there is something in libkqueue (used by Zeek) that's allocating a ton of memory when that number is high.
We need to set Zeek's ulimit -n value lower, probably 65535 or something like that. Doing so, for me, caused Zeek's memory usage to be SIGNIFICANTLY lower.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status