Description

In one of the HTTP services I am working on, we use the compress middleware provided by chi. We use standard gzip, deflate compression alongside with brotli compression.

The important aspect of the HTTP endpoint that uses the compression middleware is that the content may or may not be of a compressible type, serving json and HTML alongside png and jpg.

For the brotli compression we use the official CGO wrapper. The following code snippet is used to set the encoder:

compressor := middleware.NewCompressor(flate.DefaultCompression)

compressor.SetEncoder("br", func(w io.Writer, _ int) io.Writer {
	return cbrotli.NewWriter(w, cbrotli.WriterOptions{Quality: 5})
})

After enabling the brotli compression in the production environment, we noticed a memory leak in the service although there was no any issues with gzip compression.

After the investigation, we noticed no memory leak indicated by the Go runtime profiles. After this we started looking into the brotli library implementation.

Instrumented with the bcc tools (BPF Compiler Collection) we were able to find the leak:

[09:29:50] Top 10 stacks with outstanding allocations:
        655744 bytes in 94 allocations from stack
                0x00007a71453ecb3a      BrotliEncoderCreateInstance+0x30a [libbrotlienc.so.1.1.0]
                0x0000000001072263      _cgo_eb62034131f6_Cfunc_BrotliEncoderCreateInstance+0x23 [server]
                0x000000000040a755      runtime.cgocall+0x75 [server]
                0x000000000106956c      github.com/google/brotli/go/cbrotli._Cfunc_BrotliEncoderCreateInstance.abi0+0x4c [server]
                0x0000000001069919      github.com/google/brotli/go/cbrotli.NewWriter+0x39 [server]

It is clear from the report we are leaking the memory instantiated by the brotli writer. Looking at the source code of the compress middleware, we should call the underlying Close of the CGO writer which in turn should free up the C-allocated memory. We checked if the BrotliEncoderDestroyInstance is being called using the funccount-bpfcc:

sudo funccount-bpfcc '/usr/lib/x86_64-linux-gnu/libbrotlienc.so.1.1.0:BrotliEncoder*'
Tracing 13 functions for "b'/usr/lib/x86_64-linux-gnu/libbrotlienc.so.1.1.0:BrotliEncoder*'"... Hit Ctrl-C to end.
^C
FUNC                                    COUNT
BrotliEncoderDestroyInstance             4311
BrotliEncoderCreateInstance              5760
BrotliEncoderSetParameter                5760
BrotliEncoderCompressStream              8632
BrotliEncoderHasMoreOutput               8632
BrotliEncoderTakeOutput                  8632

Turned out, the BrotliEncoderDestroyInstance function is not being called at the same rate as BrotliEncoderCreateInstance which explains the leak.

After carefully looking into the go-chi middleware I realized we are calling the Close function returned by the writer() function, and writer() function returns instantiated writer only if the content is compressible.

In this PR we are trying to close the underlying Writer if it satisfies the WriteCloser interface, no matter if the content being served is compressible or not.

Related links

• go/cbrotli
• https://github.com/iovisor/bcc