DATA RACE in Souin #576
Description
We keep getting DATA RACE conditions (or total crashes and outages that require manual restarts in the middle of the night)
and besides other issues with concurrency suspected in Vars Middleware, Replacer, Matcher
also the (newly in us) cache-handler/souin "middleware" is one of the suspects.
It's possible that this happened in a "HTTP-Connection-Destructor" or similar related to closing a lot of open tabs, but I could be totally wrong.
Posting this here for future reference:
{"level":"info","ts":1731590006.6006794,"logger":"http.handlers.cache","msg":"Reused response from concurrent request with the key POST-https-test-app.aiprm.com-/e83e6a6a-d8a5-4f07-93b8-5c382e67b4c5?cache=false-e2780c892f1da632ee66eaf799df47596cc068df114a8b93efa74c82bfab79fa"}
==================
WARNING: DATA RACE
Read at 0x00c025080000 by goroutine 418435:
runtime.slicecopy()
/usr/local/go/1.21.8/src/runtime/slice.go:310 +0x0
bytes.(*Buffer).Write()
/usr/local/go/1.21.8/src/bytes/buffer.go:181 +0x118
[github.com/darkweak/souin/pkg/middleware.(*CustomWriter).Write()](http://github.com/darkweak/souin/pkg/middleware.(*CustomWriter).Write())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/writer.go:75 +0x1b5
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:399 +0x8b7
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:709 +0x8e
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:710 +0x4f
Previous write at 0x00c025080000 by goroutine 416060:
runtime.slicecopy()
/usr/local/go/1.21.8/src/runtime/slice.go:310 +0x0
bytes.(*Buffer).Write()
/usr/local/go/1.21.8/src/bytes/buffer.go:181 +0x118
[github.com/darkweak/souin/pkg/middleware.(*CustomWriter).Write()](http://github.com/darkweak/souin/pkg/middleware.(*CustomWriter).Write())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/writer.go:75 +0x1b5
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:399 +0x8b7
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:709 +0x8e
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:710 +0x4f
Goroutine 418435 (running) created at:
[github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP()](http://github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP())
/LRT/go/pkg/mod/github.com/darkweak/[email protected]/pkg/middleware/middleware.go:708 +0x1e75
[github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP()](http://github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP())
/LRT/go/pkg/mod/github.com/caddyserver/[email protected]/httpcache.go:84 +0x112
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/headers/headers.go:110 +0x4c9
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP())
<autogenerated>:1 +0x94
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/headers/headers.go:110 +0x4c9
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP())
<autogenerated>:1 +0x94
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:300 +0x471
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.Rewrite.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.Rewrite.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/rewrite/rewrite.go:137 +0x4c1
[github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.(*Rewrite).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.(*Rewrite).ServeHTTP())
<autogenerated>:1 +0x12b
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:300 +0x471
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/subroute.go:74 +0x96
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:300 +0x471
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp/encode.(*Encode).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/encode.(*Encode).ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/encode/encode.go:138 +0x425
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/headers/headers.go:110 +0x4c9
[github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP())
<autogenerated>:1 +0x94
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:331 +0x81
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/routes.go:300 +0x471
[github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x47
[github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP())
/LRT/go/pkg/mod/github.com/christophcemper/caddy/[email protected]/modules/caddyhttp/subroute.go:74 +0x96
[github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()](http://github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1())
Not able to spend more time on collecting all "standard" details, as it's obvious that none of them help reproduce such rare issues, until we have a way to reproduce both data+timing in a consistent way.
The fork/build used is this one, which already. uses rwmutex + for the Vars Middleware so at least some potential problems can be excluded.
We've seen more Cache/Souin-related DATA RACE problems, as mentioned, but only on local builds.
so