Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Unexpected restarts of the coder pod #4807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
klauserber opened this issue Oct 31, 2022 · 0 comments · Fixed by #4821
Closed

Unexpected restarts of the coder pod #4807

klauserber opened this issue Oct 31, 2022 · 0 comments · Fixed by #4821
Assignees
@mafredri
Labels
api Area: HTTP API

Comments

@klauserber
Copy link

klauserber commented Oct 31, 2022 •
edited by coadler
Loading

Hello, I see sporadically this error on a GKE cluster:

(coder 0.11.0, GKE 1.24.6)

2022-10-31 07:34:58.861 [INFO]	(coderd.tailnet)	<./tailnet/conn.go:200>	NewConn.func3	wireguard status	{"status": "\u0026{AsOf:2022-10-31 07:34:58.861635364 +0000 UTC m=+1406.588420654 Peers:[{TxBytes:802036 RxBytes:5117168 LastHandshake:2022-10-31 07:33:55.787139294 +0000 UTC NodeKey:nodekey:cd32e74502524f98a828bf4adb7d4d23b0dd8ad0f0ce25006bf49c01f3e89123}] LocalAddrs:[{Addr:xxx.xxx.xxx.xxx:58148 Type:stun} {Addr:10.108.2.51:58148 Type:local}] DERPs:1}", "err": null}
2022-10-31 07:34:58.861 [INFO]	(coderd.tailnet)	<./tailnet/conn.go:456>	(*Conn).sendNode.func1	sending node	{"node": {"id": 5166511059428960592, "as_of": "2022-10-31T07:34:58.861635364Z", "key": "nodekey:33a3060e0d5648c2e31468634fc7811d4abb089049905814aac0eabd4f6d0c40", "disco": "discokey:282dfd24191766d8cc5d93b40231dced685b3227020aed41b18f57cc2eeaaf0e", "preferred_derp": 999, "derp_latency": {"999-v4": 0.008294595}, "addresses": ["fd7a:115c:a1e0:4970:805a:f00a:6f25:63b/128"], "allowed_ips": ["fd7a:115c:a1e0:4970:805a:f00a:6f25:63b/128"], "endpoints": ["xxx.xxx.xxx.xxx:58148", "xxx.xxx.xxx.xxx:58148"]}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x683178]

goroutine 107607 [running]:
bufio.(*Writer).Flush(0x800?)
	/opt/hostedtoolcache/go/1.19.2/x64/src/bufio/bufio.go:623 +0x18
net/http.(*chunkWriter).flush(0xc0150dc840?)
	/opt/hostedtoolcache/go/1.19.2/x64/src/net/http/server.go:402 +0x3f
net/http.(*response).Flush(0xc01bbaa000)
	/opt/hostedtoolcache/go/1.19.2/x64/src/net/http/server.go:1707 +0x48
github.com/coder/coder/coderd/tracing.(*StatusWriter).Flush(0x7f1929df9e78?)
	/home/runner/work/coder/coder/coderd/tracing/status_writer.go:81 +0x35
github.com/coder/coder/coderd/httpapi.ServerSentEventSender.func1()
	/home/runner/work/coder/coder/coderd/httpapi/httpapi.go:206 +0x202
created by github.com/coder/coder/coderd/httpapi.ServerSentEventSender
	/home/runner/work/coder/coder/coderd/httpapi/httpapi.go:191 +0x37d
@coadler coadler added bug api Area: HTTP API labels Oct 31, 2022
@mafredri mafredri self-assigned this Nov 1, 2022
mafredri added a commit that referenced this issue Nov 1, 2022
@mafredri
fix: Avoid panic in ServerSentEventSender by keeping handler alive
ed0189c 
The goroutine launched by `ServerSentEventSender` can perform a write
and flush after the calling http handler has exited, at this point the
resources (e.g. `http.ResponseWriter`) are no longer safe to use.

To work around this issue, heartbeats and sending events are now handled
by the goroutine which signals its closure via a channel. This allows
the calling handler to ensure it is kept alive until it's safe to exit.

Fixes #4807
@mafredri mafredri mentioned this issue Nov 1, 2022
Merged
mafredri added a commit that referenced this issue Nov 1, 2022
@mafredri
fix: Avoid panic in ServerSentEventSender by keeping handler alive (#…
e508057 
…4821)

The goroutine launched by `ServerSentEventSender` can perform a write
and flush after the calling http handler has exited, at this point the
resources (e.g. `http.ResponseWriter`) are no longer safe to use.

To work around this issue, heartbeats and sending events are now handled
by the goroutine which signals its closure via a channel. This allows
the calling handler to ensure it is kept alive until it's safe to exit.

Fixes #4807
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mafredri mafredri

Labels
api Area: HTTP API
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Avoid panic in ServerSentEventSender by keeping handler alive coder/coder
3 participants
@mafredri @coadler @klauserber