Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat: do not fail DERP healthcheck if WebSocket is used #10714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Nov 17, 2023
Merged

feat: do not fail DERP healthcheck if WebSocket is used #10714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
01cbf06
Add warnings property
mtojek Nov 15, 2023
6c84dc8
make gen
mtojek Nov 15, 2023
b66e38e
test derp
mtojek Nov 15, 2023
cec2e6d
another test
mtojek Nov 15, 2023
f1d7643
Warnings
mtojek Nov 15, 2023
98bd84c
make fmt
mtojek Nov 17, 2023
18f147a
Merge branch 'main' into 9470-warnings
mtojek Nov 17, 2023
5bc1366
one more
mtojek Nov 17, 2023
1fb7d56
TS
mtojek Nov 17, 2023
2f468fc
test fix
mtojek Nov 17, 2023
e98b9fc
one more
mtojek Nov 17, 2023
1671f5e
Better unit
mtojek Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions coderd/apidoc/docs.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 36 additions & 0 deletions coderd/apidoc/swagger.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion coderd/healthcheck/accessurl.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import (

// @typescript-generate AccessURLReport
type AccessURLReport struct {
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`

AccessURL string `json:"access_url"`
Healthy bool `json:"healthy"`
Reachable bool `json:"reachable"`
StatusCode int `json:"status_code"`
HealthzResponse string `json:"healthz_response"`
Expand Down
4 changes: 3 additions & 1 deletion coderd/healthcheck/database.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ const (

// @typescript-generate DatabaseReport
type DatabaseReport struct {
Healthy bool `json:"healthy"`
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`

Reachable bool `json:"reachable"`
Latency string `json:"latency"`
LatencyMS int64 `json:"latency_ms"`
Expand Down
34 changes: 25 additions & 9 deletions coderd/healthcheck/derphealth/derp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,14 @@ import (
"github.com/coder/coder/v2/coderd/util/ptr"
)

const (
warningNodeUsesWebsocket = `Node uses WebSockets because the "Upgrade: DERP" header may be blocked on the load balancer.`
)

// @typescript-generate Report
type Report struct {
Healthy bool `json:"healthy"`
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a map[string][]string so that it's easier to identify the source of the warnings?

That way we don't need to worry about doing fmt.Sprintf("[%s] %s", prefix, warning") all the time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might also make easier for the UI to display them separately.

Copy link
Member Author

@mtojek mtojek Nov 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong preference here, but []string is convenient as it is defined the same way for every section. Frontend can just render all warnings as is. If you find it useful, I can just adjust the implementation though.

Copy link
Member Author

@mtojek mtojek Nov 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might also make easier for the UI to display them separately.

My main concern is consistency with other warnings []string fields, but I don't mind changing it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what would be easier for the UI, to be honest. Maybe we just ship and iterate?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree 👍


Regions map[int]*RegionReport `json:"regions"`

Expand All @@ -39,8 +44,9 @@ type Report struct {

// @typescript-generate RegionReport
type RegionReport struct {
mu sync.Mutex
Healthy bool `json:"healthy"`
mu sync.Mutex
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`

Region *tailcfg.DERPRegion `json:"region"`
NodeReports []*NodeReport `json:"node_reports"`
Expand All @@ -52,8 +58,10 @@ type NodeReport struct {
mu sync.Mutex
clientCounter int

Healthy bool `json:"healthy"`
Node *tailcfg.DERPNode `json:"node"`
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`

Node *tailcfg.DERPNode `json:"node"`

ServerInfo derp.ServerInfoMessage `json:"node_info"`
CanExchangeMessages bool `json:"can_exchange_messages"`
Expand Down Expand Up @@ -108,6 +116,10 @@ func (r *Report) Run(ctx context.Context, opts *ReportOptions) {
if !regionReport.Healthy {
r.Healthy = false
}

for _, w := range regionReport.Warnings {
r.Warnings = append(r.Warnings, fmt.Sprintf("[%s] %s", regionReport.Region.RegionName, w))
}
mu.Unlock()
}()
}
Expand Down Expand Up @@ -159,6 +171,10 @@ func (r *RegionReport) Run(ctx context.Context) {
if !nodeReport.Healthy {
r.Healthy = false
}

for _, w := range nodeReport.Warnings {
r.Warnings = append(r.Warnings, fmt.Sprintf("[%s] %s", nodeReport.Node.Name, w))
}
r.mu.Unlock()
}()
}
Expand Down Expand Up @@ -208,14 +224,14 @@ func (r *NodeReport) Run(ctx context.Context) {

// We can't exchange messages with the node,
if (!r.CanExchangeMessages && !r.Node.STUNOnly) ||
// A node may use websockets because `Upgrade: DERP` may be blocked on
// the load balancer. This is unhealthy because websockets are slower
// than the regular DERP protocol.
r.UsesWebsocket ||
// The node was marked as STUN compatible but the STUN test failed.
r.STUN.Error != nil {
r.Healthy = false
}

if r.UsesWebsocket {
r.Warnings = append(r.Warnings, warningNodeUsesWebsocket)
}
}

func (r *NodeReport) doExchangeMessage(ctx context.Context) {
Expand Down
9 changes: 6 additions & 3 deletions coderd/healthcheck/derphealth/derp_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,11 +170,14 @@ func TestDERP(t *testing.T) {

report.Run(ctx, opts)

assert.False(t, report.Healthy)
assert.True(t, report.Healthy)
assert.NotEmpty(t, report.Warnings)
for _, region := range report.Regions {
assert.False(t, region.Healthy)
assert.True(t, region.Healthy)
assert.NotEmpty(t, region.Warnings)
for _, node := range region.NodeReports {
assert.False(t, node.Healthy)
assert.True(t, node.Healthy)
assert.NotEmpty(t, node.Warnings)
assert.True(t, node.CanExchangeMessages)
assert.NotEmpty(t, node.RoundTripPing)
assert.Len(t, node.ClientLogs, 2)
Expand Down
20 changes: 20 additions & 0 deletions coderd/healthcheck/healthcheck_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,25 @@ func TestHealthcheck(t *testing.T) {
},
healthy: false,
failingSections: []string{healthcheck.SectionDERP},
}, {
name: "DERPWarning",
checker: &testChecker{
DERPReport: derphealth.Report{
Healthy: true,
Warnings: []string{"foobar"},
},
AccessURLReport: healthcheck.AccessURLReport{
Healthy: true,
},
WebsocketReport: healthcheck.WebsocketReport{
Healthy: true,
},
DatabaseReport: healthcheck.DatabaseReport{
Healthy: true,
},
},
healthy: true,
failingSections: nil,
}, {
name: "AccessURLFail",
checker: &testChecker{
Expand Down Expand Up @@ -153,6 +172,7 @@ func TestHealthcheck(t *testing.T) {
assert.Equal(t, c.healthy, report.Healthy)
assert.Equal(t, c.failingSections, report.FailingSections)
assert.Equal(t, c.checker.DERPReport.Healthy, report.DERP.Healthy)
assert.Equal(t, c.checker.DERPReport.Warnings, report.DERP.Warnings)
assert.Equal(t, c.checker.AccessURLReport.Healthy, report.AccessURL.Healthy)
assert.Equal(t, c.checker.WebsocketReport.Healthy, report.Websocket.Healthy)
assert.NotZero(t, report.Time)
Expand Down
10 changes: 6 additions & 4 deletions coderd/healthcheck/websocket.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@ type WebsocketReportOptions struct {

// @typescript-generate WebsocketReport
type WebsocketReport struct {
Healthy bool `json:"healthy"`
Body string `json:"body"`
Code int `json:"code"`
Error *string `json:"error"`
Healthy bool `json:"healthy"`
Warnings []string `json:"warnings"`

Body string `json:"body"`
Code int `json:"code"`
Error *string `json:"error"`
}

func (r *WebsocketReport) Run(ctx context.Context, opts *WebsocketReportOptions) {
Expand Down
Loading