Thanks to visit codestin.com
Credit goes to github.com

Skip to content

CI: test/go/postgres is flakey #1626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
greyscaled opened this issue May 20, 2022 · 1 comment · Fixed by #1759
Closed

CI: test/go/postgres is flakey #1626

greyscaled opened this issue May 20, 2022 · 1 comment · Fixed by #1759
Assignees
@coadler
Labels
api Area: HTTP API
Milestone
Community MVP

Comments

@greyscaled
Copy link
Contributor

greyscaled commented May 20, 2022 •
edited
Loading

Overview

In PRs that don't touch the data layer (for example A FE-only PR #1623), we're seeing that test/go/postgres often fails.

Truncated Log

The full log is too long, please see the actual action run.

=== Failed
=== FAIL: cli TestTTL/ShowOK (2.00s)
t.go:56: 2022-05-20 15:25:54.871 [DEBUG] coderd.go:372 POST /api/v2/users/first
t.go:56: 2022-05-20 15:25:54.904 [DEBUG] coderd.go:372 POST /api/v2/users/login
t.go:56: 2022-05-20 15:25:54.919 [DEBUG] coderd.go:372 POST /api/v2/files
t.go:56: 2022-05-20 15:25:54.932 [DEBUG] coderd.go:372 POST /api/v2/organizations/d0d2209f-c686-4baf-b53f-956c0454ca64/templateversions
t.go:56: 2022-05-20 15:25:54.989 [DEBUG] (provisionerd) provisionerd.go:128 connected
t.go:56: 2022-05-20 15:25:55.012 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.036 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.048 [DEBUG] (provisionerd-cranky_gauss6) provisionerdaemons.go:142 locked job from database {"id": "5e425b34-7615-473a-bb7c-378590f051b6"}
t.go:56: 2022-05-20 15:25:55.054 [INFO] (provisionerd) provisionerd.go:239 acquired job {"initiator_username": "testuser", "provisioner": "echo", "id": "5e425b34-7615-473a-bb7c-378590f051b6"}
t.go:56: 2022-05-20 15:25:55.062 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.073 [INFO] (provisionerd) provisionerd.go:353 unpacking template source archive {"size_bytes": 3072}
t.go:56: 2022-05-20 15:25:55.075 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.parse.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.079 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.provision.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.080 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.provision.dry.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.082 [DEBUG] (provisionerd) provisionerd.go:413 acquired job is template import
t.go:56: 2022-05-20 15:25:55.086 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.091 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.111 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.134 [INFO] (provisionerd) provisionerd.go:640 parse complete {"parameter_schemas": null}
t.go:56: 2022-05-20 15:25:55.140 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.142 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.166 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.187 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.195 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.206 [INFO] (provisionerd) provisionerd.go:722 parse dry-run provision successful {"resource_count": 0, "resources": null, "state_length": 0}
t.go:56: 2022-05-20 15:25:55.213 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.243 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.244 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.242 [INFO] (provisionerd) provisionerd.go:722 parse dry-run provision successful {"resource_count": 0, "resources": null, "state_length": 0}
t.go:56: 2022-05-20 15:25:55.269 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.290 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.290 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.332 [DEBUG] coderd.go:372 GET /api/v2/templateversions/8f5d7b3c-10e8-4f14-a801-81c689ab668c
t.go:56: 2022-05-20 15:25:55.342 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.339 [DEBUG] (provisionerd-cranky_gauss6) provisionerdaemons.go:527 marked import job as completed {"job_id": "5e425b34-7615-473a-bb7c-378590f051b6"}
t.go:56: 2022-05-20 15:25:55.350 [DEBUG] coderd.go:372 POST /api/v2/organizations/d0d2209f-c686-4baf-b53f-956c0454ca64/templates
t.go:56: 2022-05-20 15:25:55.375 [INFO] (provisionerd) provisionerd.go:451 completed job {"id": "5e425b34-7615-473a-bb7c-378590f051b6"}
t.go:56: 2022-05-20 15:25:55.375 [DEBUG] (provisionerd) provisionerd.go:315 cleaned up work directory {"error": null}
t.go:56: 2022-05-20 15:25:55.382 [DEBUG] provisionerdaemons.go:88 drpc server error ...
"error": stream closed
storj.io/drpc/drpcstream.(*Stream).sendPacket:268
storj.io/drpc/drpcstream.(*Stream).CloseSend:501
storj.io/drpc/drpcserver.(*Server).handleRPC:126
storj.io/drpc/drpcserver.(*Server).ServeOne:66
storj.io/drpc/drpcserver.(*Server).Serve.func2:112
storj.io/drpc/drpcctx.(*Tracker).track:52
t.go:56: 2022-05-20 15:25:55.411 [DEBUG] coderd.go:372 POST /api/v2/organizations/d0d2209f-c686-4baf-b53f-956c0454ca64/workspaces
t.go:56: 2022-05-20 15:25:55.455 [DEBUG] coderd.go:372 PUT /api/v2/workspaces/418ac39d-f91c-4dfa-a594-ed4453eb81e7/ttl
t.go:56: 2022-05-20 15:25:55.510 [DEBUG] (provisionerd-cranky_gauss6) provisionerdaemons.go:142 locked job from database {"id": "4436132f-47b5-4bc8-8856-de2465dd049a"}
t.go:56: 2022-05-20 15:25:55.530 [DEBUG] coderd.go:372 GET /api/v2/users/me/organizations
t.go:56: 2022-05-20 15:25:55.566 [DEBUG] coderd.go:372 GET /api/v2/organizations/d0d2209f-c686-4baf-b53f-956c0454ca64/workspaces/me/nostalgic-bassi
t.go:56: 2022-05-20 15:25:55.595 [INFO] (provisionerd) provisionerd.go:239 acquired job {"initiator_username": "testuser", "provisioner": "echo", "id": "4436132f-47b5-4bc8-8856-de2465dd049a"}
t.go:56: 2022-05-20 15:25:55.599 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.625 [INFO] (provisionerd) provisionerd.go:353 unpacking template source archive {"size_bytes": 3072}
t.go:56: 2022-05-20 15:25:55.628 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.parse.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.629 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.provision.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.630 [INFO] (provisionerd) provisionerd.go:910 failing running job {"error_message": "provisioner daemon was shutdown gracefully", "job_id": "4436132f-47b5-4bc8-8856-de2465dd049a"}
t.go:56: 2022-05-20 15:25:55.632 [DEBUG] (provisionerd) provisionerd.go:403 extracted file {"size_bytes": 2, "path": "/tmp/TestTTLShowOK2129889090/001/0.provision.dry.protobuf", "mode": "-rw-------"}
t.go:56: 2022-05-20 15:25:55.633 [DEBUG] (provisionerd) provisionerd.go:418 acquired job is workspace provision {"workspace_name": "nostalgic-bassi", "state_length": 0, "parameters": null}
t.go:56: 2022-05-20 15:25:55.691 [DEBUG] (provisionerd) provisionerd.go:931 marked running job as failed
t.go:56: 2022-05-20 15:25:55.694 [DEBUG] (provisionerd) provisionerd.go:206 skipping acquire; job is already running
t.go:56: 2022-05-20 15:25:55.695 [DEBUG] (provisionerd) provisionerd.go:905 job has already been marked as failed {"error_messsage": "write log: context canceled"}
t.go:56: 2022-05-20 15:25:55.698 [DEBUG] (provisionerd) provisionerd.go:905 job has already been marked as failed {"error_messsage": "write log: context canceled"}
t.go:56: 2022-05-20 15:25:55.704 [DEBUG] (provisionerd) provisionerd.go:315 cleaned up work directory {"error": null}
t.go:56: 2022-05-20 15:25:55.712 [DEBUG] (provisionerd) provisionerd.go:1000 closing server with error {"error": null}
testing.go:1312: race detected during execution of test
--- FAIL: TestTTL/ShowOK (2.00s)

Full Action Attempt

https://github.com/coder/coder/actions/runs/2359166491/attempts/1

Impact

This check is required (as it should be). We don't want to get into bad habits of either ignoring a failing CI check, or having it constantly fail and hold up PRs (especially those that don't touch the data layer). Lastly, we want to be certain it's catching true positives in the data layer.
@greyscaled greyscaled added bug 🐛 api Area: HTTP API and removed bug 🐛 labels May 20, 2022
@coadler coadler mentioned this issue May 20, 2022
Closed
@johnstcn johnstcn mentioned this issue May 23, 2022
Closed
@johnstcn
Copy link
Member

The following PRs attempt to address this:

@coadler coadler self-assigned this May 24, 2022
@coadler coadler linked a pull request May 25, 2022 that will close this issue
fix: race panic in test/go/postgres #1752
Closed
coadler added a commit that referenced this issue May 25, 2022
@coadler
chore: update cdr.dev/slog
47b8d0e 
Fixes #1626
@coadler coadler mentioned this issue May 25, 2022
Merged
coadler added a commit that referenced this issue May 25, 2022
@coadler
chore: update cdr.dev/slog (#1759)
b5d6153 
Fixes #1626
@coadler coadler removed a link to a pull request May 25, 2022
fix: race panic in test/go/postgres #1752
Closed
@misskniss misskniss added this to the Community MVP milestone Jun 3, 2022
kylecarbs pushed a commit that referenced this issue Jun 10, 2022
@coadler @kylecarbs
chore: update cdr.dev/slog (#1759)
7e3da30 
Fixes #1626
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coadler coadler

Labels
api Area: HTTP API
Projects
None yet
Milestone
Community MVP
Development

Successfully merging a pull request may close this issue.

chore: update cdr.dev/slog coder/coder
4 participants
@misskniss @johnstcn @coadler @greyscaled