feat: Allow running standalone provisioner daemons #3563

dwahler · 2022-08-18T20:17:13Z

This PR adds support for running out-of-process provisioner daemon instances, authenticated by tokens (basically the same way we authenticate workspace agents).

Example usage

# Run coderd with the in-process provisioners disabled
$ coder server --address http://127.0.0.1:3000 --provisioner-daemons 0
[...]

$ coder provisioners create foobar
A new provisioner daemon has been registered.

Start the provisioner daemon with the following command:

coder provisioners run --token 033dc6c6-48a6-4316-83e3-536433a57521
$ coder provisioners run --token 033dc6c6-48a6-4316-83e3-536433a57521
[...]

Other notable things about this change:

ProvisionerDaemon API resources now have an auth_token field, which is null for in-process provisioners. External connections must include a non-null token in the session_token cookie. To avoid leaking tokens, we now prevent non-owner users from listing provisioner daemons through the API.
For more convenient testing, adds the ability to pass flags to coder server using ./scripts/develop.sh -- <flags...>.
Since we want to include the echo provisioner when running tests, but not when running in production, the provisioner daemon now performs a Connect RPC as its first action on each connection to coderd. This message is used to register the set of supported provisioner names and store them in the provisioner_daemons.provisioners database field.
As a side effect of the previous point, the provisioner_daemons.updated_at field, which was previously never set, is now updated on every provisioner daemon connection.

Fixes #1391, fixes #1392, fixes #1393, fixes #1605

This reverts commit a03615a.

github-actions · 2022-08-26T00:12:09Z

This Pull Request is becoming stale. In order to minimize WIP, prevent merge conflicts and keep the tracker readable, I'm going close to this PR in 3 days if there isn't more activity.

mafredri

Awesome job on this PR! In general I think it looks really good, despite the number of comments I left.

mafredri · 2022-08-30T11:05:47Z

cli/provisionerrun.go

+			return exitErr
+		},
+	}
+	defaultCacheDir := filepath.Join(os.TempDir(), "coder-cache")


Suggestion (optional): We could consider fixing #2534 (partially) here too? Something like adrg/xdg and using xdg.CacheHome could work (I only took a quick look, and it seems pretty fully-featured).

Then again, perhaps we should do a more thorough fix all at once, (i.e. respect XDG elsewhere too, like config).

Great suggestion, but I think this changeset is already kind of big and sprawling as it is (it fixes four separate issues) so I would lean toward doing that in a separate PR.

mafredri · 2022-08-30T11:08:18Z

cli/provisionerrun_test.go

+			require.ErrorIs(t, err, context.Canceled, "provisioner command terminated with error")
+		}()
+
+		ctx, cancelFunc := context.WithCancel(context.Background())


Consider allowing tests to timeout individually, e.g.:

Suggested change

ctx, cancelFunc := context.WithCancel(context.Background())

ctx, cancelFunc := context.WithTimeout(context.Background(), testutil.WaitLong)

mafredri · 2022-08-30T11:20:04Z

coderd/database/dump.sql

@@ -203,7 +203,8 @@ CREATE TABLE provisioner_daemons (
    created_at timestamp with time zone NOT NULL,
    updated_at timestamp with time zone,
    name character varying(64) NOT NULL,
-    provisioners provisioner_type[] NOT NULL
+    provisioners provisioner_type[] NOT NULL,
+    auth_token uuid


What's the use-case for allowing auth_token to be NULL? Do we want to be able to revoke an auth token without deleting the provisioner daemon? Maybe we want to lookup what provisioner daemon had a specific auth token at some point? In that case it could make sense to have a deleted bool field instead of a nullable auth token.

Maybe auth tokens should be renewable too? That could also work as a delete/create using the same name (would also leave a "trace" about which auth token has produced what builds).

Another alternative would be to have provisioner_daemon_auth_tokens with fields like daemon_id, token, granted, revoked.

Just putting out some ideas, since I wasn't sure of the purpose of the nullability.

Whoops, this could have used a better explanation. It doesn't have anything to do with changing or revoking auth tokens.

With this change, we support both in-process and out-of-process provisioners, both of which are persisted in the database. An in-process provisioner has a NULL auth_token to represent the fact that external connections are not allowed to "become" that provisioner. That seemed cleaner than assigning a token that would never be used.

mafredri · 2022-08-30T11:23:36Z

coderd/provisionerdaemons.go

+	api.websocketWaitMutex.Unlock()
+	defer api.websocketWaitGroup.Done()
+
+	conn, err := websocket.Accept(rw, r, &websocket.AcceptOptions{


Do consider that r.Context() is invalid from this point onwards (due to http Hijack). So if it's relied upon for connection closure / cancellation, it will not work.

You can consider using func websocketNetConn to rewire the context below instead of websocket.NetConn(...).

mafredri · 2022-08-30T11:27:35Z

cli/provisionerrun.go

+			}
+
+			errCh := make(chan error, 1)
+			provisionerDaemon, err := newProvisionerDaemon(ctx, client.ListenProvisionerDaemon, logger, cacheDir, errCh, useEchoProvisioner)


I'm not saying it has to happen here, but since the provisioner has a name, consider using filepath.Join(cacheDir, "provisionerd", name). Perhaps in newProvisionerDaemon.

This will allow multiple provisioners to run on the same machine without potentially breaking terraform init.

mafredri · 2022-08-30T11:28:36Z

coderd/httpmw/provisionerdaemon.go

+	return user
+}
+
+// ExtractWorkspaceAgent requires authentication using a valid provisioner token.


Suggested change

// ExtractWorkspaceAgent requires authentication using a valid provisioner token.

// ExtractProvisionerDaemon requires authentication using a valid provisioner token.

mafredri · 2022-08-30T11:31:55Z

coderd/httpmw/provisionerdaemon.go

+			}
+			token, err := uuid.Parse(cookie.Value)
+			if err != nil {
+				httpapi.Write(rw, http.StatusUnauthorized, codersdk.Response{


I'd consider this a bad request, but perhaps there's a reason it's unauthorized?

Suggested change

httpapi.Write(rw, http.StatusUnauthorized, codersdk.Response{

httpapi.Write(rw, http.StatusBadRequest, codersdk.Response{

This is mostly just for consistency with ExtractWorkspaceAgent, but I think a 401 error is reasonable here.

From a client's point of view, a token is an opaque string, and our implementation happens to generate tokens that look like UUIDs. Since we return a 401 error if the cookie is a valid UUID but isn't a token that exists in the DB, it makes sense to return the same error if it's not a valid UUID (and therefore can't be a valid token).

mafredri · 2022-08-30T11:34:37Z

coderd/httpmw/provisionerdaemon.go

+			if err != nil {
+				if errors.Is(err, sql.ErrNoRows) {
+					httpapi.Write(rw, http.StatusUnauthorized, codersdk.Response{
+						Message: "Provisioner token is invalid.",


This message is a bit misleading, token is a valid (format), but not registered or revoked. Simply saying Forbidden. could work.

Like my other comment, this is basically just mimicking the way we handle agent tokens. I think if we get a token that isn't equal to the auth_token of a valid provisioner, we should generate the same error message regardless of whether it happens to be formatted like a UUID.

But I'm totally open to changing the wording if there's a better way to describe that situation than "invalid".

In the context of auth, I think "invalid" encompasses both format and non-format problems with the credential. Wording here is fine IMO

@spikecurtis I agree from a security perspective (don't reveal too much), but from a usability perspective I think it could be more helpful to the user. But I'm fine with either or.

mafredri · 2022-08-30T11:38:36Z

codersdk/provisionerdaemons.go

+	}
+	conn, res, err := websocket.Dial(ctx, serverURL.String(), &websocket.DialOptions{
+		HTTPClient: httpClient,
+		// Need to disable compression to avoid a data-race.


This isn't your code, but I wonder if anyone knows what the data race is with compression enabled? 😄

mafredri · 2022-08-30T11:41:41Z

cli/provisionercreate.go

+				return err
+			}
+
+			if provisionerDaemon.AuthToken == nil {


Kind of related to my other question about nullability of the auth_token field, but why would this ever be allowed to happen during create?

Yeah, it shouldn't be possible. A null auth_token would indicate that the provisioner was incorrectly registered as "in-process" rather than "out-of-process". But if that does somehow happen, this is just a sanity check so that we generate a meaningful error message rather than panicking.

Could we make it so the API errors instead (and doesn't return a nullable UUID)? (I think that would be nicer for consumers in general.)

presleyp

Frontend ✅

Emyrk

Just a general question. If coderd or the provisionerd go down, do they reconnect?

Emyrk · 2022-08-30T17:03:31Z

coderd/provisionerdaemons.go

+}
+
+func (api *API) postProvisionerDaemon(rw http.ResponseWriter, r *http.Request) {
+	// Create the user on the site.


Incorrect comment

Suggested change

// Create the user on the site.

Emyrk · 2022-08-30T17:05:43Z

provisionerd/provisionerd.go

@@ -141,6 +141,9 @@ func (p *Server) connect(ctx context.Context) {
 		if p.isClosed() {
 			return
 		}
+
+		p.sendConnectRequest(ctx)


Can this fail? If it does, is it ok to just go to the for loop?

github-actions · 2022-09-07T00:11:44Z

This Pull Request is becoming stale. In order to minimize WIP, prevent merge conflicts and keep the tracker readable, I'm going close to this PR in 3 days if there isn't more activity.

github-actions · 2022-09-17T00:12:28Z

This Pull Request is becoming stale. In order to minimize WIP, prevent merge conflicts and keep the tracker readable, I'm going close to this PR in 3 days if there isn't more activity.

bpmct · 2022-09-19T19:28:57Z

Tried it and worked like a charm (both as a separate process and on a different machine). Some feedback/questions. Don't have to be addressed in this PR.

Wondering if we should show another message instead of Queued when there are no provisioners present. Maybe even the place in line could help?

codertester@coder-v2:/tmp/docker$ coder templates create
> Create and upload "/tmp/docker"? (yes/no) yes
⧗  Queued

Is there a way to "assign" a workspace to a provisioner daemon? Some use cases in mind

Specific provisioner/Docker host is used for workspaces
Template requires a specific provisioner daemon that has credentials

Can we add a coder provisioners ls?

github-actions · 2022-09-27T00:19:54Z

This Pull Request is becoming stale. In order to minimize WIP, prevent merge conflicts and keep the tracker readable, I'm going close to this PR in 3 days if there isn't more activity.

dwahler added 4 commits August 18, 2022 16:59

Revert "feature: disable provisionerd listen endpoint (#1612)"

97052e0

This reverts commit a03615a.

Add support for standalone provisioner daemons with token authentication

48fc97a

fix indentation

6067403

Merge remote-tracking branch 'origin/main' into dwahler/provisionerd

6100206

kylecarbs mentioned this pull request Aug 24, 2022

Provisionerd should register the running provisioners when it connects #1605

Closed

github-actions bot added the stale This issue is like stale bread. label Aug 26, 2022

dwahler added 4 commits August 26, 2022 00:17

Merge remote-tracking branch 'origin/main' into dwahler/provisionerd

940f458

Add tests for standalone provisionerd

a56220a

Check for successful provisioner connection in test

57926bb

fix protoc version

097ebc9

kylecarbs removed the stale This issue is like stale bread. label Aug 29, 2022

dwahler added 2 commits August 30, 2022 00:40

have provisionerd register its supported provisioners on connection

73f1e5c

Merge remote-tracking branch 'origin/main' into dwahler/provisionerd

e211ddb

dwahler marked this pull request as ready for review August 30, 2022 00:50

dwahler requested a review from a team as a code owner August 30, 2022 00:50

dwahler requested review from presleyp and a team and removed request for a team August 30, 2022 00:50

mafredri reviewed Aug 30, 2022

View reviewed changes

presleyp approved these changes Aug 30, 2022

View reviewed changes

Emyrk reviewed Aug 30, 2022

View reviewed changes

github-actions bot added the stale This issue is like stale bread. label Sep 7, 2022

dwahler removed the stale This issue is like stale bread. label Sep 9, 2022

github-actions bot added the stale This issue is like stale bread. label Sep 17, 2022

github-actions bot removed the stale This issue is like stale bread. label Sep 20, 2022

github-actions bot added the stale This issue is like stale bread. label Sep 27, 2022

github-actions bot closed this Sep 30, 2022

github-actions bot deleted the dwahler/provisionerd branch March 1, 2023 00:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Allow running standalone provisioner daemons #3563

feat: Allow running standalone provisioner daemons #3563

dwahler commented Aug 18, 2022 •

edited

Loading

github-actions bot commented Aug 26, 2022

mafredri left a comment

mafredri Aug 30, 2022

dwahler Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

dwahler Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

dwahler Aug 30, 2022

mafredri Aug 30, 2022

dwahler Aug 30, 2022

spikecurtis Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

mafredri Aug 30, 2022

dwahler Aug 30, 2022

mafredri Aug 30, 2022

presleyp left a comment

Emyrk left a comment

Emyrk Aug 30, 2022

Emyrk Aug 30, 2022

github-actions bot commented Sep 7, 2022

github-actions bot commented Sep 17, 2022

bpmct commented Sep 19, 2022

github-actions bot commented Sep 27, 2022

	ctx, cancelFunc := context.WithCancel(context.Background())
	ctx, cancelFunc := context.WithTimeout(context.Background(), testutil.WaitLong)

	// ExtractWorkspaceAgent requires authentication using a valid provisioner token.
	// ExtractProvisionerDaemon requires authentication using a valid provisioner token.

	httpapi.Write(rw, http.StatusUnauthorized, codersdk.Response{
	httpapi.Write(rw, http.StatusBadRequest, codersdk.Response{

feat: Allow running standalone provisioner daemons #3563

feat: Allow running standalone provisioner daemons #3563

Conversation

dwahler commented Aug 18, 2022 • edited Loading

Example usage

github-actions bot commented Aug 26, 2022

mafredri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

presleyp left a comment

Choose a reason for hiding this comment

Emyrk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2022

github-actions bot commented Sep 17, 2022

bpmct commented Sep 19, 2022

github-actions bot commented Sep 27, 2022

dwahler commented Aug 18, 2022 •

edited

Loading