feat: add provisioner daemon and jobs endpoints and commands #15940

mafredri · 2024-12-19T18:21:32Z

Fixes #15190
Fixes #15191
Updates #15084

johnstcn · 2024-12-20T15:23:09Z

cli/testdata/coder_provisioners_jobs_list.golden

+==========[version job ID]==========  ====[timestamp]=====  succeeded  map[owner: scope:organization]  template_version_import  Coder                
+======[workspace build job ID]======  ====[timestamp]=====  succeeded  map[owner: scope:organization]  workspace_build          Coder                


It would be nice if tags were listed in k1=v1,k2=v2 format.
We can also probably "lie" a bit about the case {"owner": "", "scope": "organization"} and just pretend it's empty.

Good suggestions, and I agree we should hide default owner/scope. We should probably update table formatter to change how maps are output.

I would move "Queue" column to be the first one.

I would move "Queue" column to be the first one.

@mtojek I'm going to create a follow-up issue for this as ordering columns is not supported in our table formatter currently. I would like to avoid us having to craft custom structs every time we want to re-order columns.

cli/provisionerjobs.go

Emyrk · 2024-12-20T16:08:08Z

cli/provisionerjobs.go

+func convertSlice[D, S ~string](dstType []D, src []S) []D {
+	for _, item := range src {
+		dstType = append(dstType, D(item))
+	}
+	return dstType
+}


Maybe throw this into slice as ToEnum 🤷‍♂️

Similar to ToStrings

coder/coderd/util/slice/slice.go

Lines 7 to 14 in dc70114

// ToStrings works for any type where the base type is a string.

func ToStrings[T ~string](a []T) []string {

tmp := make([]string, 0, len(a))

for _, v := range a {

tmp = append(tmp, string(v))

}

return tmp

}

Can do 👍🏻

Btw, any good reason these utils under coderd? I wouldn't move it in this PR but could make it more accessible.

no good reason at all. I think it just grew over time

Emyrk · 2024-12-20T16:11:36Z

codersdk/provisionerdaemons.go

+	Version        string            `json:"version" table:"version"`
+	APIVersion     string            `json:"api_version" table:"api version"`
+	Provisioners   []ProvisionerType `json:"provisioners" table:"-"`
+	Tags           map[string]string `json:"tags" table:"tags"`


I do not love this output: map[owner: scope:organization], but I'm ok with it.

We could have the table formatter know how to handle some basic types like map[string]string. Just leave it for this PR though.

I had a similar thought, something human-readable and copy/paste friendly:

owner = (none) scope = organization

Emyrk · 2024-12-20T16:13:52Z

enterprise/cli/provisionerdaemons.go

+	cmd := r.RootCmd.Provisioners()
+	cmd.AddSubcommands(
+		r.provisionerDaemonStart(),
+		r.provisionerKeys(),
+	)


This is a crafty

Emyrk · 2024-12-20T16:15:13Z

enterprise/cli/proxyserver.go

@@ -114,6 +114,8 @@ func (r *RootCmd) proxyServer() *serpent.Command {
 		),
 		Handler: func(inv *serpent.Invocation) error {
 			var closers closers
+			defer closers.Close()


Lol, closers was a nice thought, but never closing the closers is quite the funny bug.

var losers closers defer losers.Close()

:)

Emyrk · 2024-12-20T16:18:33Z

coderd/coderd.go

+				r.Route("/provisionerdaemons", func(r chi.Router) {
+					r.Get("/", api.provisionerDaemons)
+				})


If we add filtering by key, can we merge this with /organizations/{organization}/provisionerkeys/daemons? Or at least merge the code they both invoke? If possible, just to make sure we are consistent.

coder/enterprise/coderd/provisionerkeys.go

Line 129 in dc70114

func (api *API) provisionerKeyDaemons(rw http.ResponseWriter, r *http.Request) {

That's a good idea, I wasn't sure how much to move to AGPL vs enterprise but none of this code is critical to keep in enterprise 👍🏻

Generally I think GET endpoints can be made AGPL, and POST/PATCH should be enterprise.

mtojek · 2024-12-23T09:31:58Z

cli/cliui/table.go

@@ -195,6 +197,10 @@ func renderTable(out any, sort string, headers table.Row, filterColumns []string
 				if val != nil {
 					v = val.Format(time.RFC3339)
 				}
+			case codersdk.NullTime:


nit: shouldn't we extract it to a separate PR?

mtojek · 2024-12-23T09:32:50Z

cli/provisionerjobs.go

+
+			jobs, err := client.OrganizationProvisionerJobs(ctx, org.ID, &codersdk.OrganizationProvisionerJobsOptions{
+				Status: slice.ToStringEnums[codersdk.ProvisionerJobStatus](status),
+				Limit:  int(limit),


nit: paging?

I want to keep it simple for now, I realized we don't have a precedent on the CLI for paging so I didn't want to introduce it here. Granted --limit is also new, so I could remove that too but I was worried about how many entries there will be.

The original issue #15084 mentioned we should have "created at" (or rather "created after"), and I added that initially, but then I realized we don't have a precedent for filtering on time on the CLI either. (There's even a lack of serpent support there.) For this reason both pagination and created at filter have been omitted and should IMO be part of a grander plan for how to support these in a unified way on the CLI.

mtojek · 2024-12-23T09:42:42Z

cli/testdata/coder_provisioners_jobs_list.golden

+==========[version job ID]==========  ====[timestamp]=====  succeeded  map[owner: scope:organization]  template_version_import  Coder                
+======[workspace build job ID]======  ====[timestamp]=====  succeeded  map[owner: scope:organization]  workspace_build          Coder                


I would move "Queue" column to be the first one.

mtojek · 2024-12-23T09:44:43Z

coderd/database/dbtestutil/postgres.go

+	if err != nil {
+		return "", nil, xerrors.Errorf("open container: %w", err)
+	}


nit: double-checking, is it intentional?

Yes, in some cases this will cause a nil pointer dereference otherwise.

mtojek · 2024-12-23T09:48:53Z

coderd/provisionerdaemons.go

+// @Param tags query object false "Provisioner tags to filter by (JSON of the form {'tag1':'value1','tag2':'value2'})"
+// @Success 200 {array} codersdk.ProvisionerDaemonWithStatus
+// @Router /organizations/{organization}/provisionerdaemons [get]
+func (api *API) provisionerDaemons(rw http.ResponseWriter, r *http.Request) {


Is there a reason why endpoint changes and CLI go together? Alternatively, you could split the PR in half, and keep all golden files aside. Large PRs are typically a problem when somebody needs to revert them.

Yes and no, for development I'm working on them in tandem, CLI requirements have and keep resulted in API changes, etc. Stacking could be a better approach though.

I don't think the golden files should be separate as that would make the feature harder to revert, but API and CLI can definitely be.

It should be possible to have the DB stuff in its own PR and the CLI stuff then in a separate PR based off that. No need to have a really deep stack here IMO.

I'm fine with reviewing initially here "all-in-one".

mtojek · 2024-12-23T09:51:32Z

codersdk/provisionerdaemons.go

+	Version        string            `json:"version" table:"version"`
+	APIVersion     string            `json:"api_version" table:"api version"`
+	Provisioners   []ProvisionerType `json:"provisioners" table:"-"`
+	Tags           map[string]string `json:"tags" table:"tags"`


I had a similar thought, something human-readable and copy/paste friendly:

owner = (none) scope = organization

mtojek · 2024-12-23T09:53:55Z

docs/manifest.json

-							"path": "reference/cli/provisioner_keys.md"
+							"path": "reference/cli/provisioners_keys.md"


nit: renaming could go in a separate PR, right?

mtojek · 2024-12-23T09:55:05Z

enterprise/cli/proxyserver.go

@@ -114,6 +114,8 @@ func (r *RootCmd) proxyServer() *serpent.Command {
 		),
 		Handler: func(inv *serpent.Invocation) error {
 			var closers closers
+			defer closers.Close()


var losers closers defer losers.Close()

:)

mtojek · 2024-12-23T09:57:34Z

enterprise/coderd/coderd.go

@@ -377,7 +377,7 @@ func New(ctx context.Context, options *Options) (_ *API, err error) {
 		//
 		// We may in future decide to scope provisioner daemons to organizations, so we'll keep the API
 		// route as is.
-		r.Route("/organizations/{organization}/provisionerdaemons", func(r chi.Router) {
+		r.Route("/organizations/{organization}/provisionerdaemons/serve", func(r chi.Router) {


nit: does this change give any special benefits? Most likely I'm missing something...

We removed the other endpoints from here that were under /organizations/{organization}/provisionerdaemons and only /organizations/{organization}/provisionerdaemons/serve remains. There's no reason to keep the broader one and IIRC it also interfered with the AGPL endpoint.

johnstcn · 2024-12-23T11:07:09Z

coderd/provisionerdaemons.go

+	if err != nil {
+		httpapi.Write(ctx, rw, http.StatusInternalServerError, codersdk.Response{
+			Message: "Internal error fetching provisioner daemons.",
+			Detail:  err.Error(),
+		})
+		return
+	}


nit: does sql.ErrNoRows need to be handled separately?

Co-authored-by: Steven Masley <[email protected]>

Closes #15191 Updates #15084 Supercedes #15940

Closes #15190 Updates #15084 Supercedes #15940

Closes #15191 Updates #15084 Supercedes #15940

mafredri · 2025-01-13T10:04:44Z

Closing this as it's split up into #16028, #16029 and #16030 (and a few other small PRs that fixed unrelated things).

github-actions bot assigned mafredri Dec 19, 2024

mafredri marked this pull request as draft December 19, 2024 18:21

mafredri force-pushed the mafredri/feat-provisioner-and-jobs-list branch from b776b1b to af6789a Compare December 20, 2024 15:05

johnstcn reviewed Dec 20, 2024

View reviewed changes

Emyrk reviewed Dec 20, 2024

View reviewed changes

mtojek reviewed Dec 23, 2024

View reviewed changes

johnstcn reviewed Dec 23, 2024

View reviewed changes

github-actions bot added the stale This issue is like stale bread. label Dec 31, 2024

mafredri force-pushed the mafredri/feat-provisioner-and-jobs-list branch 5 times, most recently from 5b54ed6 to 720b715 Compare December 31, 2024 13:11

This was referenced Dec 31, 2024

fix(enterprise/cli): add missing defer close of closer funcs #15986

Merged

test: fix incorrectly placed cleanup defer in dbtestutil #15987

Merged

github-actions bot removed the stale This issue is like stale bread. label Jan 1, 2025

mafredri and others added 14 commits January 2, 2025 10:26

feat: add provisioners and jobs endspoints and CLI commands

650cbaf

wip

449df5d

key

fa261b1

golden

c5a0429

name

b138faf

gen

a0af6eb

test

8975d41

dbmem

96a278e

typo

e9b8725

fix migrations

c5211ef

cleanup

c6e1cc6

improve tests and dbmem

9e64cdd

Update cli/provisionerjobs.go

8fd1a6a

Co-authored-by: Steven Masley <[email protected]>

lint

b57bc8c

mafredri added 7 commits January 2, 2025 10:26

remove unused

32fde66

gen, remove unused

38116d7

slice.ToStringEnums

e7ac032

lint & gen

f6c2a3f

dbmem/golden

876bd80

revert

0788d56

undo rename

24d49da

mafredri force-pushed the mafredri/feat-provisioner-and-jobs-list branch from 720b715 to 24d49da Compare January 2, 2025 10:27

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

5edf87e

Closes #15191 Updates #15084 Supercedes #15940

This was referenced Jan 3, 2025

feat(coderd): add endpoint to list provisioner daemons #16028

Merged

feat(coderd): add endpoint to list provisioner jobs #16029

Merged

feat(cli): add provisioner list and provisioner jobs list #16030

Merged

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(coderd): add endpoint to list provisioner jobs

664d9b7

Closes #15190 Updates #15084 Supercedes #15940

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

90a2e64

Closes #15191 Updates #15084 Supercedes #15940

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

90f19d1

Closes #15191 Updates #15084 Supercedes #15940

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

180bf45

Closes #15191 Updates #15084 Supercedes #15940

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

4dcb7cb

Closes #15191 Updates #15084 Supercedes #15940

mafredri added a commit that referenced this pull request Jan 3, 2025

feat(cli): add provisioner list and provisioner jobs list

10475cc

Closes #15191 Updates #15084 Supercedes #15940

mafredri mentioned this pull request Jan 7, 2025

Add column ordering support to cliui table #16055

Closed

github-actions bot added the stale This issue is like stale bread. label Jan 10, 2025

github-actions bot closed this Jan 13, 2025

johnstcn reopened this Jan 13, 2025

johnstcn removed the stale This issue is like stale bread. label Jan 13, 2025

mafredri closed this Jan 13, 2025

github-actions bot locked and limited conversation to collaborators Jan 13, 2025

mafredri deleted the mafredri/feat-provisioner-and-jobs-list branch January 13, 2025 10:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add provisioner daemon and jobs endpoints and commands #15940

feat: add provisioner daemon and jobs endpoints and commands #15940

mafredri commented Dec 19, 2024

johnstcn Dec 20, 2024 •

edited

Loading

mafredri Dec 20, 2024

mtojek Dec 23, 2024

mafredri Dec 23, 2024

Emyrk Dec 20, 2024

mafredri Dec 20, 2024

mafredri Dec 20, 2024

Emyrk Dec 20, 2024

Emyrk Dec 20, 2024

mtojek Dec 23, 2024

Emyrk Dec 20, 2024

Emyrk Dec 20, 2024 •

edited

Loading

mtojek Dec 23, 2024

Emyrk Dec 20, 2024

mafredri Dec 20, 2024

Emyrk Dec 20, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mafredri Dec 23, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mafredri Dec 23, 2024

mtojek Dec 23, 2024

mafredri Dec 23, 2024 •

edited

Loading

johnstcn Dec 23, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mtojek Dec 23, 2024

mafredri Dec 23, 2024

johnstcn Dec 23, 2024

mafredri commented Jan 13, 2025 •

edited

Loading

		==========[version job ID]========== ====[timestamp]===== succeeded map[owner: scope:organization] template_version_import Coder
		======[workspace build job ID]====== ====[timestamp]===== succeeded map[owner: scope:organization] workspace_build Coder

	// ToStrings works for any type where the base type is a string.
	func ToStrings[T ~string](a []T) []string {
	tmp := make([]string, 0, len(a))
	for _, v := range a {
	tmp = append(tmp, string(v))
	}
	return tmp
	}

		"path": "reference/cli/provisioner_keys.md"
		"path": "reference/cli/provisioners_keys.md"

feat: add provisioner daemon and jobs endpoints and commands #15940

feat: add provisioner daemon and jobs endpoints and commands #15940

Conversation

mafredri commented Dec 19, 2024

johnstcn Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Emyrk Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri commented Jan 13, 2025 • edited Loading

johnstcn Dec 20, 2024 •

edited

Loading

Emyrk Dec 20, 2024 •

edited

Loading

mafredri Dec 23, 2024 •

edited

Loading

mafredri commented Jan 13, 2025 •

edited

Loading