feat: add cli first class validation #8374

Emyrk · 2023-07-07T14:41:52Z

What this does

A standard way to validate cli flags.

Problem

Validating cli flags tends to be in the cli handler body. This just adds some extra if !valid { return err } type code blocks that add length to the handlers. We also do if flag == "" { return xerrors.Errorf("<flag> must be specified")} errors.

These validation & required errors are not standardized. So the messaging & display of these errors really depends on how much care the developer put in at the time.

This PR aims to standardize how we handle cli flag validation and "require" such that all returned errors will be standard and ideally all errors are returned in a group if possible (this needs to get better on the cmd handling).

Future work

Make the returned standard errors pretty.

Example

An example of requiring a value be set, and adding validation to said value.

clibase.Option{
	Name:        "Coderd (Primary) Access URL",
	Description: "URL to communicate with coderd. This should match the access URL of the Coder deployment.",
	Flag:        "primary-access-url",
	Env:         "CODER_PRIMARY_ACCESS_URL",
	YAML:        "primaryAccessURL",
	// Makes the user provide a value for this option.
	Required:    true,
	// Standard way to validate the input values from the user.
	Value: clibase.Validate(&primaryAccessURL, func(value *clibase.URL) error {
		if !(value.Scheme == "http" || value.Scheme == "https") {
			return xerrors.Errorf("'--primary-access-url' value must be http or https: url=%s", primaryAccessURL.String())
		}
		return nil
	}),
	Group:  &externalProxyOptionGroup,
	Hidden: false,
},

Before

After

ammario · 2023-07-07T16:58:28Z

cli/clibase/option.go

@@ -23,6 +23,10 @@ const (
 type Option struct {
 	Name        string `json:"name,omitempty"`
 	Description string `json:"description,omitempty"`
+	// Required means this value must be set by some means. It requires
+	// `ValueSource != ValueSourceNone`
+	// If `Default` is set, then `Required` is ignored.


Why not make Required a composable Validation rule instead?

It cannot be a validation rule because the validation rule is only called on Set(). If the user never provides a value, and no default is set, then Set() is never called.

So it has to happen in the flag parsing logic.

ammario · 2023-07-07T17:00:10Z

cli/clibase/values.go

@@ -16,6 +16,49 @@ import (
 	"gopkg.in/yaml.v3"
 )

+// Validator is a wrapper around a pflag.Value that allows for validation
+// of the value after or before it has been set.


I like that this implements pflag.Value.

Maybe there's a function Required[T any]() ValidateFunc[T]

Maybe there's a function Required[T any]() ValidateFunc[T]

I was trying, but unfortunately Validate() can only validate inputs. If there is no input, Set() is never called for this to trigger on.

The only way I could think of making it work is have some post-processing step for the option? It felt easier to make it a flag on the Options and just touch it at the flag parsing.

I think it's a common enough validation requirement that making it apart of the Option field is justified.

Agreed. The downside is that if Default is set, then Required is ignored. Maybe we should throw an error somewhere if you configure an option like that.

enterprise/cli/proxyserver.go

ammario

I like the improvement and idea. Well done!

mafredri

I love how nicely you plugged in this feature without much extra code.

I did notice a slight inconsistency though:

Required: true

Required: false

With required flags, the actual error is lost and the message becomes incorrect (the value isn't missing, just didn't pass validation).

(I simply played around a bit by modifying an option in config-ssh, picked randomly)

		{
			Flag:          "ssh-option",
			FlagShorthand: "o",
			Env:           "CODER_SSH_CONFIG_OPTS",
			Description:   "Specifies additional SSH options to embed in each host stanza.",
			Required:      true,
			Value: clibase.Validate(clibase.StringArrayOf(&sshConfigOpts.sshOptions), func(value *clibase.StringArray) error {
				for _, v := range *value {
					if !strings.Contains(v, " ") {
						return xerrors.Errorf("ssh option %q must contain a space", v)
					}
				}
				return nil
			}),
		},

enterprise/cli/proxyserver.go

mafredri · 2023-07-10T20:36:14Z

cli/clibase/values.go

+	validate func(T) error
+}
+
+func Validate[T pflag.Value](opt T, validate ...func(value T) error) *Validator[T] {


What's the use-case for variadic validate funcs? Do we foresee passing multiple funcs? Typically I'm all for it, but with generics it seems to hurt VSCode autocomplete-ability (autocompletes to []func 😒).

Eww on the autocomplete.

The only rational was a comment by @johnstcn that it might make sense to create building block style validators that you could reuse. But I can put it back to a regular, and then we can create a CombineValidator or something that allows combining.

I had no strong opinions either way.

Huh my autocomplete works fine in Goland. Still we can make the varadic some helper func. Or make it a variadic when we need that

Emyrk · 2023-07-11T12:55:55Z

{
			Flag:          "ssh-option",
			FlagShorthand: "o",
			Env:           "CODER_SSH_CONFIG_OPTS",
			Description:   "Specifies additional SSH options to embed in each host stanza.",
			Required:      true,
			Value: clibase.Validate(clibase.StringArrayOf(&sshConfigOpts.sshOptions), func(value *clibase.StringArray) error {
				for _, v := range *value {
					if !strings.Contains(v, " ") {
						return xerrors.Errorf("ssh option %q must contain a space", v)
					}
				}
				return nil
			}),
		},

This is a really good catch thanks. I moved the Required check too far up the command run stack.

After this PR I think some of that code should be refactored to accumulate all errors and make nicer error messages.

$ coder config-ssh -o test                             
parsing flags ([config-ssh -o test]) for "coder config-ssh": invalid argument "test" for "-o, --ssh-option" flag: ssh option "test" must contain a space

Emyrk · 2023-07-11T13:00:16Z

Added a unit test for the invalid case with a required field

feat: add cli first class validation

9572e90

github-actions bot assigned Emyrk Jul 7, 2023

feat: add required flag to cli options

1681c99

Emyrk requested a review from ammario July 7, 2023 15:09

Emyrk added 3 commits July 7, 2023 11:17

Make gen

262ecf9

rename to just validate

b1094fc

Make function a varadic

5c571dd

ammario reviewed Jul 7, 2023

View reviewed changes

Emyrk added 2 commits July 10, 2023 10:29

Formatting

6d2079d

Add unit test

30fee84

Emyrk requested a review from mafredri July 10, 2023 16:58

Remove golangci lint change

b84238e

mafredri approved these changes Jul 10, 2023

View reviewed changes

Add unit test to catch invalid and missing flag

c1486ac

Emyrk merged commit bc102d6 into main Jul 11, 2023

Emyrk deleted the stevenmasley/validate branch July 11, 2023 13:59

github-actions bot locked and limited conversation to collaborators Jul 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add cli first class validation #8374

feat: add cli first class validation #8374

Uh oh!

Emyrk commented Jul 7, 2023 •

edited

Loading

Uh oh!

ammario Jul 7, 2023

Uh oh!

Emyrk Jul 7, 2023

Uh oh!

ammario Jul 7, 2023

Uh oh!

ammario Jul 7, 2023

Uh oh!

Emyrk Jul 7, 2023

Uh oh!

ammario Jul 7, 2023

Uh oh!

Emyrk Jul 10, 2023

Uh oh!

Uh oh!

ammario left a comment

Uh oh!

mafredri left a comment

Uh oh!

Uh oh!

mafredri Jul 10, 2023

Uh oh!

Emyrk Jul 11, 2023

Uh oh!

Emyrk Jul 11, 2023

Uh oh!

Emyrk commented Jul 11, 2023

Uh oh!

Emyrk commented Jul 11, 2023

Uh oh!

Uh oh!

feat: add cli first class validation #8374

feat: add cli first class validation #8374

Uh oh!

Conversation

Emyrk commented Jul 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Problem

Future work

Example

Before

After

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ammario left a comment

Choose a reason for hiding this comment

Uh oh!

mafredri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Emyrk commented Jul 11, 2023

Uh oh!

Emyrk commented Jul 11, 2023

Uh oh!

Uh oh!

Emyrk commented Jul 7, 2023 •

edited

Loading