Add `api` command for direct API access #909

mislav · 2020-05-12T17:39:22Z

Based on a subset of hub api features:

authenticates every request with the user's token;
exposes both GitHub REST (v3) and GraphQL (v4) APIs;
offers JSON serialization via parameter flags;
allows overriding request method;
allows adding request headers;
allows printing response headers.

$ gh help api
Makes an authenticated HTTP request to the GitHub API and prints the response.

The <endpoint> argument should either be a path of a GitHub API v3 endpoint, or
"graphql" to access the GitHub API v4.

The default HTTP request method is "GET" normally and "POST" if any parameters
were added. Override the method with '--method'.

Pass one or more '--raw-field' values in "<key>=<value>" format to add
JSON-encoded string parameters to the POST body.

The '--field' flag behaves like '--raw-field' with magic type conversion based
on the format of the value:

- literal values "true", "false", "null", and integer numbers get converted to
  appropriate JSON types;
- if the value starts with "@", the rest of the value is interpreted as a
  filename to read the value from. Pass "-" to read from standard input.

Usage:
  gh api <endpoint> [flags]

Flags:
  -F, --field stringArray       Add a parameter of inferred type
  -H, --header stringArray      Add an additional HTTP request header
  -i, --include                 Include HTTP response headers in the output
  -X, --method string           The HTTP method for the request (default "GET")
  -f, --raw-field stringArray   Add a string parameter

The -X, -F, -H, and -i flags were originally largely modelled after curl, since that's a tool that a lot of command-line users already have experience with. The gh api command is basically curl for GitHub.

Examples:

# print information about the authenticating user
$ gh api user

# pipe the output to `jq` (installed separately) to extract the "name" field
$ gh api user | jq .name

# comment on an issue in `cli/cli`
# note: this is a POST request since we've added parameters
$ gh api repos/cli/cli/issues/909/comments -f body='hello from cli!'

# mark a pull request as ready-for-review
$ gh api -X PATCH repos/cli/cli/pulls/909 -F draft=false

# make a GraphQL request by using the query read from file
$ gh api graphql -F query=@path/to/myquery.graphql

TODO:

Ref. #332

mislav · 2020-05-14T16:40:01Z

Added some features and started to experiment with incorporating ideas from #759 and #770 when it comes to the api command:

there is a factory method to make the command instance: NewCmdApi()
flag parsing logic is separate (and therefore testable separately) from the rest of api implementation
the new command lives under its own pkg/cmd/api/ package, which allows it to be split into multiple files and resists adding more and more coupling to the already giant command package.

vilmibm · 2020-05-15T18:26:12Z

I'm excited about all of this!

I think you showed me that custom output format that made json a little easier to grep (like, line mode or something?); are you considering bringing that over? I've been excited about that.

mislav · 2020-05-20T13:19:58Z

@vilmibm So hub supports the -t, --flat flag to output a line-based, tab-delimited format suitable for shell scripts:

$ hub api -t graphql -f query='{viewer{login,name}}'
.data.viewer.login      mislav
.data.viewer.name       Mislav Marohnić

I didn't port this over to here yet because even though the format works well for me, it was invented by me, it's not based on anything that exists elsewhere, I don't have any confirmation that others are benefitting from it, and to drill down to extract values from specific fields you still need to use regex:

hub api -t ... | awk -F'\t' '/\.name\t/ { print $2 }'

So I propose to address alternative formats in follow-up. Something that I'd like to explore is support a jq-like query filter that would allow you to extract specific fields without too much shell hackery ✨

mislav · 2020-05-20T14:02:13Z

I've wrapped this up as something shippable, so this is now ready for review. I plan to address advanced topics (e.g. custom input/output formats) in follow-up PRs.

I've used this branch as a playground to experiment with how we might organize commands differently. The new api command lives under pkg/cmd/api/, with new commands to be added in this package pattern: pkg/cmd/<command>/<subcommand>/. The main executable with its "root" command would import the individual packages and nest them under the gh command.

I like this design for these reasons:

It follows the loose principle of the Go community to tend to keep packages small. I feel that smaller packages = better isolation between interfaces = more thoughtful organization of code.
The implementation for each command can be split into multiple *.go files if needed. This would be very useful for gigantic commands such as pr create.
Absolutely no package-global state: no vars, no overridable functions, no init() function. The "context" for every command, e.g. information about stdin/stdout streams or a pre-configured HTTP client, is passed via a cmdutil.Factory at the time of command initialization. That allows tests to pass in stubs instead of having to stub things on package level.
The new cli/cli/pkg/cmd/api command package does not depend on absolutely any code from either cli/cli/command or cli/cli/api packages.
A new cmdutil package is a place for shared helpers used by individual command packages.
The new IOStreams object will allow tests to simulate stdin/stdout being connected to a terminal, faking terminal width/hight, having color disabled/enabled, and so on which we so far weren't able to simulate in tests. Avoid printing header if piped to a script #796
Command-line flags are now parsed to a single ApiOptions struct instead to individual variables which previously led to a lot of boilerplate around reading flag values.
Command flag parsing and its implementation can be tested in isolation from each other.

Drawbacks:

There will be more boilerplate when setting up each new command in this manner.
I haven't figured out a transition plan to convert old command/*.go commands to individual pkg/cmd/*. However, a gradual transition method should be possible. If this gets merged to master, we can figure out how to port some old commands together, for example gh repo commands would be a good candidate.

vilmibm

I'm ✔️ on the command itself; I like this approach for gh api and don't have any misgivings about it.

I do want us to be careful and thoughtful about adopting this new style for commands. I'm curious about @probablycorey 's opinion as well as the answers to the questions I left.

vilmibm · 2020-05-20T16:04:18Z

pkg/cmd/api/api.go

+	HttpClient func() (*http.Client, error)
+}
+
+func NewCmdApi(f *cmdutil.Factory, runF func(*ApiOptions) error) *cobra.Command {


It's true that this is "more" boilerplate, but I much prefer it to the boilerplate we have now. I find this a lot easier to comprehend and reproduce than our current approach of top-level stuff and init()ing flags, which often confuses me when I'm adding a new command.

vilmibm · 2020-05-20T16:26:10Z

pkg/cmd/api/api_test.go

+			})
+
+			argv, err := shlex.Split(tt.cli)
+			assert.NoError(t, err)


I've been opposed to adding testify but I'm begrudgingly ok with it seeing that it can be used in this light footprint, piecemeal fashion.

vilmibm · 2020-05-20T16:45:38Z

command/root.go


-func (fe FlagError) Unwrap() error {
-	return fe.Err
+	// TODO: iron out how a factory incorporates context


This seems important. I'm on board with this new approach to structuring our commands but dealing with context is an important part of almost every command. Do you have ideas or sketches about what this might look like? Without a clear plan it's going to be hard to motivate ourselves to apply this pattern to existing commands.

It is indeed important, thanks for bringing it up. The api command was not really appropriate to explore more aspects of reading context in because it's so isolated, i.e. it doesn't need anything from the context other than the API token, and that part is covered. I agree that we should have a clear migration plan for other commands and, after merging this, what I plan to do is migrate a single older command over to the new approach and have that migration serve as a guidebook on how to apply this pattern.

vilmibm · 2020-05-20T16:50:47Z

pkg/cmd/api/api.go

+  filename to read the value from. Pass "-" to read from standard input.
+`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(c *cobra.Command, args []string) error {


I like this clean separation of RunE being flag/argument processing and then the actual command logic going in its own function. It feels sustainable to me.

probablycorey

The new api command lives under pkg/cmd/api/, with new commands to be added in this package pattern: pkg/cmd///. The main executable with its "root" command would import the individual packages and nest them under the gh command.

I'm 👍 on this idea! I've been thinking we need it for more isolation and to make it easier to understand where a specific command code lives

The API stuff is super powerful, I really like the interface you've come up with. Its hard for me to say too much about it until I get to use it a bit more, but my first reaction is very cool!

probablycorey · 2020-05-22T16:09:16Z

command/issue.go

-			return FlagError{errors.New("issue number or URL required as argument")}
-		}
-		return nil
-	},


probablycorey · 2020-05-22T16:20:28Z

command/root.go

+			return httpClient(token), nil
+		},
+	}
+	RootCmd.AddCommand(apiCmd.NewCmdApi(cmdFactory, nil))


I'm assuming this would be the one place where we put all the top level commands? I think this will make it much more clear where commands are added. I like it.

probablycorey · 2020-05-22T16:20:55Z

pkg/cmd/api/api.go

+	cmd.Flags().StringArrayVarP(&opts.RequestHeaders, "header", "H", nil, "Add an additional HTTP request header")
+	cmd.Flags().BoolVarP(&opts.ShowResponseHeaders, "include", "i", false, "Include HTTP response headers in the output")
+	return cmd
+}


I love how each command can now have a dedicated function to set itself up 💯

probablycorey · 2020-05-22T16:47:34Z

pkg/cmd/api/api.go

+	method := opts.RequestMethod
+	if len(params) > 0 && !opts.RequestMethodPassed {
+		method = "POST"
+	}


It seems a little weird that opts has RequestMethod and RequestMethodPassed. Moving this logic into the RunE function and assuming RequestMethod is always set correctly is also a little weird, but I think it might be easier to understand.

That can also work! I can swing both ways with this approach in particular; I just wanted to set some precedent to communicating via opts struct that a certain flag has been passed or not, since we use that information in some other commands as well. I agree that since NewCmdApi() is only concerned with setting up the cobra.Command object and defining flag parsing, it should also handle as much of the flag processing logic as it can before dispatching to the apiRun() command.

bradennapier · 2020-08-01T08:15:10Z

Not being able to do a massive number of api calls really killed the vibe on this feature...

Cant even add a label to an issue since it requires an array ?

zhigang1992 · 2020-08-02T00:38:00Z

@bradennapier #921 (comment)

mislav · 2020-08-04T09:24:08Z

@bradennapier We will be adding array support in a future release. In the meantime:

echo '{"labels":["foo", "bar"]}' | gh api repos/:owner/:repo/issues/123/labels --input -

bradennapier · 2020-08-04T19:43:06Z

Yep, thanks! Figured it out.

niallpaterson · 2020-08-24T21:27:49Z

hello from cli!

mislav mentioned this pull request May 12, 2020

Direct access to GitHub API via gh #332

Closed

This was referenced May 15, 2020

Support gh deployment create #921

Closed

Change the default branch from "master" to something else #929

Closed

mislav added 9 commits May 20, 2020 15:21

Add api command

1609afe

Serialize GraphQL parameters under variables

fa3e25b

Promote api command to a pkg/cmd/api package

90fa193

Allow setting multiple values for a request header

7ffbde3

Extract parseFields to a func

a7100b1

Add iostreams package

4c762d5

Extract cmdutil package

d8146cd

Add api tests

f58e0bf

Make NewCmdApi testable

bef62fa

mislav force-pushed the api-command branch from 2d83a79 to bef62fa Compare May 20, 2020 13:22

mislav marked this pull request as ready for review May 20, 2020 13:36

mislav requested review from probablycorey and vilmibm May 20, 2020 13:36

mislav added 2 commits May 20, 2020 16:28

Add test for showing response headers

292b428

Ensure that cobra command tests don't write to system stdout/stderr

ea3a55c

mislav mentioned this pull request May 20, 2020

Respect GITHUB_TOKEN environment variable #976

Merged

1 task

vilmibm reviewed May 20, 2020

View reviewed changes

probablycorey approved these changes May 22, 2020

View reviewed changes

mislav mentioned this pull request May 27, 2020

List and filter issues from all repos #974

Closed

mislav added 2 commits May 27, 2020 12:56

Merge remote-tracking branch 'origin/master' into api-command

2e93a06

Respect GITHUB_TOKEN in api command

13ba0aa

mislav merged commit 658d548 into master May 27, 2020

mislav deleted the api-command branch May 27, 2020 11:14

danielhelfand mentioned this pull request Jun 2, 2020

Restructure commands in isolated units #770

Closed

mislav mentioned this pull request Jun 3, 2020

gh api: fix passing file/stdin contents via field arguments #1093

Merged

tamalsaha mentioned this pull request Jun 4, 2020

Install GitHUB CLI on Ubuntu actions/runner-images#851

Merged

4 tasks

Add api command for direct API access #909

Add api command for direct API access #909

Uh oh!

Conversation

mislav commented May 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mislav commented May 14, 2020

Uh oh!

vilmibm commented May 15, 2020

Uh oh!

mislav commented May 20, 2020

Uh oh!

mislav commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vilmibm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mislav May 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

probablycorey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bradennapier commented Aug 1, 2020

Uh oh!

zhigang1992 commented Aug 2, 2020

Uh oh!

mislav commented Aug 4, 2020

Uh oh!

bradennapier commented Aug 4, 2020

Uh oh!

niallpaterson commented Aug 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Add `api` command for direct API access #909

Add `api` command for direct API access #909

mislav commented May 12, 2020 •

edited

Loading

mislav commented May 20, 2020 •

edited

Loading

mislav May 27, 2020 •

edited

Loading