Codecov Report

Merging #941 (04db053) into main (19b4323) will decrease coverage by 4.66%.
The diff coverage is n/a.

❗ Current head 04db053 differs from pull request most recent head e234374. Consider uploading reports for the commit e234374 to get more accurate results

@@            Coverage Diff             @@
##             main     #941      +/-   ##
==========================================
- Coverage   66.22%   61.56%   -4.67%     
==========================================
  Files         240      107     -133     
  Lines       14408     1137   -13271     
  Branches      115      115              
==========================================
- Hits         9542      700    -8842     
+ Misses       3904      412    -3492     
+ Partials      962       25     -937     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19b4323...e234374. Read the comment docs.

Agree on a standard data field

I also know that returning a top level json array is a security issue - http://haacked.com/archive/2009/06/25/json-hijacking.aspx/ , and this would make sure that the caller never has to know about or consider this risk.

That article appears to call out security vulnerabilities that happened in 2009, but I'd be hesitant to say anything exists like this today. GitHub, Google, GitLab and more return top-level JSON arrays and have for years: https://docs.github.com/en/rest/reference/repos

How is httpapi.Response a unified pattern if most routes (anything that sends data) not use it? It seems we have a cognitive split when returning responses by using two different packages.

If we want to continue to just return data types directly I think we need to rename httpapi.Response to be more specific to errors since that's really its only value.

If we value status codes, I think we should not be returning 200s with messages like "I did the thing" because it's useless info. We should just send back an empty 200 or the updated data types.

Every route does use it, just depends on the status code being responded. Every error gets httpapi.Response returned right now.

httpapi.Response is intentionally not named error, because the status code represents whether it was an error or not. We can remove the message that acknowledges a success message, I agree that's useless info. It's primarily there to avoid sending back an empty object or no payload at all, because we'd be breaking the API expectation that our responses are JSON.

I've seen quite a few APIs, mostly from clouds, that do something similar. By doing this pattern, our clients can always expect the same format of response for success messages, errors, data, etc. I've always felt it was easier to work with as a consumer of the JSON API.

@f0ssel The cloud's have enormous APIs that demand peculiar patterns. The GCP compute API comes to mind. They have to rely on patterns like the one you proposed to scale to thousands of API endpoints in a consistent way. These abstractions are necessary but I don't enjoy interacting with these APIs.

Can you provide examples of smaller, more Coder-like APIs that use something like your proposed pattern? I haven't ever worked with an API where the meat of every response was nested under a specific field. I assume most APIs avoid this nesting to save developers from one level of destructuring on every response parse.

I do see the value of standard message and errors fields. Perhaps the httpapi.Response could be genericized and extended to other response types if we want to enforce those standards while keeping the structure as flat as possible. We'd have to override MarshalJSON too but this should be simple.

I actually don't care that much about the json format, but I think the real problem I have is we just have a weird mix of "generic" functions that can only be used sometimes right now. We use httpapi.Write for error and "empty " responses, but not actual data. If httpapi had a clear public api that handled the different types of responses I'd be happy.

Maybe we just need httpapi.Write to take an interface{}, I think id be happy today with that.

@kylecarbs what about an api like:

• httpapi.Write(wr, code int, body interface{})
• httpapi.Response to httpapi.Error
• we get rid of the "empty" success message responses in favor of passing nil or actually returning the updated data.

That seems like a pattern we could all follow implicitly I think.

I think httpapi.Write(rw, code, body) seems reasonable. I'm hesitant to rename httpapi.Response to httpapi.Error, because I think it's fine to be used for success messages too! It's really a generic response, and the status code indicates whether the message is an error or not.

If it's called httpapi.Error, we have two separate indicators in the payload that indicate error, which seems unnecessary. I'd be happy if you made the mass change in this PR for httpapi.Write. I think we should still use render, just so we don't have to write marshal code ourselves (even though it's simple if it works it works).

I missed your point about always sending valid JSON instead of empty responses. I'll concede that does seem important to keep standard.

You found an interesting nit where the existing httpapi.Write didn't use render. We don't have to, but I agree it should cerrrrrrtainly be the same!

Ahh up to you whether we use render or not actually. I see we'd have to pass the r object to every write, which seems a bit excessive. Feel free to copy/paste from render in that case if you'd like, or we can keep the previous write code and I still think this is a substantial improvement.

Can we just return http.StatusNoContent instead of response messages?

@coadler I'm game in theory but I do feel like always returning JSON can't hurt. Also I do think status codes are important. What if we added something like:

func OK() Response {
	return Response{
		Message: "OK",
	}
}

Usage:

httpapi.Write(rw, http.StatusOK, httpapi.OK())

I'd rather us just write something a lil friendly in the API. Another benefit of the message not being static is us dev-folk will never check the message content, and will always check the status code 😎

It seems a bit wasteful to me in a SaaS context to send back a bunch of data a user will never consume, but I don't feel super strongly. I really like how Discord uses http 204, which is why I bring it up.

Ahh if Discord uses it, then that's good external precedent beyond my own judgment. I'm fine with doing that in cases where no response is needed. Sending fewer bytes is certainly the hackery thing to do 🤓!

Good catches on the responses that weren't even being checked properly (OK instead of created)! 🧹

I also know that returning a top level json array is a security issue - http://haacked.com/archive/2009/06/25/json-hijacking.aspx/ , and this would make sure that the caller never has to know about or consider this risk.

Also regarding this, I typically don't like returning raw JSON arrays because it's always a breaking change if we wanted to add extra fields, such as paging info. In V1 we always wrap array responses.

I also know that returning a top level json array is a security issue - http://haacked.com/archive/2009/06/25/json-hijacking.aspx/ , and this would make sure that the caller never has to know about or consider this risk.

Also regarding this, I typically don't like returning raw JSON arrays because it's always a breaking change if we wanted to add extra fields, such as paging info. In V1 we always wrap array responses.

This sounds like a good case for a custom lint rule for not passing arrays to httpapi.Write

I think we should make that change if the time comes. Considering many APIs haven't needed to do that (eg. GitHub), I'm weary to implement it on a hunch.

I think we should make that change if the time comes. Considering many APIs haven't needed to do that (eg. GitHub), I'm weary to implement it on a hunch.

I just checked and you are right, GitHub doesn't seem to mind sending back top level arrays. Good enough for me.

@kylecarbs do you know what may have changed to make this happen? https://github.com/coder/coder/runs/5980847240?check_suite_focus=true

Woah... zero clue. I'd call this a flake, but I have no idea how it'd happen.

@f0ssel it's because of this: https://github.com/coder/coder/pull/941/files#diff-5499c2aac6d11c8e0574c18b1dd90a14bfb063426e07115d0e453adb08c33574R35

You probably shouldn't use our API package for this, because we can just rw.Write instead.