Right now, the HTTP client that can be invoked via the from operator treats every response to a request the same (i.e., HTTP response codes 2xx, 4xx, 5xx, etc). In the spirit of #4225, it's likely that users that arrive at SuperDB after using common tools like curl will seek ways to handle exceptions to the HTTP 2xx that typically implies "success", as well as flexibility in other possible errors such as at the transport layer.

Details

Repro is with super commit fc5aa27 and the attached server.py. Once it's running (python3 server.py), it will return different HTTP response codes on its different endpoints.

As a baseline with curl, by default it does also treat any response as "success" in terms of the command's exit code, and only a transport error results in a non-zero exit code, e.g.,

$ curl -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/ok ; echo "exit_code=$?"
{"note":"real data"}
HTTP response code: 200
exit_code=0

$ curl -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/notfound ; echo "exit_code=$?"
{"note":"HTTP 404 ERROR BODY -- not your data"}
HTTP response code: 404
exit_code=0

$ curl -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/boom ; echo "exit_code=$?"
{"note":"HTTP 500 ERROR BODY -- not your data"}
HTTP response code: 500
exit_code=0

$ curl -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8189/ok ; echo "exit_code=$?"
curl: (7) Failed to connect to 127.0.0.1 port 8189 after 0 ms: Couldn't connect to server
HTTP response code: 000
exit_code=7

However, once the --fail option is added, now the non-2xx response codes trigger non-zero exit codes.

$ curl --fail -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/ok ; echo "exit_code=$?"
{"note":"real data"}
HTTP response code: 200
exit_code=0

$ curl --fail -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/notfound ; echo "exit_code=$?"
curl: (22) The requested URL returned error: 404
HTTP response code: 404
exit_code=22

$ curl --fail -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8188/boom ; echo "exit_code=$?"
curl: (22) The requested URL returned error: 500
HTTP response code: 500
exit_code=22

$ curl --fail -w "HTTP response code: %{http_code}\n" http://127.0.0.1:8189/ok ; echo "exit_code=$?"
curl: (7) Failed to connect to 127.0.0.1 port 8189 after 0 ms: Couldn't connect to server
HTTP response code: 000
exit_code=7

The current super behavior is in some ways similar to the curl default, but with the shortcoming that the 4xx/5xx error body silently becomes the data with no signal this happened. So unless the server sends known response payloads associated with certain non-2xx response codes, the user may not be able to add conditional logic to handle errors differently.

$ super -version
Version: v0.3.0-237-gfc5aa27bd

$ super -s -c "from 'http://127.0.0.1:8188/ok' (format json)" ; echo "exit_code=$?"
{note:"real data"}
exit_code=0

$ super -s -c "from 'http://127.0.0.1:8188/notfound' (format json)" ; echo "exit_code=$?"
{note:"HTTP 404 ERROR BODY -- not your data"}
exit_code=0

$ super -s -c "from 'http://127.0.0.1:8188/boom' (format json)" ; echo "exit_code=$?"
{note:"HTTP 500 ERROR BODY -- not your data"}
exit_code=0

$ super -s -c "from 'http://127.0.0.1:8189/ok' (format json)" ; echo "exit_code=$?"
Get "http://127.0.0.1:8189/ok": dial tcp 127.0.0.1:8189: connect: connection refused
exit_code=1

Ideas

As we often say, "once we know what we want, implementing should be easy". Here's some initial thoughts of my own on how we might approach this.

Similar to what we've done in other parts of the language, it seems like error values could play a role here, e.g., non-2xx responses could be packaged as structured errors with the response code and any response payload. The user could then use functions like is_error() and has_error() to conditionally act on errors when they occur. This could also be combined with the proposed #5298 halt-on-condition functionality so the user could choose to halt the runtime when any/some errors occur or let the query continue.

Handling of transport errors could benefit from similar treatment. While today they're fatal, perhaps instead error values could be generated so the user would have the option to let the pipeline keep running. This also brings to mind the subject of retry logic, though perhaps that's orthogonal.