403 error while using github API #5801
Description
Summary
TL;DR: requests raises a 403 while requesting an authenticated Github API route, which otherwise succeeds while using curl/another python library like httpx
Was initially discovered in the 'ghexport' project; I did a reasonable amount of debugging and created this repo before submitting this issue to PyGithub, but thats a lot to look through, just leaving it here as context.
It's been hard to reproduce, the creator of ghexport (where this was initially discovered) didn't have the same issue, so I'm unsure of the exact reason
Expected Result
requests succeeds for the authenticated request
Actual Result
Request fails, with:
{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-repository-clones'}
failed
Traceback (most recent call last):
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 47, in <module>
main()
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 44, in main
make_request(requests.get, url, headers)
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 31, in make_request
resp.raise_for_status()
File "/home/sean/.local/lib/python3.9/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
Reproduction Steps
Apologies if this is a bit too specific, but otherwise requests works great on my system and I can't find any other way to reproduce this -- Is a bit long as it requires an auth token
Go here and create a token with scopes like:
I've compared this to httpx, where it doesn't fail:
#!/usr/bin/env python3
from typing import Callable, Any
import requests
import httpx
# extract status/status_code from the requests/httpx item
def extract_status(obj: Any) -> int:
if hasattr(obj, "status"):
return obj.status
if hasattr(obj, "status_code"):
return obj.status_code
raise TypeError("unsupported request object")
def make_request(using_verb: Callable[..., Any], url: str, headers: Any) -> None:
print("using", using_verb.__module__, using_verb.__qualname__, url)
resp = using_verb(url, headers=headers)
status = extract_status(resp)
print(str(resp.json()))
if status == 200:
print("succeeded")
else:
print("failed")
resp.raise_for_status()
def main():
# see https://github.com/seanbreckenridge/pygithub_requests_error for token scopes
auth_token = "put your auth token here"
headers = {
"Authorization": "token {}".format(auth_token),
"User-Agent": "requests_error",
"Accept": "application/vnd.github.v3+json",
}
# replace this with a URL you have access to
url = "https://api.github.com/repos/seanbreckenridge/albums/traffic/clones"
make_request(httpx.get, url, headers)
make_request(requests.get, url, headers)
if __name__ == "__main__":
main()That outputs:
using httpx get https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
{'count': 15, 'uniques': 10, 'clones': [{'timestamp': '2021-04-12T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-14T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-17T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-18T00:00:00Z', 'count': 2, 'uniques': 2}, {'timestamp': '2021-04-23T00:00:00Z', 'count': 9, 'uniques': 5}, {'timestamp': '2021-04-25T00:00:00Z', 'count': 1, 'uniques': 1}]}
succeeded
using requests.api get https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-repository-clones'}
failed
Traceback (most recent call last):
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 48, in <module>
main()
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 45, in main
make_request(requests.get, url, headers)
File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 32, in make_request
resp.raise_for_status()
File "/home/sean/.local/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
Another thing that may be useful as context is the pdb trace I did here, which was me stepping into where the request was made in PyGithub, and making all the requests manually using the computed url/headers. Fails when I use requests.get but httpx.get works fine:
> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(484)__requestEncode()
-> self.NEW_DEBUG_FRAME(requestHeaders)
(Pdb) n
> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(486)__requestEncode()
-> status, responseHeaders, output = self.__requestRaw(
(Pdb) w
/home/sean/Repos/ghexport/export.py(109)<module>()
-> main()
/home/sean/Repos/ghexport/export.py(84)main()
-> j = get_json(**params)
/home/sean/Repos/ghexport/export.py(74)get_json()
-> return Exporter(**params).export_json()
/home/sean/Repos/ghexport/export.py(60)export_json()
-> repo._requester.requestJsonAndCheck('GET', repo.url + '/traffic/' + f)
/home/sean/.local/lib/python3.8/site-packages/github/Requester.py(318)requestJsonAndCheck()
-> *self.requestJson(
/home/sean/.local/lib/python3.8/site-packages/github/Requester.py(410)requestJson()
-> return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(486)__requestEncode()
-> status, responseHeaders, output = self.__requestRaw(
(Pdb) url
'/repos/seanbreckenridge/advent-of-code-2019/traffic/views'
(Pdb) requestHeaders
{'Authorization': 'token <MY TOKEN HERE>', 'User-Agent': 'PyGithub/Python'}
(Pdb) import requests
(Pdb) requests.get("https://api.github.com" + url, headers=requestHeaders).json()
{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-page-views'}
(Pdb) httpx.get("https://api.github.com" + url, headers=requestHeaders).json()
{'count': 0, 'uniques': 0, 'views': []}
(Pdb) httpx succeeded??
System Information
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": "3.4.7"
},
"idna": {
"version": "2.10"
},
"implementation": {
"name": "CPython",
"version": "3.9.3"
},
"platform": {
"release": "5.11.16-arch1-1",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "101010bf",
"version": "20.0.1"
},
"requests": {
"version": "2.25.1"
},
"system_ssl": {
"version": "101010bf"
},
"urllib3": {
"version": "1.25.9"
},
"using_pyopenssl": true
}
$ pip freeze | grep -E 'requests|httpx'
httpx==0.16.1
requests==2.25.1