Closed
Description
Right now we're using httplib2
because of oauth2client
working really nicely with that, however httplib2
doesn't support a streaming download.
This means, a .request()
will load the entire response content into memory, which won't work at all for a big file.
To get around this, we chunk the file up using the Range HTTP header and request pieces of it until the end, however the better way to handle this is with a streaming HTTP library:
stream = <send http request>
with open('output.txt', 'wb') as output:
data = stream.read(CHUNK_SIZE)
while data:
output.write(data)
data = stream.read(CHUNK_SIZE)
To make this work we need to:
- use a library that does support streaming
- update
oauth2client
to work with that library