Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Make gcloud.storage use a streaming HTTP library for downloads #11

Closed
@jgeewax

Description

@jgeewax

Right now we're using httplib2 because of oauth2client working really nicely with that, however httplib2 doesn't support a streaming download.

This means, a .request() will load the entire response content into memory, which won't work at all for a big file.

To get around this, we chunk the file up using the Range HTTP header and request pieces of it until the end, however the better way to handle this is with a streaming HTTP library:

stream = <send http request>
with open('output.txt', 'wb') as output:
  data = stream.read(CHUNK_SIZE)
  while data:
    output.write(data)
    data = stream.read(CHUNK_SIZE)

To make this work we need to:

  • use a library that does support streaming
  • update oauth2client to work with that library

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions