md5 validation fails on gzipped PNG image #50
Description
This is essentially the same issue as was fixed in #36, but now occuring specifically for PNG files with Content-Type image/png and Content-Encoding gzip. I can successfully download neighboring ascii files that are also gzipped (but without a Content-Type set).
This occurs with google-resumable-media 0.3.1:
> pip3 freeze | grep google
google-api-core==1.4.0
google-auth==1.5.1
google-cloud-core==0.28.1
google-cloud-storage==1.11.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
If I downgrade to google-resumable-media 0.2.3 the PNG download succeeds.
Error info:
/usr/local/lib/python3.6/dist-packages/google/cloud/storage/blob.py:538: in download_to_file
transport, file_obj, download_url, headers, start, end)
/usr/local/lib/python3.6/dist-packages/google/cloud/storage/blob.py:480: in _do_download
download.consume(transport)
/usr/local/lib/python3.6/dist-packages/google/resumable_media/requests/download.py:174: in consume
self._write_to_stream(result)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <google.resumable_media.requests.download.Download object at 0x7f81c0f38cf8>, response = <Response [200]>
def _write_to_stream(self, response):
"""Write response body to a write-able stream.
.. note:
This method assumes that the ``_stream`` attribute is set on the
current download.
Args:
response (~requests.Response): The HTTP response object.
Raises:
~google.resumable_media.common.DataCorruption: If the download's
checksum doesn't agree with server-computed checksum.
"""
expected_md5_hash = self._get_expected_md5(response) [20/1803]
if expected_md5_hash is None:
md5_hash = _DoNothingHash()
else:
md5_hash = hashlib.md5()
with response:
# NOTE: This might "donate" ``md5_hash`` to the decoder and replace
# it with a ``_DoNothingHash``.
local_hash = _add_decoder(response.raw, md5_hash)
body_iter = response.iter_content(
chunk_size=_SINGLE_GET_CHUNK_SIZE, decode_unicode=False)
for chunk in body_iter:
self._stream.write(chunk)
local_hash.update(chunk)
if expected_md5_hash is None:
return
actual_md5_hash = base64.b64encode(md5_hash.digest())
# NOTE: ``b64encode`` returns ``bytes``, but ``expected_md5_hash``
# came from a header, so it will be ``str``.
actual_md5_hash = actual_md5_hash.decode(u'utf-8')
if actual_md5_hash != expected_md5_hash:
msg = _CHECKSUM_MISMATCH.format(
self.media_url, expected_md5_hash, actual_md5_hash)
> raise common.DataCorruption(response, msg)
E google.resumable_media.common.DataCorruption: Checksum mismatch while downloading:
E
E https://www.googleapis.com/download/storage/v1/b/****/o/******.png?alt=media
E
E The X-Goog-Hash header indicated an MD5 checksum of:
E
E rlofKEyAqWgDmEc3WO61BA==
E
E but the actual MD5 checksum of the downloaded contents was:
E
E h9fia1VRkKYNo656ukQ4hQ==
/usr/local/lib/python3.6/dist-packages/google/resumable_media/requests/download.py:137: DataCorruption