Add checksum validation for non-chunked non-composite downloads #31

mfschwartz · 2017-10-06T19:54:35Z

This partially addresses #22

Note current limitations:

doesn't handle chunked download case
doesn't handle case of downloading composite objects
doesn't do anything about upload checksumming

I ran the unit and system tests only against Python 2.7, because my current OS doesn't support Python 3.6.

dhermes

@mfschwartz I am happy to maintainer-ify this PR if you are OK with it? I think it may be easier than a nit-picky review.

google/resumable_media/requests/download.py

+                server-computed checksum.
        """
+        md5_hash = hashlib.md5()
+        # TODO: Add support for validating CRC32C for composite objects. Note


google/resumable_media/requests/download.py

+        # make getting google-cloud-python working well more difficult).
+        expected_md5_hash = None
+        if ('X-Goog-Hash' in response.headers
+            and response.headers['X-Goog-Hash']):


google/resumable_media/requests/download.py

+                if name == 'md5':
+                    expected_md5_hash = value
+        if not expected_md5_hash:
+            logging.getLogger().info(


setup.py

 setuptools.setup(
    name='google-resumable-media',
-    version='0.2.3',
+    version='0.2.4',


google/resumable_media/requests/download.py

+                               .rstrip('\n'))
+        else:
+            actual_md5_hash = (base64.encodebytes(bytes(md5_hash.digest(),
+                                                        'UTF-8')).rstrip('\n'))


mfschwartz · 2017-10-10T22:34:13Z

Hi - I'd appreciate if we could draw this review to a close soon because the lack of checksumming support in this library is causing customer problems. Thanks.

lukesneeringer · 2017-10-11T15:11:37Z

@mfschwartz I am happy to maintainer-ify this PR if you are OK with it? I think it may be easier than a nit-picky review.

Yes please.

dhermes · 2017-10-11T18:28:34Z

@mfschwartz I'm closing this PR in favor of #32 (which uses your 3 commits as a diffbase).

Usually GitHub allows maintainers to push into a PR branch, but I couldn't for some reason (possibly because you used the master branch).

As for not being able to install Python 3 on your machine, if you are on Mac OS X or Linux, I highly recommend pyenv (it does not require root privileges). If you are on Windows, I recommend conda or Miniconda (ditto on admin privileges). I'm happy to help if it's relevant to you. If not, no worries.

dhermes · 2017-10-11T18:30:09Z

... the lack of checksumming support in this library is causing customer problems

If you are referring to googleapis/google-cloud-python#4129, it seems that the checksumming is a theory for that breakage (or rather, something that could pinpoint the breakage), but we have not been able to reproduce the customer's report.

mfschwartz · 2017-10-11T18:41:41Z

Thanks for the commits Danny.

The problem I was referring to was not googleapis/google-cloud-python#4129, but rather a customer who reached out to the GCS team directly.

mfschwartz · 2017-10-11T18:42:25Z

Can you please tell me when you have the two libraries in a state where it's possible to build new releases (at a min I think that means updated version #s and dependencies)? Thanks!

dhermes · 2017-10-11T18:55:20Z

Yes. I'm worried that google-cloud-core will make releasing painful. Hopefully not.

dhermes · 2017-10-11T18:59:24Z

@mfschwartz Am I correct in assuming you are OK with a release with the partial support implemented?

mfschwartz · 2017-10-11T19:05:24Z

Yes, a release with partial support would be good.
I'm sure you're going to but can you please make sure all the tests pass? I tried to fix the test failure on python 3.6 but since I don't currently have a python 3.6 installed I can't test it.
(I saw your note about installing 3.6 on various platforms. I could do that on my Mac; I just did this development on an Ubuntu version that currently doesn't yet support 3.6.)

dhermes · 2017-10-11T19:08:41Z

Yes, they are all passing: https://circleci.com/gh/GoogleCloudPlatform/google-resumable-media-python/198

My primary dev machine is Ubuntu and I use pyenv and never looked back. It's as easy as:

git clone https://github.com/pyenv/pyenv.git ~/.pyenv
~/.pyenv/bin/pyenv install 3.6.3
~/.pyenv/versions/3.6.3/bin/python3.6

If you want to just type python3.6, check out the README on the GitHub repo for things to add to your bashrc (or zshrc, etc.)

mfschwartz · 2017-10-11T19:23:40Z

Ah, thanks, that was quite easy!

mfschwartz added 2 commits October 5, 2017 20:04

Add checksum validation for non-chunked non-composite downloads

1d42dd0

Changes in response to review comments

ae9b025

mfschwartz mentioned this pull request Oct 6, 2017

Catch checksum validation failure from download, delete corrupt file googleapis/google-cloud-python#4133

Merged

dhermes reviewed Oct 6, 2017

View reviewed changes

mfschwartz mentioned this pull request Oct 9, 2017

Add (optional) hash validation of objects #22

Open

6 tasks

changes per dhermes comments

ffaf316

mfschwartz commented Oct 9, 2017

View reviewed changes

google/resumable_media/requests/download.py

.rstrip('\n'))

else:

actual_md5_hash = (base64.encodebytes(bytes(md5_hash.digest(),

'UTF-8')).rstrip('\n'))

This comment was marked as spam.

Sign in to view

dhermes added a commit to dhermes/google-resumable-media-python that referenced this pull request Oct 11, 2017

Re-factor PR googleapis#31, added lots of unit tests.

2518f96

dhermes mentioned this pull request Oct 11, 2017

Add checksum validation for non-chunked non-composite downloads #32

Merged

dhermes added a commit that referenced this pull request Oct 11, 2017

Auto-gen docs update for PR #31/32.

e565a2b

dhermes added download requests-transport labels Oct 11, 2017

dhermes closed this Oct 11, 2017

mcsimps2 mentioned this pull request Sep 8, 2020

Allow space in-between checksums in response headers #169

Closed

Add checksum validation for non-chunked non-composite downloads #31

Add checksum validation for non-chunked non-composite downloads #31

Uh oh!

Conversation

mfschwartz commented Oct 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhermes left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

mfschwartz commented Oct 10, 2017

Uh oh!

lukesneeringer commented Oct 11, 2017

Uh oh!

dhermes commented Oct 11, 2017

Uh oh!

dhermes commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mfschwartz commented Oct 11, 2017

Uh oh!

mfschwartz commented Oct 11, 2017

Uh oh!

dhermes commented Oct 11, 2017

Uh oh!

dhermes commented Oct 11, 2017

Uh oh!

mfschwartz commented Oct 11, 2017

Uh oh!

dhermes commented Oct 11, 2017

Uh oh!

mfschwartz commented Oct 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mfschwartz commented Oct 6, 2017 •

edited

Loading

dhermes commented Oct 11, 2017 •

edited

Loading