Enable option to requests/skip checksums #519
Conversation
for more information, see https://pre-commit.ci
…f argument checksum
|
@ndp-opendap @jgallagher59701 ExampleRequest a 1D array of int16 sequentual values. This is the variable Y in this dataset import requests
session = requests.Session()
r_true = session.get("http://test.opendap.org/opendap/dap4/SimpleGroup.nc4.h5.dap?dap4.ce=/SimpleGroup/Y%5B0:1:39%5D&dap4.checksum=true")
r_false = session.get("http://test.opendap.org/opendap/dap4/SimpleGroup.nc4.h5.dap?dap4.ce=/SimpleGroup/Y%5B0:1:39%5D&dap4.checksum=false")
# test for differences between the two:
assert t_true == t_false
>>> TrueSo no difference in the two responses... |
|
There are no differences because test.opendap.org is ignoring the |
OK! I missed the part about ignoring the checksum query string parameter... |
|
@jgallagher59701 @ndp-opendap Last comment regarding checksums:
ExampleI looked at a test file available on the test TDS. I downloaded it and put it also on the test.opendap.org server. These 2 datasets are identical. Now, I will compare the data values in the array import numpy as np
from pydap.client import open_url
hyx_url = "http://test.opendap.org/opendap/dap4/mydata1.nc"
tds_url = "https://thredds-test.unidata.ucar.edu/thredds/dap4/testdata/mydata1.nc"
### NOTE
# pydap internal's can now append `&dap4.checksum=true` to the url when user defines checksum=True as argument below
# create a pydap array with the TDS url
pyds_TDS = open_url(tds_url, protocol='dap4', checksum=True)
# pydap array with the Hyrax url
pyds_HYX = open_url(hyx_url, protocol='dap4', checksum=True)
print("TDS crc32 checksum: ", pyds_TDS['temperature'][:].attributes['_DAP4_Checksum_CRC32'])
>>> TDS crc32 checksum: 3486137609 # <----------------------- checksum computed by thredds data server
print("Hyrax crc32 checksum: ", pyds_HYX['temperature'][:].attributes['_DAP4_Checksum_CRC32'])
>>> Hyrax crc32 checksum: 3457240079 # <----------------------- checksum computed by hyrax data serverSo checksums differ between the two Data servers... I now look at the data itself and compare values and the checksum I get using python's zlib.crc32 library. # check data is identical between the two dap responses:
temp_TDS = pyds_TDS['temperature'][:].data # downloads the entire data array from TDS
temp_HYX = pyds_HYX['temperature'][:].data # downloads the entire data array from Hyrax
np.testing.assert_equal(temp_TDS, temp_HYX) # If False, an AssertionError is returned. It True, nothing is returned
zlib.crc32(temp_TDS.reshape(1, np.prod(temp.shape))) == zlib.crc32(temp_HYX.reshape(1, np.prod(temp.shape)))
>>> TrueSo data is identical whether I download it from TDS or Hyrax. The checksum are the same when computed using the print("python zlib checksum: ", zlib.crc32(temp_TDS.reshape(1, np.prod(temp.shape))))
>>> python zlib checsksum: 3457240079 # <----------------------- checksum computed by Python's zlib.crc32 library |
The following Pull Request:
on a local environment.