Add support for handling zarr urls - #7113

Codecov Report

❌ Patch coverage is 94.04762% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.34%. Comparing base (149dfd7) to head (ddca8b2).

@@            Coverage Diff             @@
##             main    #7113      +/-   ##
==========================================
+ Coverage   90.32%   90.34%   +0.02%     
==========================================
  Files          92       92              
  Lines       25284    25342      +58     
  Branches     4719     4731      +12     
==========================================
+ Hits        22838    22896      +58     
  Misses       1675     1675              
  Partials      771      771              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Some Notes

Since @pp-mo @stephenworsley discussed this yesterday, here's my take on some things we found...

• I pointed out that I'm not too sure about the approach here , because I think it undermines the intent of the "schemes"

  • if done more rigorously, I think it should be possible to supply a url to a remote zarr dataset, similar to the OpenDAP protocol supported by the netcdf file loader
    • .. but I don't know if there's a server solution that actually supports such url-based remote access ?
  • .. and admittedly the whole framework of "schemes" is a bit clunky :
    Notably, in the case of netcdf files, we can access from a filepath, a url, or an open dataset ...
    • first, we have separate "schemes" for handling filepaths, urls, and dataset(objects), which are managed here
    • we then supply separate "format agents" for the three different routes here , here and here
    • but then all of those call the same iris.fileformats.netcdf.loader.load_cubes handler function
      • .. and other code in the CFReader then needs to separately detect+handle the open-dataset case anyway.

• we also found that saving to zarr by this means currently fails for lazy data streaming (though OK for all-real data)

  • since we added "deferred writes" for netcdf (Lazy netcdf saves #5191), our standard approach is to create a variable and not write the data. We then close the file, and the proxies re-open it in modify ("r+") mode, to fill in a chunk of data.
  • .. it looks like this simply doesn't work with netCDF4-python any more if it is writing Zarr :
    when such a file is re-opened, no such variable exists (the metadata is not saved ?)
  • .. it's not clear though whether this is an essential limitation of Zarrr itself (controlling metadata cannot exist when there are no recorded chunks for a given array ??). Or possibly, this is just a problem with the netCDF4-python implementation.
  • we could however fix this by giving the 'lazy stream' operation an extra mode control, to be specifically turned on for Zarr ..
    This would use a "da.store" directly when creating the original file, i.e. a "third way" for this code to function, where the 'store' operation is just da.store -- which is exactly what the code used to do before Lazy netcdf saves #5191.
    N.B. I already found in Lazy netcdf saves #5191 that you can't create a write proxy and just 'da.store' it immediately (? at least for process-based/distributed scheduler? ), since it needs to re-open the file with write privilege, which will then fail (since the file is already, or "still" open for write). So, to store immediately, you must call da.store directly.

• we need to discuss whether this makes this whole approach too awkward for supporting Zarr via this route?
  @ukmo-ccbunney to reconsider versus alternative mechanisms in Zarr I/O #6961 ??