[Tool] Best Guess Netcdfs diff #177
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oelbert
previously requested changes
Jul 9, 2025
Collaborator
oelbert
left a comment
There was a problem hiding this comment.
Small changes, but also how many of these did you run into when diffing NetCDFs? Do you want to add checks for extra dimensions while we're at it
fmalatino
approved these changes
Nov 20, 2025
First draft of tool - can be made better later
FlorianDeconinck
added a commit
that referenced
this pull request
Nov 27, 2025
* Check gt4py-backend options in config (#291) * For release `2025.03.00` (#127) * updating 4d handling * debug 4d test data * more iter * moving ser_to_nc here * updating datatype in translate test * typing works * fix dict, lint * remove empty line * change from 4d to Nd * Expose `k_start` and `k_end` automatically for any FrozenStencil * Fix k_start + utest * lint * Fix for 2d stencils * Add threshold overrides to the multimodal metric * Always report results, add summary with one liners * Remove "mmr" from the keys * README in testing * Better Latex (?) * Better Latex (?) * fixing a typo that breaks bools in translate tests (#80) * Fix summary filename * Fix report, filename * Fix choosing right absolute difference for F32 * Make robust for NaN value * Detect when array have different dimensions, if only one dimension, collapse Clean up type infer and log work * Lint * Add rank 0 to the data * Check data exists for rank, skip & print if not * Fix bad logic on skip test for parallel * Verbose exported names * Make boilerplate calls more nimble * New option: `which_savepoint` Better error on bad output data Fix missing integer type check * QOL for mypy/flak8 type hints * Add SECONDS_PER_DAY as a constants following mixed precision standards * Lint * Cleanups in dace orchestration Readability improvements in dace orchestration including - early returns - spelling out variable names - fixing typos * Rename program -> dace_program * Make sure all constants adhere to the floating point precision set by the system * Move `is_float` to `dsl.typing` * Move Quantity to sub-directory + breakout the subcomponent * Fix tests * Lint * Remove `cp.ndarray` since cupy is optional * Restore workaround for optional cupy * "GFS" -> "UFS" * Cupy trick for metadata * Add comments for constant explanation * Describe 64/32-bit FloatFields * Make sure the `make_storage_data` respects the array dtype. * Fix logic for MultiModal metric and verbose it * Added an MPI all_reduce for quantities based on SUM operation to communicator.py * linted * Add initial skeleton of pytest test for all reduce * Added assertion tests for 1, 2 and 3D quantities passed through mpi_allreduce_sum * Linted * Added pytest.mark to skip test if mpi4py isn't available * lint changes * Addressed PR comments and added additional CPU backends to unit test * Added setters for various Quantity properties to enable setting of Quantity metadata and data properties. * Added function in QuantityMetadata class that allows copying of Metadata properties from one class to another. Subsequent Quantity setters that performed the copying of QuantityMetadata properties were removed * Expose all SG metric terms in grid_data * Add `Allreduce` and all MPI OP * Update utest * Fix `local_comm` * Fix utest * Enforce `comm_abc.Comm` into Communicator * Fix `comm` object in serial utest * Lint + `MPIComm` on testing architecture * Make sure the correct allocator backend is used for Quantities * Add in_place option for Allreduce * Cleanup ndsl/dsl/dace/utils.py (#96) * Fix typos * DaCeProgress: avoid double assignment of prefix * Add type hints/simplify kernel_theoretical_timing Adding type hints allowed to simplify `kernel_theoretical_timing`. * Fix merge * Hotfix for grid generation use of mpi operators * Merge examples/mpi/.gitignore into top-level .gitignore * Remove hard-coded __version__ numbers Removes hard-coded version numbers from `__init__` files. * Fixing a bunch of typos * hotfix netcdf version for dockerfiles * Updated version number in setup.py to reflect new release, 2025.01.00 * Adding in exception for compute domains with sizes less than or equal to halo size (#103) * Adding in exception for compute domains with less than 4 points to vector_halo_update method * Updated exception in communicator to compare halo size to compute domain size * linting * Moved domain size checker to SubtileGridSizer class method from_tile_params * Fix passing down ak/bk for pressure coefficients when they are available from an outside source (online model case) (#107) * [QOL] Logging, Type Hints and Quantity helpers (#108) * Log on rank 0 Docstrings & typi hints on logger Stencil Config has a `verbose` option On verbose: FrozenStencil log when run (in GT backends) * Update `config` in orchestrate call to solve type hint inconcistencies * Quantity helper `to_netcdf` with multi rank support * Automatic Int precision and stencil regeneration change (#104) * Added feature to enable automatic detection of integer precision. Should remove the need for i32/i64 declaration (although their functionality is still retained) and replace both with the regular Int type * change default rebuild state to false for get_factories * Merged Float and Int precision detection functions into one common path * Re-added old function to fulfil a PACE dependency * updated docstring * Added ability to declare 32 or 64 bit IntFields, overrulling the system precision * Added one dimensional bool fields * Fix error message in typing.py Co-authored-by: Florian Deconinck <[email protected]> * output type for global_set_precision --------- Co-authored-by: Florian Deconinck <[email protected]> * Bump DaCe to v1.0.1 (#109) Our current DaCe version is some commit from September 2024. Meanwhile DaCe matured to v1 and recently release v1.0.1. This brings the DaCe submodule to the latest stable release version. Co-authored-by: Roman Cattaneo <[email protected]> * Streamline linting workflow (#110) Linting should give fast feedback. The current workflow takes ~3mins where most of the time is spent installing (unnecessary) python packages. To run `pre-commit`, we only need the source files and `pre-commit` itself, which can be installed standalone. This brings runtime of the linting stage down to ~30 seconds. Other changes - update checkout action to v4 - update python setup action to v5 - change python version from 3.11.7 to 3.11 (any patch number will do) This is a follow-up of PR NOAA-GFDL/PyFV3#40 in PyFV3. Co-authored-by: Roman Cattaneo <[email protected]> * [FIX] Type hint for precision dependant Float, Int (#111) * Fix the type hint of Float, Int * Attempt using TypeAlias * Feature: Adding documentation (#97) * Added doc files * Adding image files to docs * Linting * Updated docs to reflect changes requested in PR 97 * Linting --------- Co-authored-by: Florian Deconinck <[email protected]> * [Translate test] Save better reports & netCDF for multiple ranks on failure (#106) * Save reports & netCDF for multiple ranks on failure Fix multi modal threshold for parallel tests * Order field by name in NetCDF * Print all indices in logs. Sort by descernding ULP * Allow sorting by metrics and index with `--sort_report` option * Remove the `rank` froom SavepointCase. Access is done via `grid` * Some docstrings * Adds some quick capacities used in the post-radiation phase of the physics, including the Stefan-Boltzmann constant (#116) * add namelist option * add stephan boltzmann constant * lint * Apply suggestions from code review Change comments to docstring style Co-authored-by: Florian Deconinck <[email protected]> --------- Co-authored-by: Florian Deconinck <[email protected]> * Adding temperature of h2o triple point (#115) * add ttp * Update ndsl/constants.py Co-authored-by: Florian Deconinck <[email protected]> * switch comments to docstrings for autodocs * lint --------- Co-authored-by: Florian Deconinck <[email protected]> * [Feature] Porting workflow: enhancing errors readability (#114) * Save all fields (pass and fail) and organize them by field * Option `--no_report` to bypass logging & netcdf save Move logs per variable into a `details` subfolder * Order variable name in serialbox-to-netcdf * `extra_data_load` function to load savepoint data saved outside the canonical savepoint * Docs / Type Hint * Fixed typo in error statment --------- Co-authored-by: Charles Kropiewnicki <[email protected]> * Feature: NetCDF output precision configurable (#117) * Removed hard-code of np.float32 from NetCDFMonitor transfer_type, replaced with Float type * Added multiple options for NetCDF precision * Added checking for use of 32 precision and float64 output * Using NumPy type instead of string in NetCDFMonitor precision variable * Added warning to netcdf_monitor.py for mismatch in precision settings * Forgot f-string in warn message of netcdf_monitor * Mixed Precision fixes and QOL (#118) * Ignore `.next` caches * CNST_OP20 is a true 64-bit * Translate: Fix reading parameters with the right precision * Multimodal metric: Skip reporting on expected values * Bad commit * Add license (Apache 2.0) (#105) Co-authored-by: Roman Cattaneo <[email protected]> * Change deprecated `np.product()` to `np.prod()` (#120) Starting with numpy v1.25.0, `np.product()` is deprecated and `np.prod()` should be used instead. Co-authored-by: Roman Cattaneo <[email protected]> * Update GT4Py and DaCe to bring in refactored GT4Py/DaCe bridge that exposes control flow (#119) * Update DaCe to v1.0.2 DaCe v1.0.2 brings two fixes for DaCe transformations: one for DeadDataflowElimination and one for StateFusion. * Bump gt4py to include refactored gt4py/dace bridge * Test with modified pace pipeline - added this to re-trigger the new pace pipeline after limiting zarr to not install v3 (for now) because of breaking API changes. - added this note to re-trigger after fixing the pace pipeline to not pull requirements from `develop`. - added this note to ret-trigger after fixing the repo name * Revert "Test with modified pace pipeline" This reverts commit cd6560e. --------- Co-authored-by: Roman Cattaneo <[email protected]> * Grid Mixed Precision and Coriolis force load (+ QOL) (#121) * Pass `dtype` down in allocator utils (gt4py_utils) * Allow coriolis forces to be read in * Edge factors are always 64-bit * Quantity QOL * Make sure to pass `dtype` to load the grid cleanly * Translate grid: load coriolis forces, area 64 is 64-bit * Bad merge * Typo * GEOS version of dz_min (#122) * Doc enhancment (#123) **Description** Port and adaptation of the initial commit of the documentation. Fixes issue #113 **Checklist:** - [X] I have performed a self-review of my own code - [X] I have made corresponding changes to the documentation - [X] My changes generate no new warnings * Fix saving NetCDF for parallel translate test (#125) * Release candidate 2025.03.00 (#124) Co-authored-by: Florian Deconinck <[email protected]> * Fix for bad merge of 7fdfa5 (#129) --------- Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Roman Cattaneo <> Co-authored-by: Christopher Kung <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Tobias Wicky-Pfund <[email protected]> * check for backend existence in config * pc * update stale backend name --------- Co-authored-by: Frank Malatino <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Christopher Kung <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Frank Malatino <[email protected]> * fix: allow any Comm object in ZarrMonitor (#292) This PR is fallout from adding types in PR #257 and #258. The `ZarrMonitor` provides a `DummyComm` which is instantiated in case no `Comm` object is given. The type of the `Comm` object in `ZarrMonitor` was wrongly limited to that `DummyComm`, which only broke when we attempted to update the submodule in `pace`. Co-authored-by: Roman Cattaneo <[email protected]> * Patch domain checks to only happen once (#293) * For release `2025.03.00` (#127) * updating 4d handling * debug 4d test data * more iter * moving ser_to_nc here * updating datatype in translate test * typing works * fix dict, lint * remove empty line * change from 4d to Nd * Expose `k_start` and `k_end` automatically for any FrozenStencil * Fix k_start + utest * lint * Fix for 2d stencils * Add threshold overrides to the multimodal metric * Always report results, add summary with one liners * Remove "mmr" from the keys * README in testing * Better Latex (?) * Better Latex (?) * fixing a typo that breaks bools in translate tests (#80) * Fix summary filename * Fix report, filename * Fix choosing right absolute difference for F32 * Make robust for NaN value * Detect when array have different dimensions, if only one dimension, collapse Clean up type infer and log work * Lint * Add rank 0 to the data * Check data exists for rank, skip & print if not * Fix bad logic on skip test for parallel * Verbose exported names * Make boilerplate calls more nimble * New option: `which_savepoint` Better error on bad output data Fix missing integer type check * QOL for mypy/flak8 type hints * Add SECONDS_PER_DAY as a constants following mixed precision standards * Lint * Cleanups in dace orchestration Readability improvements in dace orchestration including - early returns - spelling out variable names - fixing typos * Rename program -> dace_program * Make sure all constants adhere to the floating point precision set by the system * Move `is_float` to `dsl.typing` * Move Quantity to sub-directory + breakout the subcomponent * Fix tests * Lint * Remove `cp.ndarray` since cupy is optional * Restore workaround for optional cupy * "GFS" -> "UFS" * Cupy trick for metadata * Add comments for constant explanation * Describe 64/32-bit FloatFields * Make sure the `make_storage_data` respects the array dtype. * Fix logic for MultiModal metric and verbose it * Added an MPI all_reduce for quantities based on SUM operation to communicator.py * linted * Add initial skeleton of pytest test for all reduce * Added assertion tests for 1, 2 and 3D quantities passed through mpi_allreduce_sum * Linted * Added pytest.mark to skip test if mpi4py isn't available * lint changes * Addressed PR comments and added additional CPU backends to unit test * Added setters for various Quantity properties to enable setting of Quantity metadata and data properties. * Added function in QuantityMetadata class that allows copying of Metadata properties from one class to another. Subsequent Quantity setters that performed the copying of QuantityMetadata properties were removed * Expose all SG metric terms in grid_data * Add `Allreduce` and all MPI OP * Update utest * Fix `local_comm` * Fix utest * Enforce `comm_abc.Comm` into Communicator * Fix `comm` object in serial utest * Lint + `MPIComm` on testing architecture * Make sure the correct allocator backend is used for Quantities * Add in_place option for Allreduce * Cleanup ndsl/dsl/dace/utils.py (#96) * Fix typos * DaCeProgress: avoid double assignment of prefix * Add type hints/simplify kernel_theoretical_timing Adding type hints allowed to simplify `kernel_theoretical_timing`. * Fix merge * Hotfix for grid generation use of mpi operators * Merge examples/mpi/.gitignore into top-level .gitignore * Remove hard-coded __version__ numbers Removes hard-coded version numbers from `__init__` files. * Fixing a bunch of typos * hotfix netcdf version for dockerfiles * Updated version number in setup.py to reflect new release, 2025.01.00 * Adding in exception for compute domains with sizes less than or equal to halo size (#103) * Adding in exception for compute domains with less than 4 points to vector_halo_update method * Updated exception in communicator to compare halo size to compute domain size * linting * Moved domain size checker to SubtileGridSizer class method from_tile_params * Fix passing down ak/bk for pressure coefficients when they are available from an outside source (online model case) (#107) * [QOL] Logging, Type Hints and Quantity helpers (#108) * Log on rank 0 Docstrings & typi hints on logger Stencil Config has a `verbose` option On verbose: FrozenStencil log when run (in GT backends) * Update `config` in orchestrate call to solve type hint inconcistencies * Quantity helper `to_netcdf` with multi rank support * Automatic Int precision and stencil regeneration change (#104) * Added feature to enable automatic detection of integer precision. Should remove the need for i32/i64 declaration (although their functionality is still retained) and replace both with the regular Int type * change default rebuild state to false for get_factories * Merged Float and Int precision detection functions into one common path * Re-added old function to fulfil a PACE dependency * updated docstring * Added ability to declare 32 or 64 bit IntFields, overrulling the system precision * Added one dimensional bool fields * Fix error message in typing.py Co-authored-by: Florian Deconinck <[email protected]> * output type for global_set_precision --------- Co-authored-by: Florian Deconinck <[email protected]> * Bump DaCe to v1.0.1 (#109) Our current DaCe version is some commit from September 2024. Meanwhile DaCe matured to v1 and recently release v1.0.1. This brings the DaCe submodule to the latest stable release version. Co-authored-by: Roman Cattaneo <[email protected]> * Streamline linting workflow (#110) Linting should give fast feedback. The current workflow takes ~3mins where most of the time is spent installing (unnecessary) python packages. To run `pre-commit`, we only need the source files and `pre-commit` itself, which can be installed standalone. This brings runtime of the linting stage down to ~30 seconds. Other changes - update checkout action to v4 - update python setup action to v5 - change python version from 3.11.7 to 3.11 (any patch number will do) This is a follow-up of PR NOAA-GFDL/PyFV3#40 in PyFV3. Co-authored-by: Roman Cattaneo <[email protected]> * [FIX] Type hint for precision dependant Float, Int (#111) * Fix the type hint of Float, Int * Attempt using TypeAlias * Feature: Adding documentation (#97) * Added doc files * Adding image files to docs * Linting * Updated docs to reflect changes requested in PR 97 * Linting --------- Co-authored-by: Florian Deconinck <[email protected]> * [Translate test] Save better reports & netCDF for multiple ranks on failure (#106) * Save reports & netCDF for multiple ranks on failure Fix multi modal threshold for parallel tests * Order field by name in NetCDF * Print all indices in logs. Sort by descernding ULP * Allow sorting by metrics and index with `--sort_report` option * Remove the `rank` froom SavepointCase. Access is done via `grid` * Some docstrings * Adds some quick capacities used in the post-radiation phase of the physics, including the Stefan-Boltzmann constant (#116) * add namelist option * add stephan boltzmann constant * lint * Apply suggestions from code review Change comments to docstring style Co-authored-by: Florian Deconinck <[email protected]> --------- Co-authored-by: Florian Deconinck <[email protected]> * Adding temperature of h2o triple point (#115) * add ttp * Update ndsl/constants.py Co-authored-by: Florian Deconinck <[email protected]> * switch comments to docstrings for autodocs * lint --------- Co-authored-by: Florian Deconinck <[email protected]> * [Feature] Porting workflow: enhancing errors readability (#114) * Save all fields (pass and fail) and organize them by field * Option `--no_report` to bypass logging & netcdf save Move logs per variable into a `details` subfolder * Order variable name in serialbox-to-netcdf * `extra_data_load` function to load savepoint data saved outside the canonical savepoint * Docs / Type Hint * Fixed typo in error statment --------- Co-authored-by: Charles Kropiewnicki <[email protected]> * Feature: NetCDF output precision configurable (#117) * Removed hard-code of np.float32 from NetCDFMonitor transfer_type, replaced with Float type * Added multiple options for NetCDF precision * Added checking for use of 32 precision and float64 output * Using NumPy type instead of string in NetCDFMonitor precision variable * Added warning to netcdf_monitor.py for mismatch in precision settings * Forgot f-string in warn message of netcdf_monitor * Mixed Precision fixes and QOL (#118) * Ignore `.next` caches * CNST_OP20 is a true 64-bit * Translate: Fix reading parameters with the right precision * Multimodal metric: Skip reporting on expected values * Bad commit * Add license (Apache 2.0) (#105) Co-authored-by: Roman Cattaneo <[email protected]> * Change deprecated `np.product()` to `np.prod()` (#120) Starting with numpy v1.25.0, `np.product()` is deprecated and `np.prod()` should be used instead. Co-authored-by: Roman Cattaneo <[email protected]> * Update GT4Py and DaCe to bring in refactored GT4Py/DaCe bridge that exposes control flow (#119) * Update DaCe to v1.0.2 DaCe v1.0.2 brings two fixes for DaCe transformations: one for DeadDataflowElimination and one for StateFusion. * Bump gt4py to include refactored gt4py/dace bridge * Test with modified pace pipeline - added this to re-trigger the new pace pipeline after limiting zarr to not install v3 (for now) because of breaking API changes. - added this note to re-trigger after fixing the pace pipeline to not pull requirements from `develop`. - added this note to ret-trigger after fixing the repo name * Revert "Test with modified pace pipeline" This reverts commit cd6560e. --------- Co-authored-by: Roman Cattaneo <[email protected]> * Grid Mixed Precision and Coriolis force load (+ QOL) (#121) * Pass `dtype` down in allocator utils (gt4py_utils) * Allow coriolis forces to be read in * Edge factors are always 64-bit * Quantity QOL * Make sure to pass `dtype` to load the grid cleanly * Translate grid: load coriolis forces, area 64 is 64-bit * Bad merge * Typo * GEOS version of dz_min (#122) * Doc enhancment (#123) **Description** Port and adaptation of the initial commit of the documentation. Fixes issue #113 **Checklist:** - [X] I have performed a self-review of my own code - [X] I have made corresponding changes to the documentation - [X] My changes generate no new warnings * Fix saving NetCDF for parallel translate test (#125) * Release candidate 2025.03.00 (#124) Co-authored-by: Florian Deconinck <[email protected]> * Fix for bad merge of 7fdfa5 (#129) --------- Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Roman Cattaneo <> Co-authored-by: Christopher Kung <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Tobias Wicky-Pfund <[email protected]> * check domain size args only once * review & test --------- Co-authored-by: Frank Malatino <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Christopher Kung <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Frank Malatino <[email protected]> * BREAKING CHANGE: change constructor of `QuantityFactory` (#228) * Breaking change: QuantityFactory from GridSizer and backend name Change `QuantityFactory` to initialize from a `GridSizer` (as previously) and a backend name (new). This effectively hides the previous `numpy` argument, which is effectively an internal allocator that users shouldn't need to know about. It's basically what `from_backend()` was doing before (which is now obsolete and was thus removed). This is a BREAKING CHANGE and users will need to update their codes if they instantiated QuantityFactories themselves. For users relying on the `boilerplate` module, no changes need to happen. * Keep QuantityFactory.from_backend() with a deprecation warning * Extended docstings This is mainly to force a new run of the pyshild workflow now that pyshield tests are exclusively using `QuantityFactory.from_backend()` which is compatible with changes proposed in this PR. * More updates to docstrings * fixup after rebase * Unrelated: tests are supposed to return `None` * fixup: move method back to current place --------- Co-authored-by: Roman Cattaneo <[email protected]> * BREAKING CHANGE: remove ndsl/exceptions (#281) * BREAKING CHANGE: remove ndsl/exceptions The module has been deprecated last release and will be removed with this release. * fixup: documentation --------- Co-authored-by: Roman Cattaneo <[email protected]> * BREAKING CHANGE: remove deprecated environment variables (#282) Those environment variable were deprecated in the last release and will be removed with this release. Co-authored-by: Roman Cattaneo <[email protected]> * ci: specialize concurrency group per repo (#287) * ci: per repo concurrency group Note: using `${{ github.repository }}` sounds like a good idea. In practice, that doesn't play nice when the workflow is called from another repository because in that case, `github.repository` resolves to the calling repository. * fix file ending of called workflows --------- Co-authored-by: Roman Cattaneo <[email protected]> * Remove ndsl.Namelist (#297) * Removing ndsl.Namelist * Removing use_legacy_namelist flag functionality while keeping the flag itself. * - Removing ndsl.Namelist - Removing use_legacy_namelist flag functionality (while keeping the flag itself for now) * linting * Removing namelist.md and test_namelist.py * [feature] Common data types for orchestration via `compiletime` (#296) * `Quantity`, `Local` & `State` default to `dace.compiletime` auto-magically in orchestration * Fix type check, remove `Local` * Unit tests * Fix for type annotations that aren't type * BREAKING CHANGE: remove deprecated ndsl/units.py (#283) The module has been deprecated in the last release and is now removed in this release cycle. Co-authored-by: Roman Cattaneo <[email protected]> * BREAKING CHANGE: removal of extra_dim_lengths (#295) `extra_dim_lengths` on the `GridSizer` was replaced by `data_dimensions` in the `2025.10.00` release. Now that the release is out, let's clean up and remove the deprecated API. This also includes `set_extra_dim_lengths()` in the `QuantityFactory`. Co-authored-by: Roman Cattaneo <[email protected]> * BREAKING CHANGE: remove deprecated ndsl/filesystem.py (#284) The module was deprecated in the last release and will now be remove in this release cycle. Co-authored-by: Roman Cattaneo <[email protected]> * docs: release checklist and documentation (#299) * release checklist and documentation * Add template for patch release * review --------- Co-authored-by: Roman Cattaneo <[email protected]> * gt4py update: fix absolute indexin in debug backend (#302) Co-authored-by: Roman Cattaneo <[email protected]> * column min/max stencil - value and index (#301) * column min/max and a unit test * working unit test, pre-commit changes * alternative type ignore method * reverted previous change * using boilerplate code * reverting previous change * build: gt4py udpdate (fix upcasting, abs k test coverage) (#303) This PR updates GT4Py to bring the following up from GT4Py - fix upcasting such that users can have variable k-offsets with expressions consisting of different types. - increase test coverage for absolute k indexing Co-authored-by: Roman Cattaneo <[email protected]> * restore default PR template (#305) Co-authored-by: Roman Cattaneo <[email protected]> * BREAKING CHANGE: last 2025.10.00 deprecations (`CopyCorners`, `Quantity.values()`, `extra_dim_lengths` on `SubtileGridSizer` (#300) * Remove deprecated extra_dim_lengths of SubtileGridSizer This is a follow-up from #295. * Remove deprecated CopyCorners * Remove deprecated `Quantity.values()` --------- Co-authored-by: Roman Cattaneo <[email protected]> * refactor: remove leftover debug print statements (#308) This PR just removes a bunch of leftover debug print statements from `ndsl/` and `tests/`. Co-authored-by: Roman Cattaneo <[email protected]> * refactor: make GridSizer an abstract base class (#306) `GridSizer` is de-facto already a base class with abstract methods `get_origin()`, `get_extent()`, and `get_shape()`. This PR just formalizes that intent. Co-authored-by: Roman Cattaneo <[email protected]> * refactor: directly use gt_storage in QuantityFactory (#307) In the past, `QuantityFactory` would allow not only allocating with gt4py storage objects, but also directly from `numpy` or `cupy`. This ability was removed in PR #228. With that removal comes the opportunity to streamline allocation in `QuantityFactory`, removing the need for a `Allocator` class in the middle. Co-authored-by: Roman Cattaneo <[email protected]> * [Feature] Schedule Tree: refine transient (#304) * Fix axis merge * Remove debug print * Refine transients + utests * Lint * Revert to deactivating the experimental stree work * Use context manager for `_INTERNAL__SCHEDULE_TREE_OPTIMIZATION` * Typo * Clean refine transients code * Derive common strides layout from backend Refactor code to make re-sizing more compact in main algorithm Fix bad recursion Add todo list and verbose state of optimization * Lint * Remove `transient` to `State` lifetime - keep PR on target * Lint * build: gt4py update (upcasting in cast operations) (#310) This PR updates GT4Py to bring the fix for upcasting inside cast operations from GT4py to NDSL. Co-authored-by: Roman Cattaneo <[email protected]> * build: gt4py update (precision of global constants) (#313) This PR updates GT4Py in NDSL to bring up a PR that fixes the precision of global constants. So far, we'd discard any type annotation on global constants and just use the default literal precision instead. With this change, we respect potential type annotations on global constants. Co-authored-by: Roman Cattaneo <[email protected]> * refactor: Quantity constructor: `gt4py_backend` -> `backend` (#312) * refactor: force kwargs in ctor of Quantity/Local Force keyword arguments for optional arguments to those constructors. This will facilitate the `gt4py_backen` -> `backend` transition. * refactor: prefer `backend` over `gt4py_backend` in Quantity --------- Co-authored-by: Roman Cattaneo <[email protected]> * refactor: prepare `ZarrMonitor` for upcomming `Comm` changes (#315) * refactor: ZarrMonitor: you'll have to bring your own comm objects * ci: run unit tests with optional zarr dependency --------- Co-authored-by: Roman Cattaneo <[email protected]> * Introduce a `single_code_path` flag in the DaCeConfig that forces a single cache to be built. (#311) * refactor: Deprecate optional backend argument to Quantity/Local (#314) Co-authored-by: Roman Cattaneo <[email protected]> * refactor: remove DummyComm as alias to LocalComm (#319) There's no need for this alias. We thus replace all occurrences for the alias with the underlying `LocalComm` directly. Co-authored-by: Roman Cattaneo <[email protected]> * Deprecate `CopyCornersXY` (#317) `CopyCornersXY` are replaced with `CopyCornersX` and `CopyCornersY` in PyFV3. The class is currently unused and will be removed after the next release of NDSL. Co-authored-by: Roman Cattaneo <[email protected]> * refactor: Deprecate `NullComm` in favor of `MPIComm` and `LocalComm` (#318) * unrelated: fix typo in warning message * refactor: change NullComm -> MPIComm in boilerplate This adds a test that the MPI communicator only has one rank if a single-tile setup is requested. * refactor: deprecate NullComm `NullComm` can be replaced with either `LocalComm` or `MPIComm`. --------- Co-authored-by: Roman Cattaneo <[email protected]> * build: gt4py update (self-assignment in serial vertical loops) (#316) This updates the gt4py dependency to bring up the fix that allows self-assignment with offset reads in K for serial (e.g. FORWARD/BACKWARD) vertical loops. See GridTools/gt4py#2388 (in particular the test cases for details on what is allowed and what not). Co-authored-by: Roman Cattaneo <[email protected]> * refactor: specify backend when allocating a Quantity (#320) This PR is a follow-up from #314 and adds the soon to be required `backend` parameter to constructor calls of `Quantity`. I missed a couple ones because PRs were merged in parallel, e.g. re-enabling the `ZarrMonitor` tests. Co-authored-by: Roman Cattaneo <[email protected]> * [Translate test] Compute the percentage of changing grid points that error (#322) * Add `inputs` to MultiModalFloat metric Compute the percentage of changing grid points that errored * Lint * Removing --no_legacy_namelist flag (#323) * Added new functions: column_min_ddim & column_max_ddim and cooresponding test (#324) Functionality is the same as column_min/max, but separate functions are needed to handle cases with off grid data dimensions * [Optimization/Experimental] Better `AxisMerge` for column physics (#325) * Add `CleanUpScheduleTree` pass to prep for merge * Decluter axis merge logs, expose new pass * Verbose Pipeline passes (with temporary stree saves) * Deactivaete IF_SCOPE push, remove attempt to keep merging if next nodes not a MapScope * Docs of TODO * Draft of more extended testing * Fix `CartesianRefineTransients` for non-array * Some lint * Clean up the Tree of ForScope.loop_range * Utest: group test under a single orchestrated class, add missing feature and expected failures * [Feature/Experimental] Stree Refine Transient optimization pass: data dimensions and proper unit tests (#327) * Rename test for axis merge * Properly refine fields with data dimensions Fix indexing in memlets properly * utest: coverage of all implemented tests * Clean up timing print of orchestration * Lint * Fix bad reference to in/out memlets, remopve dead code, better code * Share test infrastructure, rename stencils * Lint * Better naming in utest stencils * [Update] GT4Py & DaCe updated to 2025.11.25 state of `main` (#330) * DaCe update: fix networkx dependency breaking with 3.6 * GT4Py: Runtime interval bounds in `debug` * [Tool] Best Guess Netcdfs diff (#177) * Best guess netcdfs compare * Add FieldBundle to debugger * lint * Move executable to `pyproject` * Lint * Update `gt4py` to capture improvement to user error (#331) * [Rework/Experimental] Refine Transient v2: `Ranges` for all! (#328) * Rework the `RefineTransient` to use `Range` - simpler, cleaner and more robust. Also props us for a better refine * Remove unused code --------- Co-authored-by: Tobias Wicky-Pfund <[email protected]> * [Fix] [Translate] Update API for parallel test when using `MultiModalMetric` (#332) * Remove old options for `MultiModalFloatMetric` * Defensive programming: bail out if we can't measure the ref vs input diff --------- Co-authored-by: Tobias Wicky-Pfund <[email protected]> Co-authored-by: Frank Malatino <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Florian Deconinck <[email protected]> Co-authored-by: Oliver Elbert <[email protected]> Co-authored-by: Christopher Kung <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Roman Cattaneo <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Charles Kropiewnicki <[email protected]> Co-authored-by: Frank Malatino <[email protected]> Co-authored-by: Janice Kim <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Main change
This PR proposes a new tool to check NetCDFs against one another with a series of guess to match compute domains. The tool describe it's logic as it goes down and fail if it can't resolve.
Options for name mapping & halo size.
After (re)installing NDSL, it can be called with
where
name_mappingis a yaml file with a single dictionary mapping name in A to B.Minor QOL
quantity.to_netcdfwhen no rank is given looks at the rank itselfndsl_logis imported early to avoid circular dependencies