Post 3.9.5 release reset

Revise the release documentation

add a test trying to reproduce #2628

Followup to #2333, small step toward replacing mfstructure.py

More followup on #2333, store the new structured spec on the generated classes beside the old one. This allows gradual substitution and replacement of mfstructure.py. Don't replace the generated flopy.mf6.modflow module yet, this just clears the way for development

Continuing the effort begun in #2333 and #2630 to replace mfstructure.py, starting with some cleanup to clarify what is going on

* remove unused methods
* make internals internal
* just inline some stuff
* some terser naming
* consolidate classes
* tidy up docstrings

Accommodate attribute "prerelease" rename to "developmode" in MODFLOW-ORG/modflow6#2556

#2636)

Rehome cell by cell budget comparison function from MF6 autotests. Rename the old compare_budget to compare_list_budget to clarify but leave an alias to compare_budget.

Deprecate set_budget_key() and just pass the appropriate budgetkey for each subclass to the parent's __init__() instead. The budgetkey can still be overridden through the initializer or directly setting the attribute as before.

fix bug where open/close fails on one cell in a SPD and support pathlib.Path in MFlist. New tests added.

---------

Co-authored-by: Mike Taves <[email protected]>

Add vectorized array support to the intersect() method on all grid types

…2656)

- Fixes TypeError when loading MFUSG models with WELLBOT option
- Adds static method get_empty() that accepts wellbot parameter
- WELLBOT option was already parsed in pakbase.py but not handled in dtype creation
- Enables loading of USG-Transport models with well bottom elevations

Resolves issue where models using 'WELLBOT' option in WEL package would fail with: TypeError: ModflowWel.get_empty() got an unexpected keyword argument 'wellbot'

…2659)

When loading a model, DATA entries from the NAM file were being added
to external_units even when they already existed in output_units
(e.g., IUNITAFR from WEL package). This caused duplicate entries
in the written NAM file.

Added check for output_units in _prepare_external_files() to prevent
adding unit numbers that are already registered as output files.

#2658)

Fix precision loss in MFUSG's CLN and GNC packages. Check if free format is enabled in the BAS file; if so, use %16.9G for full precision, otherwise, fall back to %10.2e for compatibility with fixed-width input.

Add upper bounds to numpy and pandas in pyproject.toml, and remove --prerelease-allow from uv sync command in smoke tests. CI was failing because these two factors combined to allow prerelease versions of numpy and pandas, which broke some tests. The intent behind --allow-prerelease was to get a prerelease version of vtk on ARM macs, but seems a full version is now available on PyPI.

I think this might clear the way to switch CI to pixi if we want, as described in #2424

…2665)

Fix #2420, only internal arrays in this PR. Introduce a single underlying variable to track whether max_columns_of_data has been set automatically or by the user. Keep the 2 existing user-facing flags for backwards-compatibility, but reimplemented as properties to reflect the neater internal management. This should eliminate the need to ever set these.

And reproduce the external array problem. There are tricky interactions between whether max_columns_of_data is set first or data is set external first, and whether the simulation workspace has been changed. Currently the column setting is only respected for external arrays if the simulation workspace has been changed and the column setting is set before setting data external. Ideally the setting would be respected regardless of order of ops or whether the workspace is different.

Close #2647, correcting initial buggy implementation of #2270. CellBudgetFile and HeadFile will both take padded and properly-shaped cell IDs now. Improved docstrings and expanded tests.

…2666)

Add an optional parameter replace_existing to set_all_data_external() methods on the object model classes, defaulting False. Like the replace_existing_external parameters on the store_as_external_file() routines on the data model classes, except those default True, where defaulting False here preserves laziness, i.e. not rewriting external files if they already exist, at the package/model/etc level, while allowing opting into eager overwriting.

This decision prioritizes backwards-compatibility and IO performance over respecting user IO settings. I considered doing the opposite, for consistency with store_as_external_file() and since not respecting user IO settings could be considered a bug, but decided this was safer and is at least a partial improvement. Also considered a mechanism to track whether user settings and/or workspace and/or data have changed, and determine whether to overwrite intelligently, but decided to punt to 4.x due to the complexity involved.

As mentioned in #2419, external files are written on set_all_data_external() instead of simply configured to be external, then only written on write_simulation(). This PR does nothing to address that. I have looked into a fix but again, it would be invasive so will tentatively punt to 4.x.

Add an optional parameter replace=False to the .set_data() methods on the stress period data model. Setting this True resets the data before setting provided keys, i.e. clears all preexisting entries rather than leaving them in place.

Since stress period configuration is not stored solely in the underlying data model but also in package block headers, a trap is necessary in MFBlock.write() to sync block headers to the keys existing in the period data, to avoid writing empty/spurious period blocks.

Suggested in #2663 (comment)

Previously the Grid.get_cell_vertices() method looked/worked different depending on the grid type. Make it consistent across all three grid types by supporting both cellid and node keyword arguments and being lenient about what's passed to cellid i.e. accept a node number, a fully-sized tuple, or an "abbreviated" tuple lacking layer index. Cell ID and node number are conceptually distinct but til now VertexGrid and UnstructuredGrid methods had a single parameter cellid supporting (for vertex) or expecting (for unstructured) a node number, which we don't want to break. And continue also supporting i and j for StructuredGrid.

* test all grid types
* support vertical barriers

Util2d assumes its first argument is a model, but a ParticleGroup was being passed. This caused an error when writing MP7 input files with certain release configurations. Don't pass ParticleGroup to Util2d as it's not needed, and only check self.model in Util2d if it's not None. Likewise in the ArrayFormat class.

…rt for points with z-coordinates (#2657)

Support for numpy arrays of shapely geometries
----------------------------------------------

Introduces support for arrays of shapely geometries in GridIntersect.intersects() method. Returns rec-array with field containing the index of the original shape and the cellids of the candidate cells the polygons intersect with.

Fast point locating
-------------------

GridIntersect was missing a method to obtain one cellid for each input point. The implementation of GridIntersect.intersects() drops all points outside the model grid, and potentially returns all intersection results for each point, meaning there is no 1:1 output between input points and output cellids. The GridIntersect.intersect() method allows a MultiPoint as input, but is inefficient, and groups points in the same cell as MultiPoints, also losing the 1:1 relation.

So a new method is introduced GridIntersect.points_to_cellids(). This method guarantees len(input) == len(output). Cellids are returned for each point and NaN is returned if the point is outside the model grid. For points on boundaries the lowest grid cellid is returned.

Support for points with z-coordinates
-------------------------------------

Adds support for points with z-coordinates in GridIntersect.intersect() and GridIntersect.points_to_cellids() with the handle_z-kwarg.

* The default is "ignore" which ignores the z-coordinates, similar to what GridIntersect did before.
* The second option is "drop" which removes the intersection results if the point lies above or below the model grid. This option is only included in intersect().
* The final option is "return" which returns the computed layer position for each point. This returns NaN if the point is outside the model domain, +inf if the point is above the model and -inf the point is below the model.

Deprecations
------------

Removes everything related to method="structured" which was previously deprecated

Other stuff
-----------

* Cleaned up the module somewhat with some renaming since the structured methods dropped out
* Used shapely plotting routines for linestrings and points to simplify plotting calls, make static plot methods accept shapely geometries instead of rec-array results so they can also be used to plot input geometries.
* Update tests to test new features and clean up old tests
* Update example notebook.

Add a get_node() method to VertexGrid and UnstructuredGrid, consistent with StructuredGrid, for cell ID -> node number conversions. Rename the lrc_list parameter on StructuredGrid.get_node() to cellids and deprecate lrc_list. Add an optional node2d parameter to determine whether to get the 2D or 3D node number, useful for structured and vertex grids, ignored for unstructured.

The cell ID -> node number conversion was done ad hoc in a few places in the codebase, this allows some simplification. Maybe in 4.x we can standardize the signature and pull an abstract method into the base Grid class.

Add ihc (horizontal connection) init parameter and property to UnstructuredGrid and populate it from the DISU package when initializing MFModel.modelgrid. Then use it for vertical barrier detection in plotutil.py instead of floating point comparisons on coordinates, if available (but don't assume it's available).

…#2671)

EVT packages with nseg > 1, and auxiliary variables, failed on file write with:

  IndexError: invalid index (7)
  MFDataException: An error occurred while processing auxiliary variables in the "_get_file_entry_record" method.

When nseg > 1, the EVT package has optional fields pxdp, petm, petm0 that may or may not be present in the structured array depending on nseg and surf_rate_specified values. #2496 added _dependent_opt() and _optional_nval() methods to handle these fields when reading, but _get_file_entry_record() in mfdatalist.py still iterated through all DFN-defined fields when writing. The index then advanced past the array bounds before reaching auxiliary variables.

At write time, skip EVT-specific optional fields that aren't present in the structured array, using the same methods #2496 added for reading.

Close #2683

Follow up on #2611 to support DISU. Closes #2170.

Plus forward-looking refactoring in case we add more grid types before 4.x comes out. Rather than conditionals everywhere to access/modify grid properties differently based on the grid type, add an adapter exposing what's needed consistently. This is necessary because grid properties may have different shapes depending on the grid type. In 4.x this can be avoided by design I think. But for now this makes the essence of the transmissivities calculation clearer.

Reorder the transform to accommodate opposite y axis conventions. This matches the order in export/utils.py::export_array(). Close #2553

* address review comments and support plotAll with subset
* Add assertions for the number of subset bcs

…utine (#2691)

…2690)

Close #2610

Apply np.atleast_1d() to r and c for consistency with x and y. Previously, passing single values for r and c resulted in IndexError.

Until we can make sure we've accommodated the changes. The pyproject.toml already has an upper bound

Test the hanging node check on Voronoi grids. Mention that failure is possible with grids that have artifacts like near-duplicate vertices. Add an optional non-convergence check to the method. I figure it could be always on, as it only triggers if no progress is made by the algorithm, but not 100% sure about this, so kept it optional. Close #2427

… free-format array output (#2696)

When free_format_npl is set (e.g., model.free_format_npl = 10), free-format
arrays are written with that many values per line instead of all values on a
single line. This produces block-format output matching Groundwater Vistas
style, improving readability for large unstructured models.

Affects both INTERNAL arrays (Util2d.string) and EXTERNAL/OPENCLOSE arrays
(Util2d.get_file_entry). Default is None (no change in behavior).

Close #1502. The issue is that get_alldata() assumes regular structure that USG files don't have. We could return e.g. a list of arrays or some other ragged representation but I figure better to acknowledge the limitation clearly than to return something technically valid but confusing. So, block usage of the method without providing a layer.

Improve the shared face finding algorithm for HFB (Horizontal Flow Barrier) plotting. Factor out @jlarsen-usgs's index-based approach from #2671 into reusable function(s) and use it instead of floating point comparisons as before. Should be faster.

I moved the new face-related functions to a new module faceutil.py instead of plotutil.py as they are used not only for plotting but for export, and may be useful in other cases. I considered putting them in gridutil.py but that is used by the grid classes, while the face functions use the grid classes, so it would have created a circular import.

I also changed the name of is_vertical_barrier -> is_vertical and inverted the semantics so it works on faces, not barriers (a face between horizontally adjacent cells is a vertical polygon, and a face between vertically adjacent cells is a horizontal polygon, but we call the former a "horizontal"  flow barrier and the latter a "vertical" flow barrier). I figure writing these utils in terms of faces for the general case makes sense, and just interpreting the results for the specific case of HFBs.

This PR builds on

- #2671's shared face finding approach
- #2680 which gives all grid types a consistent get_node() method
- #2681 which adds an ihc init parameter/attribute to UnstructuredGrid

Some ad hoc cell ID -> node number conversion utilities are removed and get_node() used instead.

I am classifying all this a refactor though it introduces a new utility module, as the new utils are in service of the refactor.

Handle time-indexed and period-indexed transient array data independently. TAS data uses float keys (0.0, 6.0, 12.0) representing actual simulation times. Standard transient data uses integer keys (0, 1, 2) representing stress period indices. Only the latter were supported before.

Also allow setting array data of unknown shape (like TAS) external, this was not supported before.

Fix #2702.

As suggested in #2702 (comment). Numbering 1-based as elsewhere

sync mf6 module to https://github.com/MODFLOW-ORG/modflow6/releases/tag/6.7.0, also remove the spec property from generated classes (added in #2631 thinking we could shift usage to it from the unstructured dfn property but until we actually commit to doing that, YAGNI)

… update changelog