+ ifgram_inversion.mask_stack_obs():
   - distinguish between msk and msk_data for more flexibility
   - keep high SNR pixels for offset when masking based on offsetSTD, with an hardwired threshold for now.
   - calc offset SNR via (off - off_med) / off_std

+ utils.readfile.read(): read 4D hdf5 file reading

+ utils.writefile.layout_hdf5/write_hdf5_block(): support 4D matrix wirting to HDF5 file

+ objects/cluster.collect_results(): support 4D matrix

+ ifgram_inversion: calc TS full covariance matrix and save as a 4D matrix in HDF5 file
   - rename --calc-std to --calc-cov
   - rename estimate_timeseries_std() to estimate_timeseries_cov()
   - use the linear propagation of a unweighted estimator for the error propagation
   - use np.linalg.multi_dot() to speedup chain dot operations

+ timeseries2velocity.py: add --ts-cov-file option to support full cov matrix for error propagation
   - rename --ts-std-file to --ts-cov-file
   - support both 4D full covar and 3D var time series file as input
   - apply the linear propagation for ordinary least squares, to be consistent with the time func estimation. Previously, the weighted least squares version is used in this step, which is not appropriate.
   - update notes on the math equations in the comments.

+ readfile: support reading UAVSAR .ann/lkv/llh files
   - add read_uavsar_ann() to read the *.ann file into an dict object
   - read_attribute(): add metadata preparation for *.lkv and *.llh file
   - get_slice_list(): provide explicit list of dataset names for *.lkv and *.llh files
   - read_binary_file(): support multiple bands for non-traditional-InSAR processors.

+ prep_isce: add generate_random_baseline_timeseries() to support `-b random` input to generate a bperp time series with random values as place holders for mintpy processing.

+ solid_earth_tides: support T in time [for UAVSAR]

+ readfile/writefile: use numpy data type string as standard for mintpy, and use it for the convention translation during data I/O.
   - DATA_TYPE_ENVI2NUMPY
   - DATA_TYPE_NUMPY2ENVI
   - DATA_TYPE_GDAL2NUMPY
   - DATA_TYPE_NUMPY2GDAL
   - DATA_TYPE_ISCE2NUMPY
   - DATA_TYPE_NUMPY2ISCE

+ ifg_inv: use diff name for numInvOffRg/Az.h5 files

Add utils0.auto_lat_lon_step_size() to calculate the default output lat/lon steps for geocoding.

By default, the current version ensure the same number of rows and columns between input and output files; while this new version ensures the same pixel area between the input and output files and with proper dimension ratio using the bounding box of an rotated rectangle. Thus, this is a better version, especially when the azimuth angle is far from the pure north-south, such as the UAVSAR flight path, or when the space-borne SAR track is long.

+ readfile.read_attribute: move GDAL for .tif/.grd file up on the priority than non-rsc metadata files, to fix reading issue related to downloaded hyp3 *.tif file

+ docs: add notes on preparing hyp3 interferograms via Vertex website

+ circleci/config: switch circle CI resource class from medium to large

+ tests/config/*.txt: increase the max plot memory from 2G to 4G

+ smallbaselineApp: bugfix for --plot when installed via pip/conda

+ add_attribute: bugfix for input argv checking when installed via pip/conda

* Minor typo in dask documentation

…ines (#693)

+ `plot_network`: support color-coding temporal and perpendicular baseline via `-d tbase/pbase` and range control still via `-v` option.

+ utils1.get_residual_std(): add `std_file` variable to the return

Co-authored-by: Zhang Yunjun <[email protected]>

+ load_data.py: fix a runtime bug when re-run after removing the ISCE stack processing folders

+ plot_network: fix the output file names bug

+ docs/install: a little bit more notes on dask

+ fix the `AttributeError: 'AxesSubplot' error, because `fig` can not be used as return for subplot() anymore.

+ image_stitch: improved auto plot settings

Co-authored-by: Zhang Yunjun <[email protected]>

When running `timeseries2velocity.py` with bootstrapping, we use the function `time_func.estimate_time_func` to repeatedly fit the time series data. This function returns the design matrix, `G`,  however when bootstrapping we don't assign the design matrix to a variable, meaning that if we try to save the residual file we get the error `UnboundLocalError: local variable 'G' referenced before assignment` at line 702.

If we want to output the residual when bootstrapping, we need to get the full design matrix. I've added a line of code to do this.

+ reference_point: add a validity checking for input `ref_lat` value

+ info: --num will turn ON --date
  - --num will turn ON --date, so that one can run info.py timeseries.h5 --num to print date list and number, without adding --date option, for conveniency.
  - remove --slice --num combo, as it's not used much, to simplify the code.

+ sensor: use rs1/rs2/rcm for radarsat series

+ identify SLC stack as "FILE_TYPE==timeseries" and "DATA_TYPE==complex(64/128)"

+ stack.timeseries.open): change CENTER_LINE_UTC from required to optional as SLC stack might not have it

+ readfile.read_attribute(): check SLC stack and reset UNIT to 1

+ coord.geo2radar(): ensure the consistency between point longitude and source file, regardless of the source file convention in (-180, 180] or [0, 360) conventions. For geocoded files only.

+ plot:
   - auto_colormap_name(): use gray as default cmap for slc stack
   - auto_adjust_xaxis_date(): smart every_year/month for extra long time series, e.g. GRACE products.

+ view:
   - read_data4figure(): take abs() if data type is complex
   - move code to form check_map_projection() for easy re-use, e.g. in tsview.py

+ tsview:
   - pass `disp_unit` directly for the model estimation result display, for more simple and robust performance against input files that are not in the unit of meters, e.g. giant product or GRACE products.
   - grab WAVELENGTH if and only if wrap-range is 2 pi. This allows applying --wrap-range option to files w/o WAVELENGTH attributes, e.g. the GRACE products.
   - support cartopy map projection and its related features, e.g. --coastline, via view.check_map_projection()
   - take abs() if data is complex
   - bugfix for --flip-lr/ud options

+ rename `legacy/load_hdf5.py` to `legacy/load2hdf5.py`
+ refactor to be more flexible on the input/output file structure
+ add --sub-x/y
+ add --meta to add custom metadata
+ support --dname timeseries with "date" dataset added automatically
+ add DATA_TYPE_STR2OBJ to support most common numpy data types
+ check if the output file exists with --force option

+ add a new standard metadata: NO_DATA_VALUE

+ readfile.py:
   - read_attribute(): grab NO_DATA_VALUE with 'none' as default
   - add get_no_data_value() to get the value of the input file
   - add SPECIAL_STR2NUM() to facilitate no_data_value handling
   - read_isce_xml() / read_gdal_vrt(): grab NoDataValue if exists.

+ view.py: mask out pixels with no_data_value for single subplot

+ tests/config/*.txt: speedup testing by decrease plot.maxMemory to 2GB to use more parallel processes for plotting

+ `add.py` & `diff.py`: generic multi-dataset file support, by searching the common dataset list among all the input files, and doing a for loop for them. Ignore dataset names if all input files have one dataset, to support single dataset files with different dataset names, e.g. mask, mask and waterMask. In the later case, the 1st file dataset name will be used as the reference.

+ `utils1.run_deramp()`: when operating on velocity files, operate on velocity dset as default

Co-authored-by: Zhang Yunjun <[email protected]>

… dict (#737)

+ merge `lut_meta` and `src_meta` and pass to `utils0.auto_lat_lon_step_size()` for a better chance of having enough metadata to calculate the pixel area constrained new lat/lon step.

+ fix the mis-functioned `try/except` fallback logic bug by using `except KeyError:` instead of `except AttributeError:`.

+ docs/requirements: remove `h5py<3` constrain

+ fix bool array index bug for `h5py>3` in the following scripts:
   - objects/stack.py
   - utils/readfile.py

+ timeseries2velocity: correct typo in example string

+ add `--geom-only` option for only creating geometry (useful for making masks)

+ add `-r / --range` and `-a / --azimuth` option to specify looks (this is only for tracking metadata), if extra multilooking is applied after fringe processing.

+ create `inputs/ifgramStack.h5` file if *.unw.conncomp files exist. The non-first-date reference date is supported when writing ifgramStack to be more agnostic against the potential change in fringe output in the future.

+ view.py: add --no-data-value option to manually sppecify the value to ignore, for plotting only
+ plot.plot_colorbar(): a more robust cbar extend style checking, to ignore the numerical precision level value leakage.

+ generate_mask.py: add --vs / vy / vroipoly options to specify the AOI for thresholding operations, the area outside will be untouched. This is usef to create an mask, without pixels on the fault, for phase unwrapping of co-seismic deformation.

+ two 2-band *.unw file IO in the following scripts, instead of single band for phase only, to preserve the native *.unw file format:
   - mask.py
   - reference_point.py
   - tropo_pyaps3.py

+ptime.(yy)yymmdd_date12(): support both str and list type inputs

+ rename prep_snap.get_ellpsoid_local_radius() to utils.utils0.xyz_to_local_radius()

+ utils.readfile:
   - add read_snap_dim() to read SNAP *.dim file with XML modules, instead of searching/parsing in text file in the old prep_snap.py, for simpler, more robust and faster performance.
   - read_isce_xml(): 1) add if check for coordinate1/2.delta/startingvalue; 2) force lower() case key names, to produce consistent output

+ significantly simplify prep_snap.py

Save timeseries_res: instantiate the output hdf5 file accounting for the excluded dates

Co-authored-by: Zhang Yunjun <[email protected]>

+ support input files with multiple datasets, e.g. geometry files.

+ If input files do not have the exact same list of datasets, only the common elements of datasets will be used.

+ For geometry files, ignore zero incidenceAngle pixels in the geometry file, so that one could feed them without masking in advance.

+ show all figures at the end, for a smoother experience

+ utils/utils0.four_corners(): switch return order from WESN to SNWE, for consistency. Adjust its usage in the following scripts:
   - asc_desc2horz_vert.py
   - save_kmz.py
   - gps.py

+ gps.get_gps_los_obs():
   - use "meta" and "obs_type" to replace "insar_file" for input arguments, for more explicit and straightforward usage.
   - add example function usage in the comments.
   - GPS.get_gps_los_velocity(): bugfix for more robust performance

+ utils.plot.plot_gps(): bugfix when no GPS found.

+ plot_transection:
   - bugfix for the pip installed version
   - support transection with distance unit in pixel.

+ utils.utils.transect_yx(): add "distance_unit" to the output dict object, with default value of "m". In case there is not enough metadata to calculate the ground pixel size in meters, such as the raw isce-2 files, set "distance_unit = pixel" and continue.

+ utils.readfile.py:
   - read_snap_dim(): force int dtype for A/RLOOKS
   - remove TRACK_NUMBER from standardMetadata as it has not been tracked before, and has already been saved in the original snap metadata info.

+ simulation/variance.sample_data(): use np.random.default_rng()

+ remove "python<3.9" constraint from installation-related documents:
   - setup.py
   - docs/environment.yml
   - docs/requirements.txt

+ docs/installation.md:
   - add note on strict conda channel priority activation
   - remove the strict channel priority note, as it's currently not default in conda yet and NOT working all the time, especially when install mintpy together with isce2=2.6

+ .circleci/config.yml:
   - activate the strict channel priority to avoid installation packages with conflicts versions
   - remove unnecessary cd command while runing testing on example datasets as it's done inside test scripts already

+ with the above changes, $MINTPY_HOME/tests/test_smallbaselineApp.py passed without errors on both python 3.9 and 3.10 environments

+ add mintpy.utils.constants module [preliminary]

…ccupied by covariance matrix (#750)

+ cluster.DaskCluster.collect_result(): support None in the output list, as an alternative to to numpy matrix, to be more flexible

+ stack.ifgramStack.get_design_matrix4timeseries():
   - support non-sequential network (where the first date is not always the earlier date)
   - use the same reference date by default for a single reference network

+ ifgram_inversion.py:
   - estimate_timeseries(): comment out the B matrix inverrtability check, after correcting the B matrix bug above for non-sequential network
   - ifgram_inversion_patch(): set ts_cov = None if not calc_cov, to release memory from the huge 4D ts_cov matrix

Co-authored-by: Zhang Yunjun <[email protected]>

…ias (#740)

+ add a new script `closure_phase_bias.py`, to create a mask for areas susceptible to closure phase bias, based on Zheng et al. (2022, TGRS, under review).

+ setup.py: add entry point for the new script