`diff.py` btw. time-series and velocity + other changes #1018

yuankailiu · 2023-06-08T23:29:54Z

Description of proposed changes

Several modifications.

diff.py: does (timeseries) - (velocity) = (timeseries_new)
- implemented by function diff_timeseries_velocity()
- this commit is motivated by doing correction based on inputs/ITRF.h5 velocity model in the time-series domain. Thus, the commit only allows a time function model m to identify linear, annual, and semiannual functions. Later, we can write a new ts2velo.hdf5_dataset2model() to produce this m based on the dataset_list names in general velocity files.
spatial_filter.py: add a median filter from skimage.filters.median
image_stitch.py: allow for vmin/vmax, cmap, and display unit
plot.py: numTriNonzeroIntAmbiguity.h5 mask out the non-zero values (non-zero is bad pixels) for mask creating
gps.py object: GPS time function fit with a user-defined model

Reminders

Pass Pre-commit check (green)
Pass Codacy code review (green)
Pass Circle CI test (green)
Make sure that your code follows our style. Use the other functions/files as a basis.
If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
If adding new functionality, add a detailed description to the documentation and/or an example.

+ will read the velocity file and re-construct the timeseries + take care of referencing + do the timeseries diff as usual + To-Do: this commit is motivated for doing correction based on ITRF.h5 velocity model in the time-series domain. Thus, the commit only provides a time function model `m` hard-coded as a linear function. Later, we can write a new ts2velo.hdf5_dataset2model() to produce this m based on the dataset_list names in general velocity files.

+ add median filter from skimage.filters.median + handle NaN holes in lowpass gaussian filter

+ allow for input argument --vmin, --vmax for plotting the result plotting

+ Kai: handle 0 in when 'numTriNonzeroIntAmbiguity.h5' as mask_file. 0 of this file is good. Non-zero is bad

+ change --unit to --scale to be more accurate + merge --vmin/vmax into -v / --vlim option, to be consistent with view.py + use the same name in argparse and sub-functions + scale the calculated vmin/vmax from the data with the same factor.

+ rename `diff_timeseries_velocity()` to `diff_timeseries_and_velocity()` for consistency + remove unit_fac, which is for giant, and not applicable here + use one date_list as multiple date lists and their common elements are not applicable here + use REF_DATE from file1 only to check if temporal referencing is needed or not, as no reference date info is available from the velocity dataset, even though REF_DATE metadata may be available in file2. + remove periodic / step functions from the model construction, as their corresponding datasets are not read form the h5 file, for simplicity

yunjunz

Thank you @yuankailiu for the useful PR. For the diff.py time-series and velocity functionality, I modified the code on top of yours. A test based on the SanFranSenDT42 example dataset between timeseries.h5 and velocity.h5 shows a reasonable result, but I don't have a large dataset at hand, could you re-confirm the effectiveness of plate motion correction?

For the other changes, I left a few minor comments below. Take a look and let me know what you think please.

src/mintpy/utils/plot.py

src/mintpy/spatial_filter.py

src/mintpy/objects/gps.py

yuankailiu · 2023-06-11T07:20:59Z

Here is the demonstraion of the correction done in the time-series domain. It is effective.

# original time series (with SET, ERA5, Ion, and topo residual corrections) fitted with time functions 
timeseries2velocity.py timeseries_SET_ERA5_Ion_demErr.h5 --poly-order 1 --periodic 1.0 0.5 -o velocity_test1.h5 --ref-lalo 29.5463 36.0810 --ref-date 20190118

# Difference that with the ITRF14 velocity file of the Arabian plate (i.e., apply plate motion correction)
diff.py timeseries_SET_ERA5_Ion_demErr.h5 inputs/ITRF14.h5

# Fit the new time-series file `timeseries_SET_ERA5_Ion_demErr_diff_ITRF14.h5` with time functions
timeseries2velocity.py timeseries_SET_ERA5_Ion_demErr_diff_ITRF14.h5 --poly-order 1 --periodic 1.0 0.5 -o velocity_test1_ITRF.h5 --ref-lalo 29.5463 36.0810 --ref-date 20190118

# plot the original veloctiy
view.py velocity_test1.h5 velocity -c RdYlBu_r --dem ./inputs/srtm.dem --alpha 0.8 --dem-nocontour --shade-exag 0.02 --shade-min -6000 --shade-max 4000 --mask ./maskTempCoh_0.9.h5 --unit mm --ref-lalo 29.5463 36.0810 --vlim -5 5 --nodisplay --dpi 300 --figtitle no_ITRF_cor

# plot the plate motion corrected velocity
view.py velocity_test1_ITRF.h5 velocity -c RdYlBu_r --dem ./inputs/srtm.dem --alpha 0.8 --dem-nocontour --shade-exag 0.02 --shade-min -6000 --shade-max 4000 --mask ./maskTempCoh_0.9.h5 --unit mm --ref-lalo 29.5463 36.0810 --vlim -5 5 --nodisplay --dpi 300 --figtitle with_ITRF_cor

# plot the ITRF14 plate motion velocity itself
view.py inputs/ITRF14.h5 velocity -c RdYlBu_r --dem ./inputs/srtm.dem --alpha 0.8 --dem-nocontour --shade-exag 0.02 --shade-min -6000 --shade-max 4000 --mask ./maskTempCoh_0.9.h5 --unit mm --ref-lalo 29.5463 36.0810 --vlim -5 5 --nodisplay --dpi 300 --figtitle ITRF_cor

+ separate mask (output) and mask_data (data read from the mask file), for clarity + if input is numTriNonzeroIntAmbiguity.h5 file, keep pixels with value of 0, and mask out the rest, only if no --mask-vmin/vmax option is specified, as they should and do have higher priority than the default translation of numTriNonzeroIntAmbiguity. + remove prior_good variable, and move all the operations in the later stage of the code, for clarity

yunjunz · 2023-06-12T01:31:10Z

Looks all good to me.

yuankailiu · 2023-06-12T16:02:38Z

Looks all good to me.

Thanks! Appreciate for reviewing and refactoring them! I will merge it.

scottstanie · 2023-06-17T22:58:49Z

I saw that you added the nice way to ignore nans here, but that it's only for lowpass gaussian right now. Sorry I didn't see this in time, but what if we changed this

MintPy/src/mintpy/spatial_filter.py

Lines 68 to 70 in 6fffa3b

    
           elif filter_type == "highpass_gaussian": 
        
               lp_data = filters.gaussian(data, sigma=filter_par) 
        
               data_filt = data - lp_data

to be

    elif filter_type == "highpass_gaussian":
        return data - filter_data(data, "lowpass_gaussian", filter_par=filter_par):

so we don't have to redo the logic twice?

yunjunz · 2023-06-18T03:02:28Z

Good catch @scottstanie. I missed it as well. Could you issue a PR for it?

referenced here insarlab#1018 (comment)

…1030) by leveraging the implementation from the `lowpass_gaussian` filtering in #1018.

yuankailiu added 5 commits June 8, 2023 16:16

spatial_filter: add median filter

72be6f3

+ add median filter from skimage.filters.median + handle NaN holes in lowpass gaussian filter

image_stitch: allow for vmin/vmax

795cb09

+ allow for input argument --vmin, --vmax for plotting the result plotting

plot.py: numTriNonzeroIntAmbiguity.h5 maskout non-zero pixels

0ce852c

+ Kai: handle 0 in when 'numTriNonzeroIntAmbiguity.h5' as mask_file. 0 of this file is good. Non-zero is bad

gps time function fit with user-defined model

61ec977

yuankailiu changed the title ~~Dev.diff~~ diff.py and other changes Jun 8, 2023

codacy fixes

91dc375

yunjunz self-requested a review June 9, 2023 03:23

image_stitch: adjust opt names

032d8f5

+ change --unit to --scale to be more accurate + merge --vmin/vmax into -v / --vlim option, to be consistent with view.py + use the same name in argparse and sub-functions + scale the calculated vmin/vmax from the data with the same factor.

yunjunz changed the title ~~diff.py and other changes~~ diff.py btw. time-series and velocity + other changes Jun 10, 2023

yunjunz added 3 commits June 10, 2023 15:41

diff: add example usage

68426b6

codacy fix

f537bbc

yunjunz reviewed Jun 10, 2023

View reviewed changes

src/mintpy/utils/plot.py Outdated Show resolved Hide resolved

src/mintpy/spatial_filter.py Outdated Show resolved Hide resolved

src/mintpy/objects/gps.py Outdated Show resolved Hide resolved

Update src/mintpy/objects/gps.py

99df9b7

yunjunz and others added 3 commits June 11, 2023 21:07

codacy fixes

ab468f4

lowpass_gaussian: fix RuntimeWarning divided by zeros, add comments

1559018

yunjunz approved these changes Jun 12, 2023

View reviewed changes

yuankailiu merged commit ea06ec2 into insarlab:main Jun 12, 2023

yuankailiu deleted the dev.diff branch June 12, 2023 16:15

scottstanie added a commit to scottstanie/MintPy that referenced this pull request Jun 20, 2023

spatial_filter.py: use nan-ignoring filter for highpass_gaussian

2e5792e

referenced here insarlab#1018 (comment)

scottstanie mentioned this pull request Jun 20, 2023

spatial_filter.py: use nan-ignoring filter for highpass_gaussian #1030

Merged

5 tasks

yunjunz pushed a commit that referenced this pull request Jun 21, 2023

spatial_filter.py: use nan-ignoring filter for highpass_gaussian (#…

4e14bde

…1030) by leveraging the implementation from the `lowpass_gaussian` filtering in #1018.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`diff.py` btw. time-series and velocity + other changes #1018

`diff.py` btw. time-series and velocity + other changes #1018

Uh oh!

yuankailiu commented Jun 8, 2023 •

edited by yunjunz

Loading

Uh oh!

yunjunz left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuankailiu commented Jun 11, 2023

Uh oh!

yunjunz commented Jun 12, 2023

Uh oh!

yuankailiu commented Jun 12, 2023

Uh oh!

scottstanie commented Jun 17, 2023

Uh oh!

yunjunz commented Jun 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diff.py btw. time-series and velocity + other changes #1018

diff.py btw. time-series and velocity + other changes #1018

Uh oh!

Conversation

yuankailiu commented Jun 8, 2023 • edited by yunjunz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yunjunz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuankailiu commented Jun 11, 2023

Uh oh!

yunjunz commented Jun 12, 2023

Uh oh!

yuankailiu commented Jun 12, 2023

Uh oh!

scottstanie commented Jun 17, 2023

Uh oh!

yunjunz commented Jun 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`diff.py` btw. time-series and velocity + other changes #1018

`diff.py` btw. time-series and velocity + other changes #1018

yuankailiu commented Jun 8, 2023 •

edited by yunjunz

Loading

yunjunz left a comment •

edited

Loading