integrate ionospheric correction into smallbaselineApp

[yunjunz]

Description of proposed changes

This PR integrates the ionospheric correction into the routine workflow of smallbaselineApp via a new step correct_ionosphere. The detailed changes are as below:

• smallbaselineApp: add a new correct_ionosphere step with the following new template options, to support ionospheric correction using split spectrum results from isce2 stack processing

  • mintpy.ionosphericDelay.method
  • mintpy.ionosphericDelay.excludeDate
  • mintpy.ionosphericDelay.excludeDate12

• add cli/iono_split_spectrum.py for the cmd interface interaction

• add iono_split_spectrum.py for the actual workflow of estimating/correcting iono time-series by calling the following individual scripts:

  • modify_network
  • reference_point
  • ifgram_inversion
  • diff

• add the new script to the entrypoint and sub parsers via changes in the following scripts:

  • pyproject.toml
  • src/mintpy/__main__.py

• defaults/template.py: add correct_ionosphere to the STEP_LIST

Reminders

• Fix https://groups.google.com/g/mintpy/c/So8Ohn-EAPM
• Pass Pre-commit check (green)
• Pass Codacy code review (green)
• Pass Circle CI test (green)
• Pass local test (green)
• Make sure that your code follows our style. Use the other functions/files as a basis.
• If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
• If adding new functionality, add a detailed description to the documentation and/or an example.

[yunjunz]

Hi @yuankailiu and @hfattahi, I finally got some time to integrate the ionospheric correction into smallbaselineApp. Could you take a look at the PR when you get a chance?

I have tested this locally for a stack of alos2 interferograms with ionospheric corrections using the following template options, processed with alosStack, and everything looks fine to me. @yuankailiu could you test it with a stack from topsStack?

mintpy.ionosphericDelay.method          = split_spectrum
mintpy.ionosphericDelay.excludeDate12   = 190313-210728,190424-210728

[yuankailiu]

Great! All looks good to me. I've tested from my side with a stack of ionosphere pairs ionStack.h5 processed from stackSentinel.py.

iono_split_spectrum.py -t inputs/smallbaselineApp_YunjunIon.cfg -f timeseries.h5

where my config is here:

mintpy.ionosphericDelay.method        = split_spectrum  #[split_spectrum / no], auto for no
mintpy.ionosphericDelay.excludeDate   = auto  #[20080520,20090817 / no], auto for no
mintpy.ionosphericDelay.excludeDate12 = 20141012-20141024,20220608-20220702   #[20080520_20090817 / no], auto for no

So the script dropped two pairs and run the inversion with no SBAS weights (ifgram_inversion.py ... -w no). It created the following displacement files: ion.h5, timeseries_ion.h5. And the inversion quality files: temporalCoherenceIon.h5, numInvIon.h5.

Also, as we chatted, theoretical corrections (external model-based) come before empirical corrections (data-driven). Thus you put iono correction at step after SET and troposphere steps, but before the estimation of ramps and topographic residuals.

It looks like everything works as expected! @yunjunz , thank you for this long waited PR so that we don't need to manually run scripts one-by-one.

[yuankailiu]

A logistical discussion:

For convenience, one thing in the future we can consider is to support importing ionosphere timeseries from stack processor outputs directly? I am not sure. This is just an idea.

At least for topsStack, as what I know, Cunren's code produces a stack of ionosphere pairs and also inverts them in isce2 environment (a simple SBAS inversion without weight just like here). There, we are adviced to plot those pairs with e.g., plotIonPairs.py -idir ion -odir ion_plot to inspect for the bad pairs and drop them before inversion. The inversion is the same, but the plotting function there is based on mdx, thus much faster than view.py here.

There is a clear repetative workflow here and there in different environments. Having the whole ionosphere stack imported to Mintpy gives us more flexibility with the whole dataset but the file size of this ionStack.h5 is big like 2GB. Getting the inspection and inversion done in isce2 and just import the timeseries gives me 800MB (well, not a huge reduction because it depends on how dense the original ionosphere network was. So maybe not a big concern in the end).

There is also burstRamp phase estimate of the ionosphere as well. The current stackSentinel procedure is like the following:

iono_pair --> iono_azshift_pair ----inversion (& inspection)---> iono_azshift_timeseries --> burstRamp_timeseries

Thus the final product are burstRamp timeseries, not burstRamp interferograms. We cannot take them as a burstRamp ifgramStack input into Mintpy consistent as ionStack. So right now I do plotting inspection & manual removal & SBAS inversion in isce2, getting both ionosphere phase and brustRamp timeseries from there, then import these timeseries into MintPy. Anyway, enough logistics...

[yunjunz]

Thank you @yuankailiu for the testing and discussions. I agree with you. Here are the to-do list I could think of:

1. support low-resolution ionospheric delay products directly, by supporting two different resolution files in diff.py.
2. support using the ionospheric delay time-series and the burst ramp time-series from topsStack directly.
3. I was thinking about moving the iono (stack or time-series) loading process from load_data.py into iono_split_spectrum.py, but am not sure which one is simpler.