integrate advanced time func with UQ into the routine workflow (insar…

…lab#795)

* ts2vel: add uncertaintyQuantification & timeSeriesCovFile option

+ add the following two new template options into smallbaselineApp.cfg
   - mintpy.velocity.uncertaintyQuantification = residue / covariance / bootstrap, to replace the previous mintpy.velocity.bootstrap option
   - mintpy.velocity.timeSeriesCovFile with the default value of no

+ timeseries2velocity: re-use arg_group.add_reference_argument(plot=False) in timeseries2velocity.py

+ load_data: write S1A/B_date.txt file from SAFE_files.txt file if the latter exists in its default location (same level as reference folder)

* ts2vel: add timeFunc.* options to smallbaselineApp.cfg

+ integrate the advanced time func fitting into smallbaselineApp.py, via the following template options:
   - mintpy.timeFunc.polynomial
   - mintpy.timeFunc.periodic
   - mintpy.timeFunc.step
   - mintpy.timeFunc.exp
   - mintpy.timeFunc.log

+ rename the following options for consistency **[BREAKING CHANGES]**:
   - mintpy.velocity.excludeDate --> mintpy.timeFunc.excludeDate
   - mintpy.velocity.startDate --> mintpy.timeFunc.startDate
   - mintpy.velocity.endDate --> mintpy.timeFunc.endDate
   - mintpy.velocity.uncertaintyQuantifiction --> mintpy.timeFunc.uncertaintyQuantification
   - mintpy.velocity.timeSeriesCovFile --> mintpy.timeFunc.timeSeriesCovFile
   - mintpy.velocity.bootstrapCount --> mintpy.timeFunc.bootstrapCount

+ utils.time_func.py:
   - add MODEL_EXAMPLE as the single-sourced example for the time function model/configuration setup
   - add inps2model() to parse/convert the namespace "inps" object into time function dict "model" object
   - add get_num_param() to estimate the num of unknown parameters from a given model dict object

+ utils.readfile.read_template(): add "skip_chars" argument to skip certain characters in the template values, e.g. the commonly template typo from users "[" and "]". This is to simplify the read_template2inps() functions.
   - leverage the new read_template() in following scripts:
      - dem_error.py
      - geocode.py
      - modify_network.py
      - reference_point.py
      - timeseries2velocity.py
   - the insarmaps-related code is removed, and they are messy and they can and should be implemented separately in insarmaps repo if needed.

+ utils.ptime.py: add get_exclude_date_list() from previous ts2vel.py

+ utils.arg_group.py:
   - add_time_func_argument(): fix typo and better alignment for comments to improve the readability
   - adjust indentation throughout the script

+ timeseries2velocity: support time function inputs from template files
   - add time function inputs into the config keys to be saved in the h5 file
   - remove the comments on contributing authors, as they are visible in the git commit history
   - merge init_exp_log_dicts() and read_inps2model() into utils.time_func.inps2model()
   - refactor and move read_exclude_date() into utils.ptime.get_exclude_date_list()
   - read_template2inps(): support the new time function template options
   - model2hdf5_dataset(): modify the periodic/exp/log dataset names, to be more clear on the time unit

+ move ifgram_reconstrunction.py to /legacy since it's not frequently used at all

+ move tropo_pyaps.py to /legacy since it's unavailable anymore (due to the takedown of EarthDef and Python2) and not actively used.

mintpy/defaults/smallbaselineApp.cfg

@@ -276,17 +276,34 @@ mintpy.reference.date = auto   #[reference_date.txt / 20090214 / no], auto for r
 
 
 ########## 10. velocity
-## Estimate linear velocity and its standard deviation from time-series
-## and from tropospheric delay file if exists.
-## reference: Fattahi and Amelung (2015, JGR)
-mintpy.velocity.excludeDate    = auto   #[exclude_date.txt / 20080520,20090817 / no], auto for exclude_date.txt
-mintpy.velocity.startDate      = auto   #[20070101 / no], auto for no
-mintpy.velocity.endDate        = auto   #[20101230 / no], auto for no
-
-## Bootstrapping
-## reference: Efron and Tibshirani (1986, Stat. Sci.)
-mintpy.velocity.bootstrap      = auto   #[yes / no], auto for no, use bootstrap
-mintpy.velocity.bootstrapCount = auto   #[int>1], auto for 400, number of iterations for bootstrapping
+## Estimate a suite of time functions [linear velocity by default]
+## from final displacement file (and from tropospheric delay file if exists)
+mintpy.timeFunc.startDate   = auto   #[20070101 / no], auto for no
+mintpy.timeFunc.endDate     = auto   #[20101230 / no], auto for no
+mintpy.timeFunc.excludeDate = auto   #[exclude_date.txt / 20080520,20090817 / no], auto for exclude_date.txt
+
+## Fit a suite of time functions
+## reference: Hetland et al. (2012, JGR) equation (2-9)
+## polynomial function    is  defined by its degree in integer. 1 for linear, 2 for quadratic, etc.
+## periodic   function(s) are defined by a list of periods in decimal years. 1 for annual, 0.5 for semi-annual, etc.
+## step       function(s) are defined by a list of onset times in str in YYYYMMDD(THHMM) format
+## exp & log  function(s) are defined by an onset time followed by an charateristic time in integer days.
+##   Multiple exp and log functions can be overlaied on top of each other, achieved via e.g.:
+##   20110311,60,120          - two functions sharing the same onset time OR
+##   20110311,60;20170908,120 - separated by ";"
+mintpy.timeFunc.polynomial = auto   #[int >= 0], auto for 1, degree of the polynomial function
+mintpy.timeFunc.periodic   = auto   #[1,0.5 / list_of_float / no], auto for no, periods in decimal years
+mintpy.timeFunc.step       = auto   #[20110311,20170908 / 20120928T1733 / no], auto for no, step function(s)
+mintpy.timeFunc.exp        = auto   #[20110311,60 / 20110311,60,120 / 20110311,60;20170908,120 / no], auto for no
+mintpy.timeFunc.log        = auto   #[20110311,60 / 20110311,60,120 / 20110311,60;20170908,120 / no], auto for no
+
+## Uncertainty quantification methods:
+## a. residue    - propagate from fitting residue assuming normal dist. in time (Fattahi & Amelung, 2015, JGR)
+## b. covariance - propagate from time series (co)variance matrix
+## c. bootstrap  - bootstrapping (independently resampling with replacement; Efron & Tibshirani, 1986, Stat. Sci.)
+mintpy.timeFunc.uncertaintyQuantification = auto   #[residue, covariance, bootstrap], auto for residue
+mintpy.timeFunc.timeSeriesCovFile         = auto   #[filename / no], auto for no, time series covariance file
+mintpy.timeFunc.bootstrapCount            = auto   #[int>1], auto for 400, number of iterations for bootstrapping
 
 
 ########## 11.1 geocode (post-processing)

mintpy/defaults/smallbaselineApp_auto.cfg

@@ -125,13 +125,21 @@ mintpy.reference.date    = reference_date.txt
 
 
 ########## velocity
-mintpy.velocity.excludeDate    = exclude_date.txt
-mintpy.velocity.startDate      = no
-mintpy.velocity.endDate        = no
-
-## bootstrap
-mintpy.velocity.bootstrap      = no
-mintpy.velocity.bootstrapCount = 400
+mintpy.timeFunc.startDate    = no
+mintpy.timeFunc.endDate      = no
+mintpy.timeFunc.excludeDate  = exclude_date.txt
+
+## time functions
+mintpy.timeFunc.polynomial   = 1
+mintpy.timeFunc.periodic     = no
+mintpy.timeFunc.step         = no
+mintpy.timeFunc.exp          = no
+mintpy.timeFunc.log          = no
+
+## uncertainty quantification
+mintpy.timeFunc.uncertaintyQuantification  = residue
+mintpy.timeFunc.timeSeriesCovFile          = no
+mintpy.timeFunc.bootstrapCount             = 400
 
 
 ########## geocode

mintpy/dem_error.py

@@ -176,7 +176,7 @@ def read_template2inps(template_file, inps=None):
         inps = cmd_line_parse()
     iDict = vars(inps)
     print('read options from template file: '+os.path.basename(template_file))
-    template = readfile.read_template(template_file)
+    template = readfile.read_template(template_file, skip_chars=['[', ']'])
     template = ut.check_template_auto_value(template)
 
     # Read template option
@@ -189,8 +189,7 @@ def read_template2inps(template_file, inps=None):
             if key in ['polyOrder']:
                 iDict[key] = int(value)
             elif key in ['excludeDate','stepFuncDate']:
-                value = value.replace('[','').replace(']','').replace(',', ' ')
-                iDict[key] = ptime.yyyymmdd(value.split())
+                iDict[key] = ptime.yyyymmdd(value.split(','))
 
     # computing configurations
     dask_key_prefix = 'mintpy.compute.'

mintpy/geocode.py

@@ -177,7 +177,7 @@ def read_template2inps(template_file, inps):
     if not inps:
         inps = cmd_line_parse()
     inps_dict = vars(inps)
-    template = readfile.read_template(template_file)
+    template = readfile.read_template(template_file, skip_chars=['[', ']'])
     template = ut.check_template_auto_value(template)
 
     prefix = 'mintpy.geocode.'
@@ -186,8 +186,7 @@ def read_template2inps(template_file, inps):
         value = template[prefix + key]
         if value:
             if key in ['SNWE', 'laloStep']:
-                value = value.replace('[','').replace(']','').replace(',',' ')
-                inps_dict[key] = [float(i) for i in value.split()]
+                inps_dict[key] = [float(i) for i in value.split(',')]
             elif key in ['interpMethod']:
                 inps_dict[key] = value
             elif key == 'fillValue':

mintpy/load_data.py

@@ -737,7 +737,7 @@ def prepare_metadata(iDict):
 
     elif processor == 'isce':
         from mintpy import prep_isce
-        from mintpy.utils.isce_utils import get_processor
+        from mintpy.utils import s1_utils, isce_utils
 
         # --meta-file
         meta_files = sorted(glob.glob(iDict['mintpy.load.metaFile']))
@@ -756,11 +756,9 @@ def prepare_metadata(iDict):
                     'mintpy.load.rgOffFile', 'mintpy.load.azOffFile']
         obs_keys = [i for i in obs_keys if i in iDict['dset_name2template_key'].values()]
         obs_paths = [iDict[key] for key in obs_keys if iDict[key].lower() != 'auto']
+        stack_processor = isce_utils.get_processor(meta_file) if os.path.isfile(meta_file) else 'topsStack'
         if len(obs_paths) > 0:
-
-            # ifgramStack
-            processor = get_processor(meta_file) if os.path.isfile(meta_file) else 'topsStack'
-            if processor == 'alosStack':
+            if stack_processor == 'alosStack':
                 obs_dir = os.path.dirname(obs_paths[0])
             else:
                 obs_dir = os.path.dirname(os.path.dirname(obs_paths[0]))
@@ -798,6 +796,15 @@ def prepare_metadata(iDict):
         except:
             warnings.warn('prep_isce.py failed. Assuming its result exists and continue...')
 
+        # [optional] for topsStack: SAFE_files.txt --> S1A/B_date.txt
+        if stack_processor == 'topsStack':
+            safe_list_file = os.path.join(os.path.dirname(os.path.dirname(meta_file)), 'SAFE_files.txt')
+            if os.path.isfile(safe_list_file):
+                mintpy_dir = os.path.dirname(os.path.dirname(iDict['outfile'][0]))
+                s1_utils.get_s1ab_date_list_file(mintpy_dir=mintpy_dir,
+                                                 safe_list_file=safe_list_file,
+                                                 print_msg=True)
+
     elif processor == 'aria':
         from mintpy import prep_aria
 

mintpy/modify_network.py

@@ -169,7 +169,7 @@ def read_template2inps(template_file, inps=None):
         inps = cmd_line_parse()
     inpsDict = vars(inps)
     print('read options from template file: '+os.path.basename(template_file))
-    template = readfile.read_template(inps.template_file)
+    template = readfile.read_template(inps.template_file, skip_chars=['[', ']'])
     template = ut.check_template_auto_value(template)
 
     # Update inps if key existed in template file
@@ -187,12 +187,12 @@ def read_template2inps(template_file, inps=None):
             elif key in ['maskFile', 'referenceFile']:
                 inpsDict[key] = value
             elif key == 'aoiYX':
-                tmp = [i.replace('[','').replace(']','').strip() for i in value.split(',')]
+                tmp = [i.strip() for i in value.split(',')]
                 sub_y = sorted([int(i.strip()) for i in tmp[0].split(':')])
                 sub_x = sorted([int(i.strip()) for i in tmp[1].split(':')])
                 inps.aoi_pix_box = (sub_x[0], sub_y[0], sub_x[1], sub_y[1])
             elif key == 'aoiLALO':
-                tmp = [i.replace('[','').replace(']','').strip() for i in value.split(',')]
+                tmp = [i.strip() for i in value.split(',')]
                 sub_lat = sorted([float(i.strip()) for i in tmp[0].split(':')])
                 sub_lon = sorted([float(i.strip()) for i in tmp[1].split(':')])
                 inps.aoi_geo_box = (sub_lon[0], sub_lat[1], sub_lon[1], sub_lat[0])
@@ -204,11 +204,9 @@ def read_template2inps(template_file, inps=None):
             elif key in ['startDate', 'endDate']:
                 inpsDict[key] = ptime.yyyymmdd(value)
             elif key == 'excludeDate':
-                value = value.replace('[','').replace(']','').replace(',', ' ')
-                inpsDict[key] = ptime.yyyymmdd(value.split())
+                inpsDict[key] = ptime.yyyymmdd(value.split(','))
             elif key == 'excludeIfgIndex':
-                value = value.replace('[','').replace(']','').replace(',', ' ')
-                inpsDict[key] += value.split()
+                inpsDict[key] += value.split(',')
                 inpsDict[key] = read_input_index_list(inpsDict[key], stackFile=inps.file)
 
     # Turn reset on if 1) no input options found to drop ifgram AND 2) there is template input

mintpy/reference_point.py

@@ -129,7 +129,7 @@ def read_template_file2inps(template_file, inps=None):
     if not inps:
         inps = cmd_line_parse([''])
     inps_dict = vars(inps)
-    template = readfile.read_template(template_file)
+    template = readfile.read_template(template_file, skip_chars=['[', ']'])
     template = ut.check_template_auto_value(template)
 
     prefix = 'mintpy.reference.'
@@ -147,14 +147,12 @@ def read_template_file2inps(template_file, inps=None):
     if key in template.keys():
         value = template[key]
         if value:
-            value = value.replace('[','').replace(']','')
             inps.ref_y, inps.ref_x = [int(i) for i in value.split(',')]
 
     key = prefix+'lalo'
     if key in template.keys():
         value = template[key]
         if value:
-            value = value.replace('[','').replace(']','')
             inps.ref_lat, inps.ref_lon = [float(i) for i in value.split(',')]
 
     return inps

mintpy/solid_earth_tides.py

@@ -54,7 +54,7 @@
 """
 
 def create_parser():
-    parser = argparse.ArgumentParser(description='Solid Earth tides (SET) correction',
+    parser = argparse.ArgumentParser(description='Solid Earth tides (SET) correction via PySolid',
                                      formatter_class=argparse.RawTextHelpFormatter,
                                      epilog='{}\n{}\n{}'.format(REFERENCE, TEMPLATE, EXAMPLE))