Sentinel-1 SBAS-InSAR Workflow using Mintpy

This blog is about full workflow of Sentinel-1 SBAS-InSAR using Mintpy.

1. Build the environment

Install the released version via conda or mamba
1
mamba install -c conda-forge mintpy
Install the development version

# use wget or curl to download in the command line or click from the web browser
# for macOS, use Miniforge3-MacOSX-x86_64.sh instead.
cd ~/tools
wget https://ghfast.top/https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p ~/tools/miniforge
~/tools/miniforge/bin/mamba init bash

# Add "isce2"               below to install extra dependencies if you use ISCE-2
# Add "gdal"                below to install extra dependencies if you use ARIA, FRInGE, HyP3
# Add "gdal libgdal-netcdf" below to install extra dependencies if you use GMTSAR

conda create -n mintpy python=3.11
conda activate mintpy
conda install mamba
mamba install --file ~/tools/MintPy/requirements.txt
python -m pip install -e ~/tools/MintPy
# ~/tools/miniforge/bin/mamba create --name mintpy --file ~/tools/MintPy/requirements.txt

2. Pre-processing

Before run full SBAS workflow in mintpy, prep_isce.py should be run to prepare metadata.

在获取了干涉序列后，最后一步就是用MintPy去获得最终的时序InSAR形变产品,首先在~/Documents/00_my_projects/01_isce_test/stacks中创建一个mintpy文件夹，作为MintPy的工作文件夹：
1
2
3
cd ~/Documents/00_my_projects/01_isce_test/stacks mkdir mintpy; cd mintpy smallbaselineApp.py -g
smallbaselineApp.py是mintpy的核心，这里我们创建了mintpy文件夹，并使用smallbaselineApp.py -g生成了一个cfg参数配置文件，名字叫smallbaselineApp.cfg。

该文件中对所有参数都给出了超级详细的解释，按照需求进行修改即可，其他参数可以默认auto，不写在cfg文件中也可以的。我的cfg文件见另一篇bolg:

# vim: set filetype=cfg:
##------------------------ smallbaselineApp.cfg ------------------------##
########## computing resource configuration
mintpy.compute.maxMemory = 24 #分配的运算内存[float > 0.0], auto for 4, max memory to allocate in GB
mintpy.load.processor       = isce  #指定你的干涉处理平台[isce, aria, hyp3, gmtsar, snap, gamma, roipac, nisar], auto for isce
mintpy.load.updateMode      = no  #是否每次都是忽略已有的结果，重新生成结果[yes / no], auto for yes, skip re-loading if HDF5 files are complete

##---------for ISCE only:
mintpy.load.metaFile        = ../reference/IW*.xml  #指定IW对应的xml数据[path of common metadata file for the stack], i.e.: ./reference/IW1.xml, ./referenceShelve/data.dat
mintpy.load.baselineDir     = ../baselines  #基线文件夹[path of the baseline dir], i.e.: ./baselines
##---------interferogram stack:
mintpy.load.unwFile         = ../merged/interferograms/*/filt_fine.unw  #所有的解缠后相位文件unw[path pattern of unwrapped interferogram files]
mintpy.load.corFile         = ../merged/interferograms/*/filt_fine.cor  #[path pattern of spatial coherence       files]
mintpy.load.connCompFile    = ../merged/interferograms/*/filt_fine.unw.conncomp  #[path pattern of connected components    files], optional but recommended
##---------geometry用于地理编码的文件:
mintpy.load.demFile         = ../merged/geom_reference/hgt.rdr  #[path of DEM file]
mintpy.load.lookupYFile     = ../merged/geom_reference/lat.rdr  #[path of latitude /row   /y coordinate file], not required for geocoded data
mintpy.load.lookupXFile     = ../merged/geom_reference/lon.rdr  #[path of longitude/column/x coordinate file], not required for geocoded data
mintpy.load.incAngleFile    = ../merged/geom_reference/los.rdr  #[path of incidence angle file], optional but recommended
mintpy.load.azAngleFile     = ../merged/geom_reference/los.rdr  #[path of azimuth   angle file], optional
mintpy.load.shadowMaskFile  = ../merged/geom_reference/shadowMask.rdr  #[path of shadow mask file], optional but recommended
########## 4. correct_unwrap_error (optional)
## supported methods:
## a. phase_closure          - suitable for highly redundant network
## b. bridging               - suitable for regions separated by narrow decorrelated features, e.g. rivers, narrow water bodies
## c. bridging+phase_closure - recommended when there is a small percentage of errors left after bridging
mintpy.unwrapError.method          = bridging  #选择解缠误差改正方法[bridging / phase_closure / bridging+phase_closure / no], auto for no
mintpy.troposphericDelay.method = auto  #是否进行大气校正，这里不进行大气校正[pyaps / height_correlation / gacos / no], auto for pyaps
########## 9.2 reference_date
## Reference all time-series to one date in time
## reference: Yunjun et al. (2019, section 4.9)
## no     - do not change the default reference date (1st date)
mintpy.reference.date = auto   #指定参考日期，no为第一个日期[reference_date.txt / 20090214 / no], auto for reference_date.txt
mintpy.plot.maxMemory = 32  #分配给绘图的内存[float], auto for 4, max memory used by one call of view.py for plotting.

设置好smallbaselineApp.cfg配置文件后，直接运行：
1
2
smallbaselineApp.py smallbaselineApp.cfg
mintpy 结果：

Result of Mintpy

mintpy获取的结果均用hdf5文件（.h5）进行储存，可以看到mintpy文件夹下生成了很多.h5文件，其中比较重要的列举一下：
timeseries.h5累计形变序列文件
timeseries_demErr.h5经过地形DEM误差改正后的累计形变序列文件
velocity.h5形变速率文件

\geo文件夹下储存了所有系统自动进行地理编码后的结果，如geo_timeseries_demErr.h5包含了各个日期对应的累积形变序列，储存于hdf5数据格式中

\pic文件夹包括了系统给出的一些可视化结果
值得一提的是，mintpy给出了大量的程序进行DIY操作，便于我们进行出图、提取数据等工作。

首先，对于所有mintpy生成的h5文件，均可使用info.py查看其文件结构以及相关信息。如info.py timeseries_demErr.h5，这一功能类似于GDAL中的gdalinfo。
所有的timeseries文件，均可以用tsview.py进行交互式可视化查看，如tsview.py timeseries_demErr.h5。
非时间序列的 .h5文件，可以用view.py进行查看，view.py velocity.h5
save_gdal.py从.h5文件中提取栅格文件转为ENVI或Tiff格式
geocode.py对文件进行地理编码处理，可以实现rdr雷达坐标系下的栅格文件和WGS84坐标栅格文件的互相转化
以上所有的代码均可通过-h命令查看对应的参数，从而进行更加精细的DIY操作。
- 配置PyAPS下载气象数据，进行大气校正：
ERA5 data set is redistributed over the Copernicus Climate Data Store (CDS). Registration is required for the data access and downloading.
- Create a new account on the CDS website if you don’t own a user account yet.
- CDS API setup: Create the local file $HOME/.cdsapirc (in your Unix/Linux environment) and add the following two lines, replacing your-personal-access-token with your own token:
1
2
url: https://cds.climate.copernicus.eu/api key: your-personal-access-token
- Make sure that you accept the data license in the Terms of use on ECMWF website: Login, under Datasets > ERA5 hourly data on pressure levels from 1940 to present > Download > Terms of use, click Accept to accespt the license to use Copernicus Products.
- Test the account setup by running:
1
2
git clone https://github.com/insarlab/PyAPS.git --depth 1 python PyAPS/tests/test_dload.py

3. Precessing

Edit .bashrc file, change the following lines:

1	`alias load_mintpy='conda activate mintpy; source ~/tools/conda-envs/insar/config.rc; export PATH=${PATH}:${ISCE_STACK}/topsStack; echo "load ISCE-2 topsStack from "${ISCE_STACK}/topsStack'`

Load the environment

load_insar
cd $INSAR_WORK_DIR/mintpy

# Run SBAS in Mintpy
smallbaselineApp.py -g
smallbaselineApp.py smallbaselineApp.cfg

4. Error Fix

4.1 `KeyError: 'STARTING_RANGE'`

The error message KeyError: 'STARTING_RANGE' in MintPy’s load_data step, specifically originating from mintpy/utils/utils0.py, indicates that MintPy is trying to read a metadata attribute named ‘STARTING_RANGE’ from one of your ISCE2 output files (likely related to geometry or a .rsc file), but this attribute is missing.

This is a common issue when linking ISCE2 outputs to MintPy, and it often points to a problem with how the metadata is prepared for MintPy, or a discrepancy in the ISCE2 output structure that MintPy expects.

Here’s a breakdown of the problem and how to solve it:

The Problem: Missing ‘STARTING_RANGE’ Metadata

MintPy relies on a consistent set of metadata attributes (often in the ROI_PAC .rsc format internally) to understand the geometry and properties of your interferograms. ‘STARTING_RANGE’ (and ‘RANGE_PIXEL_SIZE’ as seen in the traceback line range_n, dR = float(atr['STARTING_RANGE']), float(atr['RANGE_PIXEL_SIZE'])) are crucial for converting radar coordinates to slant range distances.

This attribute is typically present in the metadata files generated by ISCE2 (like the .xml files for TOPS data or other metadata files for stripmap). MintPy uses its prep_isce.py script to extract and standardize this information into .rsc files that it can easily read.

The KeyError means that prep_isce.py either didn’t run correctly, couldn’t find the necessary information in your ISCE2 outputs, or the .rsc file it tried to create is missing this specific key.

Solutions:

The most common solutions revolve around ensuring prep_isce.py is correctly executed and that your ISCE2 environment and data structure are as MintPy expects.

Scenario 1: prep_isce.py wasn’t run, or wasn’t run correctly.

MintPy provides helper scripts to convert various InSAR processor outputs into a format it can consume. For ISCE2, this is prep_isce.py. You need to run this after your ISCE2 processing is complete and before you run smallbaselineApp.py.

Activate your ISCE2 environment: It’s highly recommended to have ISCE2 and MintPy installed in the SAME conda environment. This avoids potential version conflicts and ensures prep_isce.py can correctly interact with ISCE2’s libraries. If they are in separate environments, try to merge them or create a new one with both.
1
conda activate your_isce2_mintpy_env # Or whatever your combined environment is called
Navigate to your ISCE2 merged/ (or interferograms/) directory: This is where your .unw (unwrapped phase) and .cor (coherence) files usually reside.

Run prep_isce.py: The command will vary slightly depending on your ISCE2 workflow (TOPS or Stripmap).

For ISCE2 topsStack.py workflow (Sentinel-1):
This is the most common case for Sentinel-1. You’ll typically have an IW?.xml file (e.g., IW1.xml) in your reference/ directory within your ISCE2 stack, a baselines/ directory, and your merged/ directory with filt_*.unw files.

# From your ISCE2 main processing directory, where 'merged' and 'baselines' are subdirectories
# Example for ISCE2 topsStack:
prep_isce.py -f merged/*/filt_*.unw -m reference/IW*.xml -b baselines/ -g merged/geom_reference/

# If you only have a single IW burst, specify it directly:
# prep_isce.py -f merged/IW*/filt_*.unw -m reference/IW1.xml -b baselines/ -g merged/geom_reference/

-f: Pattern for unwrapped interferogram files.
-m: Path to a reference metadata XML file (e.g., IW1.xml or data.dat for older ISCE2 versions). This is where STARTING_RANGE and other geometric info resides.
-b: Path to your baselines directory (containing baseline.png, etc.).
-g: Path to your geometry directory (containing hgt.rdr, incLocal.rdr, lat.rdr, lon.rdr, los.rdr).

For ISCE2 stripmapApp.py workflow:
The input metaFile (-m) might be different. You might need to point it to referenceShelve/data.dat or a similar file that contains the global metadata for the stack. Check the ISCE2 documentation or examples you followed.

What prep_isce.py does: It reads the metadata from ISCE2’s native files (like .xml or data.dat) and creates corresponding .rsc files for each .unw, .cor, and geometry file. These .rsc files contain the standardized metadata in a format MintPy expects, including ‘STARTING_RANGE’.

Check for newly created .rsc files: After running prep_isce.py, you should see .rsc files alongside your .unw, .cor, hgt.rdr, incLocal.rdr, etc., files. For example, filt_000000_000001.unw.rsc.

Scenario 2: The ISCE2 output structure or content is not what MintPy’s prep_isce.py expects.

Incomplete ISCE2 processing: Ensure your ISCE2 topsStack.py or stripmapApp.py workflow completed without errors and generated all expected output files (unwrapped phase, coherence, and geometry files like hgt.rdr, incLocal.rdr, lat.rdr, lon.rdr). If a specific interferogram failed to generate all its expected files, it could cause issues.
ISCE2 version differences: If you processed your data with a very old or very new version of ISCE2 that differs significantly from what MintPy’s prep_isce.py is designed for, it might struggle to extract the correct metadata. As mentioned, having both in the same (and up-to-date) conda environment helps.
Corrupted XML/metadata files: Very rarely, the original ISCE2 XML files might be corrupted or malformed, preventing prep_isce.py from reading them.

Your `smallbaselineApp.cfg` file

Please share your smallbaselineApp.cfg file. The problem is in the load_data step, and the configuration for that step is crucial. Pay close attention to:

mintpy.load.processor = isce (This should be set correctly)
mintpy.load.metaFile = ... (This should point to the correct ISCE2 reference XML, like reference/IW1.xml or referenceShelve/data.dat or similar, depending on your ISCE2 setup).
mintpy.load.baselineDir = ... (Path to your ISCE2 baselines directory)
mintpy.load.unwFile = ... (Pattern for your unwrapped interferograms)
mintpy.load.corFile = ... (Pattern for your coherence files)

Example smallbaselineApp.cfg snippet for ISCE2 TOPS:

################################################################################
## 1. load_data: read input data and stack into HDF5 format
################################################################################
mintpy.load.processor = isce #[isce, aria, hyp3, gmtsar, snap, gamma, roipac, nisar], auto for isce

##---------for ISCE only:
mintpy.load.metaFile = /path/to/your/ISCE_Stack/reference/IW*.xml  # Adjust this path
mintpy.load.baselineDir = /path/to/your/ISCE_Stack/baselines      # Adjust this path

##---------interferogram datasets:
mintpy.load.unwFile = /path/to/your/ISCE_Stack/merged/*/filt_*.unw  # Adjust this path
mintpy.load.corFile = /path/to/your/ISCE_Stack/merged/*/filt_*.cor  # Adjust this path

##---------geometry datasets:
mintpy.load.demFile = /path/to/your/ISCE_Stack/merged/geom_reference/hgt.rdr # Adjust this path
mintpy.load.incFile = /path/to/your/ISCE_Stack/merged/geom_reference/incLocal.rdr # Adjust this path
mintpy.load.azFile = /path/to/your/ISCE_Stack/merged/geom_reference/los.rdr # Azimuth angle, often LOS angle is used
# Or if you have actual azimuth angle file from ISCE2:
# mintpy.load.azFile = /path/to/your/ISCE_Stack/merged/geom_reference/azimuth.rdr

# OPTIONAL: to subset the data
# mintpy.load.subset.lalo = 33.1:33.4, -118.0:-117.5
# mintpy.load.subset.yx = 100:500, 200:800

Action Plan:

Verify your conda environment: Ensure MintPy and ISCE2 are compatible and preferably in the same environment.
Manually run prep_isce.py in your ISCE2 stack directory, double-checking all paths in the command. Pay close attention to the output messages of prep_isce.py for any errors or warnings.
Inspect the .rsc files: After prep_isce.py runs, open one of the generated .unw.rsc or geometry .rsc files (e.g., hgt.rdr.rsc) with a text editor and look for STARTING_RANGE. If it’s still missing, the problem lies deeper in the prep_isce.py's ability to extract it from your ISCE2 outputs.
Provide your smallbaselineApp.cfg: This will help confirm if your configuration correctly points to the expected files.

Once prep_isce.py successfully generates .rsc files containing STARTING_RANGE, your smallbaselineApp.py command should proceed past the load_data step.