CLM5-PDAF Starting instructions

1. Install components

The following instructions assume you are working on the JSC machine JURECA. If you are working on another JSC machine switch the name appropriately and the instructions should still work. Running CLM5-PDAF on non JSC machines is not covered in these instructions but should be similar if TSMP-PDAF, PDAF, and CLM5 can be installed on the machine.

1.1 TSMP-PDAF

Clone the TSMP-PDAF repository to your work folder and switch to the CLM5-PDAF branch like this:

 git clone https://github.com/HPSCTerrSys/TSMP.git tsmp

 cd tsmp

1.2 PDAF

Get PDAF from https://github.com/PDAF/PDAF and clone it in the tsmp directory as pdaf.

 git clone https://github.com/PDAF/PDAF.git pdaf

1.3 CLM5

Clone CLM5 and its dependencies from the official repo to the tsmp folder like this:

git clone -b release-clm5.0 https://github.com/ESCOMP/CTSM.git clm5_0

cd clm5_0

source ../bldsva/machines/JURECA/loadenvs.Intel

./manage_externals/checkout_externals

  • source ...: Loads the software environment on JURECA that is later used for build with Intel

  • ./manage_externals/checkout_externals: Loads dependencies of CLM5, mostly a collection of git clone commands

1.4 Compile the models together

The TSMP framework works with its own build system. With the models in the folders as described before the compilation of both CLM5 and PDAF can be done with a single build command after the following preparations.

If not already configured for CLM5 the following environmental variables are needed to compile CLM5 and can be for example be set like this:

User defined data paths

Path to the root directory for all CESM / CLM input files

export CESMDATAROOT=$SCRATCH/cesm

This is the default value set by the build process. However, you can export your own CESMDATAROOT.

Path to the CSM specific input files (usually subfolder of CESMDATAROOT)

export CSMDATA=$CESMDATAROOT/inputdata

In the tsmp/bldsva directory:

./build_tsmp.ksh -c clm5-pdaf -m JURECA -O Intel

Potential problems that can happen during this step:

  • If you worked with CLM5 standalone and have created a .cime folder in your home directory it can cause a conflict during compilation. The error file will contain a line that says ERROR: No machine jureca found for example. The easiest solution to this is to move the ~/.cime folder to ~/.cime_deactivated while working with CLM5-PDAF.

  • Using the newest software stages from JSC there is an import conflict with bigint for PERL. See https://icg4geo.icg.kfa-juelich.de/ExternalRepos/tsmp-pdaf/tsmp/-/issues/67 for details. The error message will contain a line like this: err=Can't locate bigint.pm in @INC. The solution for now is to comment all lines with bigint and bignum in the script tsmp/clm5_0/bld/config_files/clm_phys_vers.pm since this is only needed to check some version numbers the PERL modules is not strictly needed.

Once successfully compiled the executable can be found in path tsmp/bin/JURECA_clm/tsmp-pdaf.

2. File preparation

For example files see Example Case.

2.1 Ensemble generation

For data assimilation with the ensemble Kalman filter we usually create an ensemble by perturbing the atmospheric forcing files as well as the sand and clay fractions in the surface file.

For CLM5 how these files are named and numbered can be customized, as shown in the next section. But a recommended approach is to have a suffix with zero filled numbering like surfdata_DE-Wue_hist_78pfts_CMIP6_simyr2000_c191001.nc_00001.nc.

The exact details of perturbation can vary from study to study, but a usable script to perturb the forcings and surface files can be found at /p/largedata/jicg41/strebel2/CLM5PDAF_Example/perturb_forcings_and_soil_properties_in_range.py.

In this script the following variables should be modified or at least considered for modification before running it for your case.

* years (range of years for atm. forcing perturb)
* num_ensembles (how many ensemble members should be created)
* sname (path and naming for the output surface files)
* sorig (path for the original surface file)
* the attribute "perturbed by" (to indicate for future use who created the files)
* the values of the random.uniform function (range of perturbation for soil characteristics)
* sd, mean, and correl (the statistical characteristics for the atm. forcing perturbations)
* fname (path and naming for the original atm. files)
* outname (path and naming for the output atm. files)

2.2 Case generation

These instructions assume you already have a running CLM5 standalone case before moving to CLM5PDAF. If not instructions of how to generate the domain file, surface file, and download the default CLM5 inputfile can be found on the GitLab.

We setup a new case in a new directory. In this directory we will need a few subfolders:

 mkdir logs

 mkdir timing/checkpoints -p

CLM5-PDAF does not use the same case setup as CLM5 standalone, instead we skip to the final namelist files that are needed to run CLM5. The minimal set of namelists to run CLM5 is:

* atm_modelio.nml
* datm_in
* drv_flds_in
* drv_in
* esp_modelio.nml
* glc_modelio.nml
* ice_modelio.nml
* lnd_in
* lnd_modelio.nml
* mosart_in
* ocn_modelio.nml
* rof_modelio.nml
* wav_modelio.nml

Since these files are needed for each ensemble member we create them with a script. An example of such a script can be found at /p/largedata/jicg41/strebel2/CLM5PDAF_Example/create_ensemble_namelists.py

In this script the following variables should be modified or at least considered for modification before running it for your case.

*  domain file (appears in write_datm_in(), as fatmlndfrc in write_lnd_in(), in write_stream_files() with path and name separated)
*  streams (in write_datm_in() modify to name of your case)
*  finidat (in write_lnd_in() your initial condition file from spinup)
*  fsurdat (in write_lnd_in() your surface files including ensemble numbering)
*  hist_mfilt, hist_nhtfrq (customize your history file settings)
*  filePath, fileNames (in write_stream_files customize the name and list of years for forcings)
*  name of the case (at the end of write_stream_files() needs to be consistent with the ones given in write_datm_in())
*  year_start, year_end (in main() give start and end year)
*  prefix (in main() for naming of output files)
*  num_ensemble (in main() number of ensemble members)
*  b_dir (in main() path to build directory of clm within tsmp)
*  r_dir (in main() path to the directory where the case will be run from i.e. the case directory)
*  Every other variable uses the default from a standalone CLM5 case, but may vary in your case, check the namelists in the run directory of your case against the variables listed in this script if you have customized the CLM5 case.

The script at the moment can either be called without arguments if all variables are modified within the script or with year_start and prefix as arguments.

To run PDAF an additional namelist is needed, enkfpf.par for details on the contents of this namelist refer to the TSMP-PDAF manual.

Lastly, we create a symbolic link to the executable in the case directory. This way we do not create unnecessary copies of the executable for each case, we make sure that all cases are run with the same executable, and modification and re-compilations of the executable will not need any changes in the case directories. The symbolic link can be created like this:

 ln -s PATH_TO_THE_EXECUTABLE/tsmp-pdaf .

Similarly, we can use a symbolic link to the loadenvs file from TSMP to make sure we use the same module versions during runtime as during compilation.

 ln -s PATH_TO_THE_TSMP_FOLDER/bldsva/machines/JURECA/loadenvs.Intel loadenvs

3. Running the simulation

To run the simulation we use can use a simple jobscript, an example can be found at /p/largedata/jicg41/strebel2/CLM5PDAF_Example/jobscript_da.slurm

The arguments for the tsmp-pdaf executable are explained in the TSMP-PDAF manual.

Important parameters are -n_modeltasks is the number of ensemble members and nodes times ntasks-per-node should be at least the same or a multiple of n_modeltasks.

The argument -delt_obs can be used to switch between open loop and DA simulation by increasing the value to be larger than the number of simulation steps set in enkfpar.par no assimilation will take place and the simulation will be an open loop ensemble simulation.

Additionally, -obs_filename has the path to the observations and the prefix of the observation files without the numbering.

4. Post-processing the output

CLM5-PDAF will create history files for each ensemble members according to the settings defined in the script create_ensemble_namelists.py with the hist variables. For large ensembles this will create a lot of files and should be processed and moved / archived away from $SCRATCH to avoid filling the number of files limit. One way to process the output is to collect the ensemble statistics of the variables in one file. An example script to perform this post-processing can be found at /p/largedata/jicg41/strebel2/CLM5PDAF_Example/collect_some_ensemble_stats.py

The script should be modified before use, by changing the collect_vars variable to a list with the CLM5 variable names that should be collected. The output netcdf file will be called Ensemble_Collection_ and contain the listed variables with their ensemble minimum, maximum, mean, and standard deviation.

At the moment the script assumes that each year is in a separate history file, but could be customized to work on multi-year or less than a year files. Currently, the script takes the arguments: year, OL or DA, num_ensemble, case name. Where case name refers to the previously mentioned prefix that can be used to differentiate between different simulations within the same case.

5. Example case:

All the input files necessary to create a 10 ensemble case for the year 2009 are located at /p/largedata/jicg41/strebel2/CLM5PDAF_Example/Example_Case and can be used to test the setup before working on your own case.