RegCM

The Regional Climate Model system RegCM , originally developed at the National Center for Atmospheric Research (NCAR), is maintained in the Earth System Physics (ESP) section of the ICTP. The first version of the model, RegCM1, was developed in 1989 and since then it has undergone major updates in 1993 (RegCM2), 1999 (RegCM2.5), 2006 (RegCM3) and most recently 2010 (RegCM4). The latest version of the model, RegCM4, is now fully supported by the ESP, while previous versions are no longer available. This version includes major upgrades in the structure of the code and its pre- and post- processors, along with the inclusion of some new physics parameterizations. The model is flexible, portable and easy to use. It can be applied to any region of the World, with grid spacing of up to about 10 km (hydrostatic limit), and for a wide range of studies, from process studies to paleoclimate and future climate simulation (source). The RegCM source code can be accessed here. It is also important that you familiarize with the RegCM users guide found here.

RegCM software installed in the HPC

As of this writing, there are a few versions of RegCM modules installed. All are compiled with CLM45 option enabled.

module whatis regcm
...
regcm/4.4.5.7-intel-openmpi: RegCM
regcm/4.5.0-intel-openmpi: RegCM
regcm/4.6.0-intel-openmpi: RegCM
...

Binaries associated with RegCM are:

$> ls
average         clm45_1dto2dCLM45  GrADSNcPlotCLM45     icbcCLM45      interp_emissions  mksurfdataCLM45  regrid        sigma2zCLM45  terrainCLM45
chem_icbcCLM45  emcre_gridCLM45    GrADSNcPrepareCLM45  interp_bionox  interp_pollen     regcmMPICLM45    sigma2pCLM45  sstCLM45
$> 

Other useful tools/scripts can be obtained from the Tools directory of the RegCM source code. It is recommended that you download the source code as some of the tools will come in handy.

Global datasets

The first step to run a RegCM simulation is to obtain static data to localize model DOMAIN and Atmosphere and Ocean global model dataset to build initial and boundary conditions ICBC to run a local area simulation. These are

  • SURFACE dataset -- topography, land type classification, lake depth, soil texture classification
  • CLM dataset -- global land surface characteristics dataset required if CLM option is enabled
  • CLM 4.5 -- global land surface characteristics dataset required if CLM45 option is enabled
  • SST -- global sea-surface temperature to feed the model with ocean temperature

These datasets can be obtained from http://clima-dods.ictp.it/regcm4/. We recommend that you download all the contents under SURFACE, CLM, CLM45 and SST directories. Store the datasets to a convenient location on your scratch directory by making this (sample) structure:

$> REGCM_GLOBEDAT=~/scratch2/global
$> mdkir -p ${REGCM_GLOBEDAT}/{SURFACE,CLM,CLM45,SST}

Obtaining and Preprocessing of Initial and Boundary Conditions Datasets

The initial and boundary conditions (ICBC) are used by the model to drive its equations. There are a number of freely available ICBC in http://clima-dods.ictp.it/regcm4/. Examples are CFSv2, EIN15, EIN75, ECHAM5, HadGEM2, etc. Others are enumerated in the RegCM users manual. In this guide, we shall use the EIN15 dataset. Preprocessing of ICBC is an integral part of the model simulation as it packages the data into something the model understands or requires. It is also important to emphasize that preprocessing of an ICBC dataset may be different from other ICBCs. For instance, no preproccessing is needed for EIN15; while a tool cfs_to_netcdf_regcm.sh should be run on CFSv2 datasets. In this guide, we will use EIN15 for simplicity. Download EIN15 data using the following as instructed in the user manual:

$> cd $REGCM_GLOBEDAT
$> mkdir -p EIN15/1990
$> cd EIN15/1990
> for type in "air hgt rhum uwnd vwnd" 
> do
>   for hh in "00 06 12 18" 
>   do
>     curl -o ${type}.1990.${hh}.nc \
>        http://clima-dods.ictp.it/regcm4/EIN15/1990/${type}.1990.${hh}.nc
>   done
> done

Running a test simulation

In this section, a rundown on how to run a RegCM simulation in the HPC is discussed while following the user's manual. Before continuing, follow the instructions in the user's manual in setting up the environment (Chapter 5.1 in RegCM User's Guide Version 4.5).

$> cd $REGCM_RUN
$> mkdir input output
$> cp $REGCM_ROOT/Testing/test_001.in .
$> cd $REGCM_RUN

The ln -sf $REGCM_ROOT/bin . step was intentionally left out because the binaries are already available in the HPC.

Create an Interactive RegCM Session

To create an interactive session in the HPC, we will use the salloc command and request for HPC resources (10 tasks) in the debug partition.

$> salloc --ntasks=10 -p debug
salloc: Granted job allocation 187738
$> squeue | grep 187738
            187738     debug     bash noah-wis  R       0:09      1 tux-01
$> ssh tux-01
Last login: Thu Mar 30 17:29:45 2017 from 192.168.224.11
$> module load regcm/4.6.0-intel-openmpi
>> netcdf-4.4.1.1-intel has been loaded.
>> regcm-4.6.0-intel-openmpi has been loaded.

Create the DOMAIN file using terrainCLM45

$> cd $REGCM_RUN
$> srun terrainCLM45 test_001.in
 ...
 ...
 ...
 Successfully completed terrain fields generation
$> ls input
test_001_DOMAIN000.nc test_001_LANDUSE

Create the SST using the sstCLM45 program

$> cd $REGCM_RUN
$> srun sstCLM45 test_001.in
 ...
 ...
 ...
 Successfully generated SST
$> ls input
test_001_DOMAIN000.nc test_001_LANDUSE test_001_SST.nc

Create the ICBC files using the icbcCLM45 program

$> cd $REGCM_RUN
$> srun icbcCLM45 test_001.in
 ...
 ...
 ...
 Successfully completed ICBC
$> ls -1 input
test_001_DOMAIN000.nc
test_001_ICBC.1990060100.nc
test_001_ICBC.1990070100.nc
test_001_LANDUSE
test_001_SST.nc

Create surface data for CLM45

$> cd $REGCM_RUN
$> srun mksurfdataCLM45 test_001.in
 ...
 ...
 ...
 Successfully completed CLM preprocessing.
$> ls -1 input
test_001_CLM45_surface.nc
test_001_DOMAIN000.nc
test_001_ICBC.1990060100.nc
test_001_ICBC.1990070100.nc
test_001_LANDUSE
test_001_SST.nc
test_001_TEXTURE

First RegCM model simulation

$> cd $REGCM_RUN
$> mpirun -np 10 regcmMPICLM45 test_001.in
 ...
 ...
 ...
 RegCM V4 simulation successfully reached end
$> ls -1 output
test_001_ATM.1990060100.nc          test_001.clm.regcm.rh0.1990060600.nc  test_001_SAV.1990060600.nc  test_001_STS.1990060100.nc
test_001.clm.regcm.r.1990060600.nc  test_001_RAD.1990060100.nc            test_001_SRF.1990060100.nc

Relinquish allocated job

$> exit
logout
Connection to tux-01 closed.
$> exit
exit
salloc: Relinquishing job allocation 187744

Putting It All Together

If you wish to automate the workflow of the your simulation, it is best to execute the tasks NOT in interactive mode but in batch mode. To do that, first copy the this simple slurm job script and save as my_first_regcm_simulation_setup.slurm under $REGCM_RUN:

#!/bin/bash

#SBATCH --partition=batch
#SBATCH --job-name=my.first.regcm.simulation.setup
#SBATCH --ntasks=1
#SBATCH --output=stdout
#SBATCH --error=stderr
#SBATCH --mail-user=<your_email_address>
#SBATCH --mail-type=ALL

echo "Started: `date`" 

echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES

# Place commands to load environment modules here
module load regcm/4.6.0-intel-openmpi
module list

# Set stack size to unlimited 
ulimit -s unlimited

# MAIN
srun terrainCLM45 test_001.in
srun sstCLM45 test_001.in
srun icbcCLM45 test_001.in
srun mksurfdataCLM45 test_001.in

echo "Finished: `date`" 

Then run the following:

$> cd $REGCM_RUN
$> sbatch my_first_regcm_simulation_setup.slurm
Submitted batch job 187773

Wait for the job to successfully finished by using the squeue command or checking stdout. Afterwhich, the create a my_first_regcm_simulation_run.slurm that will use 10 tasks instead of 1 task

#!/bin/bash

#SBATCH --partition=batch
#SBATCH --job-name=my.first.regcm.simulation.run
#SBATCH --ntasks=10
#SBATCH --output=stdout
#SBATCH --error=stderr
#SBATCH --mail-user=<your_email_address>
#SBATCH --mail-type=ALL

echo "Started: `date`" 

echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES

# Place commands to load environment modules here
module load regcm/4.6.0-intel-openmpi
module list

# Set stack size to unlimited 
ulimit -s unlimited

# MAIN
mpirun regcmMPICLM45 test_001.in

echo "Finished: `date`" 

Then,

$> cd $REGCM_RUN
$> sbatch my_first_regcm_simulation_run.slurm
Submitted batch job 187774