Running the Isca framework on the cluster

Installation

First of all define a folder ${WORK} on the /work filesystem (somewhere where you have write permissions):

export WORK=/work/FAC/...
mkdir -p ${WORK}

Load the following relevant modules and create a python virtual environment:

module load gcc/10.4.0
module load mvapich2/2.3.7
module load netcdf-c/4.8.1-mpi
module load netcdf-fortran/4.5.4
module load python/3.9.13

python -m venv ${WORK}/isca_venv

Install the required python modules:

${WORK}/isca_venv/bin/pip install dask f90nml ipykernel Jinja2 numpy pandas pytest sh==1.14.3 tqdm xarray

Download and install the Isca framework:

cd ${WORK}
git clone https://github.com/ExeClim/Isca
cd Isca/src/extra/python
${WORK}/isca_venv/bin/pip install -e .

Patch the Isca makefile:

sed -i 's/-fdefault-double-8$/-fdefault-double-8 \\\n           -fallow-invalid-boz -fallow-argument-mismatch/' ${WORK}/Isca/src/extra/python/isca/templates/mkmf.template.gfort

Create the environment file for curnagl:

cat << EOF > ${WORK}/Isca/src/extra/env/curnagl-gfortran
echo Loading basic gfortran environment

module load gcc/10.4.0
module load mvapich2/2.3.7
module load netcdf-c/4.8.1-mpi
module load netcdf-fortran/4.5.4

# this defaults to ia64, but we will use gfortran, not ifort
export GFDL_MKMF_TEMPLATE=gfort

export F90=mpifort
export CC=mpicc
EOF

Compiling and running the Held-Suarez dynamical core test case

Compilation takes place automatically at runtime. After logging in to the clustercluster, youcreate alwaysa needSLURM toscript setupfile start.sbatch with the environmentfollowing with:contents:

#!/bin/bash -l

#SBATCH --account ACCOUNT_NAME
#SBATCH --mail-type ALL 
#SBATCH --mail-user <first.lastname>@unil.ch

#SBATCH --chdir ${WORK}
#SBATCH --job-name isca_held-suarez
#SBATCH --output=isca_held-suarez.job.%j

#SBATCH --partition cpu

#SBATCH --nodes 1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 16
#SBATCH --mem 8G
#SBATCH --time 00:29:59
#SBATCH --export NONE

module load gcc/10.4.0
module load mvapich2/2.3.7
module load netcdf-c/4.8.1-mpi
module load netcdf-fortran/4.5.4

module load python/3.9.13

export GFDL_BASE=${WORK}/Isca
export GFDL_ENV=curnagl-gfortran
export GFDL_WORK=${WORK}/isca_work
export GFDL_DATA=${WORK}/isca_gfdl_data

export C_INCLUDE_PATH=${NETCDF_C_ROOT}/include
export LIBRARY_PATH=${NETCDF_C_ROOT}/lib

sourcesed -i "s/^NCORES =.*$/NCORES = $(echo ${SLURM_CPUS_PER_TASK:-1})/" ${GFDL_BASE}/exp/test_cases/held_suarez/held_suarez_test_case.py

${WORK}/isca_venv/bin/activatepython $GFDL_BASE/exp/test_cases/held_suarez/held_suarez_test_case.py

Then you can changesimply tostart the directory of the Held-Suarez experiment and run:job:

cdsbatch $GFDL_BASE/exp/test_cases/held_suarez
python held_suarez_test_case.pystart.sbatch