|
Firstly, thanks very much to Alan Iwi and Fatima Chami at JASMIN for some of the information and links below.
On my page about what is required to run JULES, I specified that you will need NetCDF libraries to be installed in order to do any run of JULES that is not just a single-point run (and I pointed to my page about checking the NetCDF libraries you have are correctly set up). For parallel runs of JULES, however, the libraries you will need may be different from the standard ones. Below is some useful information about parallel NetCDF libraries on JASMIN. If you are not using the JASMIN platform, then I am afraid I can't help you too much (!), BUT most likely you will need module load commands and environment variables not too dissimilar from those for JASMIN, so show this page to your System Administrator and ask her/him for the equivalent specifications. |
On JASMIN, you will need the following, depending on whether you are running JULES in serial or parallel, and depending on whether you are using the Gfortran or Ifort compilers:
*** FOUR IMPORTANT POINTS ***
Where do you put all this information when running JULES? In two different locations depending on whether you are using Rose or not (n.b. $JULES_ROOT is the location of the top directory of the version of JULES you are using (i.e. the model source code can be found in $JULES_ROOT/src/) and $RSUITE is the location of the top directory of the Rose suite you are using (i.e. the rose-suite.info file can be found in $RSUITE/)):
(1) If you are doing a run outside Rose (see my JULES From Scratch tutorial for what that is):
Firstly, look in $JULES_ROOT/etc/fcm-make/make.cfg and on line 22 you'll find the value of JULES_PLATFORM your run is using. That means your suite is currently pulling in the options in $JULES_ROOT/etc/fcm-make/platform/[value of JULES_PLATFORM].cfg during its compile step.
Next, EITHER look at the various $JULES_ROOT/etc/fcm-make/platform/*.cfg files available and select the one most appropriate for your run (by the way I personally added the files specifically for options #1 and #3 above jasmin-*-nompi.cfg back in 2021) and modify that line 22 to point to it OR put "custom" on line 22 and edit the custom.cfg file with the options above for $JULES_NETCDF_INC_PATH, etc.
(2) If you are doing a run using Rose (see my JULES From Scratch tutorial):
Look in $RSUITE/suite.rc and add the module load commands from above into the [runtime] section (see e.g. my suite u-da403 for exactly where) and (ii) modify the platform file your run is using as in (1).
- Compiling with GCC/Gfortran, serial runs: Do module load jaspy and then set:
$JULES_NETCDF_INC_PATH = $CONDA_PREFIX/include
$JULES_NETCDF_LIB_PATH = $CONDA_PREFIX/lib - Compiling with GCC/GFortran, parallel runs: this is not currently possible on Jasmin (according to support responses to my support tickets #59087 and #60760 from April 2023, the parallel NetCDF library via JASPY is built against mpich for GFortran, which is not compatible with the current OpenMPI implementation; for some comments on the difference between mpich and openmpi, see here)
- Compiling with Intel/ifort, serial runs: Do module load intel/19.0.0 then module load eb/OpenMPI/intel/3.1.1 (see here; the second I'm told is necessary whether using MPI or not) and then set:
$JASMIN_JULES_BASE_DIR = /gws/nopw/j04/jules
$NETCDF_FORTRAN_ROOT = $JASMIN_JULES_BASE_DIR/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0
$JULES_NETCDF_INC_PATH = $NETCDF_FORTRAN_ROOT/include
$JULES_NETCDF_LIB_PATH = $NETCDF_FORTRAN_ROOT/lib - Compiling with Intel/ifort, parallel runs: as for serial runs but I also need to set: $LD_LIBRARY_PATH = $LD_LIBRARY_PATH:$JASMIN_JULES_BASE_DIR/admin/curl/curl-lotus-parallel-intel/lib/:$NETCDF_FORTRAN_ROOT/lib
*** FOUR IMPORTANT POINTS ***
- By 'parallel' above I mean multi-node runs (MPI), not multi-thread (OMP): for multi-thread runs the memory is shared and therefore the NetCDF libraries are as for serial runs.
- You will need access to the JULES Group Workspace (GWS) for options #3 and #4.
- The requirements above are not just for the JULES model: they also apply to any other Fortran model being run on JASMIN (if I use intel to compile any simple Fortran program that includes "USE netcdf" at the top, then it needs access to the JULES GWS and will abort if it's not there).
- Can you have both Python and Intel loaded at the same time? (e.g. to compile code using ifort and then analyse the results using Python). Short answer: not currently on JASMIN (see response to support ticket #64780 from November 2023 for more details).
Where do you put all this information when running JULES? In two different locations depending on whether you are using Rose or not (n.b. $JULES_ROOT is the location of the top directory of the version of JULES you are using (i.e. the model source code can be found in $JULES_ROOT/src/) and $RSUITE is the location of the top directory of the Rose suite you are using (i.e. the rose-suite.info file can be found in $RSUITE/)):
(1) If you are doing a run outside Rose (see my JULES From Scratch tutorial for what that is):
Firstly, look in $JULES_ROOT/etc/fcm-make/make.cfg and on line 22 you'll find the value of JULES_PLATFORM your run is using. That means your suite is currently pulling in the options in $JULES_ROOT/etc/fcm-make/platform/[value of JULES_PLATFORM].cfg during its compile step.
Next, EITHER look at the various $JULES_ROOT/etc/fcm-make/platform/*.cfg files available and select the one most appropriate for your run (by the way I personally added the files specifically for options #1 and #3 above jasmin-*-nompi.cfg back in 2021) and modify that line 22 to point to it OR put "custom" on line 22 and edit the custom.cfg file with the options above for $JULES_NETCDF_INC_PATH, etc.
(2) If you are doing a run using Rose (see my JULES From Scratch tutorial):
Look in $RSUITE/suite.rc and add the module load commands from above into the [runtime] section (see e.g. my suite u-da403 for exactly where) and (ii) modify the platform file your run is using as in (1).