Changeset 1424 for trunk/LMDZ.GENERIC

Timestamp:

May 7, 2015, 5:38:31 PM (10 years ago)

Author:

milmd

Message:

Update user manuel to compile and run LMDZ.COMMON model in parallel.

File:

• trunk/LMDZ.GENERIC/DOC/run.tex (modified) (6 diffs)

trunk/LMDZ.GENERIC/DOC/run.tex

@@ -135 +135 @@
   \end{verbatim}
 
-\section{Compiling the model}
+\section{Compiling the LMDZ.GENERIC model (sequential only)}
 \label{sc:run1}
 
@@ -193 +193 @@
 makegcm -d 32x32x20 -p std -O "-g -fpe0 -traceback" gcm
 \end{verbatim}
-
+%**********
+\section{Compiling the LMDZ.COMMON model (sequential or parallel)}
+\label{sc:run1_common}
+\begin{enumerate}
+\item Prerequisites:
+\begin{itemize}
+\item[$\bullet$] Downloaded LMDZ.COMMON and LMDZ.OTHER\_MODEL containing the physic you want.
+\item[$\bullet$] Available MPI library and wrapped compiler (mpif90, mpiifort,...)
+\item[$\bullet$] Optional (but recommended) fcm:
+\begin{itemize}
+\item LMD: /distrib/local/fcm/bin
+\item Ciclad: /home/millour/FCM\_V1.2/bin
+\item Gnome: /san/home/millour/FCM\_V1.2/bin
+\item Other: fcm is just a collection of perl scripts; can be copied over on any other machine, or simply downloaded using svn:\\
+svn checkout http://forge.ipsl.jussieu.fr/fcm/svn/PATCHED/FCM\_V1.2
+\end{itemize}
+\end{itemize}
+\item Then choose the physic you want to couple with the LMDZ.COMMON dynamic core by creating a symbolic link in the LMDZ.COMMON/libf directory.\\
+If you want to use mars physic:
+\begin{verbatim}
+cd LMDZ.COMMON/libf
+ln -s path/to/LMDZ.MARS/libf/phymars .
+ln -s path/to/LMDZ.MARS/libf/aeronomars .
+\end{verbatim}
+Here, we want the LMDZ.GENERIC physic phystd:
+\begin{verbatim}
+cd LMDZ.COMMON/libf
+ln -s path/to/LMDZ.GENERIC/libf/phystd .
+\end{verbatim}
+\item  To compile in LMDZ.COMMON directory:
+\begin{verbatim}
+./makelmdz_fcm -s XX -t XX -d LONxLATxALT -b IRxVI -p physicSuffix 
+-arch archFile [-parallel mpi/mpi_omp] gcm
+\end{verbatim}
+\begin{itemize}
+\item[$\bullet$] \textbf{physicSuffix} is \verb|mars| for phymars, \verb|std| for phystd...
+\item[$\bullet$] \textbf{archFile} is the name of configuration files from LMDZ.COMMON/arch: use \verb|CICLADifort| the ifort compiler in a CICLAD environment, \verb|X64_ADA| for the ADA architecture...
+\item[$\bullet$] To compile in parallel with mpi, add \verb|-parallel mpi| option. By default it is serial code.
+\item[$\bullet$] For hybrid MPI-OpenMP parallelisation, add \verb|-parallel mpi_omp| option.
+\item[$\bullet$] For faster compilation, the option \verb|-j N| uses N simultaneous tasks.
+\item[$\bullet$] \verb|-full| option forces full (re)-compilation from scratch.
+\item[$\bullet$] Created program is in LMDZ.COMMON/bin directory, with dimensions included in the program name. e.g.: gcm\_64x48x29\_phymars\_para.e
+\end{itemize}
+\end{enumerate}
+NB: It is possible to compile without fcm by replacing \verb|makelmdz_fcm| by \verb|makelmdz|. Created program is in LMDZ.COMMON directory and named gcm.e.
+%**********
 \section{Input files (initial states and def files)}
 {\bf -} In directory \verb+LMDZ.GENERIC/deftank+
@@ -224 +269 @@
 
 [NOTE: WITH THE GENERIC MODEL WE ALMOST ALWAYS START FROM ``startplanet'' FILES]
-
+%**********
 \section{Running the model}
 \begin{figure}
@@ -232 +277 @@
 \end{figure}
 
+IMPORTANT: The following line MUST be in file run.def (or callphys.def): 
+\begin{verbatim}
+planet_type = mars
+\end{verbatim}
+for using LMDZ.MARS model or
+\begin{verbatim}
+planet_type = generic
+\end{verbatim}
+for using LMDZ.GENERIC model.
+
+\begin{itemize}
+\item[$\bullet$] To run the serial {\bf gcm.e} interactively:\\
 Once you have the program {\bf gcm.e},
 input files {\bf start.nc} {\bf startfi.nc},
@@ -240 +297 @@
 \end{verbatim}
 
+You might need more memory. Use \verb|ulimit -s unlimited| to change user limits.\\
 You might also want to keep all messages and diagnostics written to standard
 output (i.e. the screen). You should then redirect the standard output
@@ -253 +311 @@
 
 
+\item [$\bullet$] To run the MPI-parallel {\bf gcm.e} interactively:
+\begin{verbatim}
+mpirun -np N gcm.e > gcm.out 2>&1
+\end{verbatim}
+\verb|-np N| specifies the number of procs to run on.\\
+IMPORTANT: one MUST use the \verb|mpirun| command corresponding to the \verb|mpif90| compiler specified in the \verb|arch| file.\\
+Output files (restart.nc, diagfi.nc ,etc.) are just as when running in serial. But standard output messages are written by each process.\\
+If using chained simulations (run\_mcd/run0 scripts), then the command line to run the gcm in \verb|run0| must be adapted for local settings.\\
+NB: LMDZ.COMMON dynamics set to run in double precision, so keep \verb|NC_DOUBLE| declaration (and real to double precision promotion) in the arch files.
+\item [$\bullet$] To run the hybrid parallel {\bf gcm.e} interactively:
+\begin{verbatim}
+export OMP_NUM_THREADS=2
+export OMP_STACKSIZE=2500MB
+mpirun -np 2 gcm.e > gcm.out 2>&1
+\end{verbatim}
+In this exemple, each of the 2 process MPI have 2 OpenMP tasks with a 2500MB memory.
+\item[$\bullet$] To run the MPI-parallel {\bf gcm.e} with a job scheduler (different on each machine):
+\begin{verbatim}
+PBS example (on Ciclad):
+#PBS -S  /bin/bash
+#PBS -N  job_mpi08
+#PBS -q short
+#PBS -j eo
+#PBS -l "nodes=1:ppn=8"
+# go to directory where the job was launched
+cd $PBS_O_WORKDIR
+mpirun gcm_64x48x29_phymars_para.e > gcm.out 2>&1
+\end{verbatim}
+\begin{verbatim}
+LoadLeveler example (on Gnome):
+# @ job_name = job_mip8
+# standard output file  
+# @ output = job_mpi8.out.$(jobid)
+# standard error file
+# @ error =  job_mpi8.err.$(jobid)
+# job type
+# @ job_type = mpich
+# @ blocking = unlimited
+# time
+# @ class = AP
+# Number of procs 
+# @ total_tasks = 8
+# @ resources=ConsumableCpus(1) ConsumableMemory(2500 mb)
+# @ queue
+set -vx
+mpirun gcm_32x24x11_phymars_para.e > gcm.out 2>&1
+\end{verbatim}
+\begin{verbatim}
+LoadLeveler example (on Ada):
+module load intel/2012.0
+# @ output  =  output.$(jobid)
+# @ error = $(output)
+# @ job_type = parallel
+## Number of MPI process
+# @ total_tasks = 8
+## Memory used by each MPI process
+# @ as_limit = 2500mb 
+# @ wall_clock_limit=01:00:00
+# @ core_limit = 0
+# @ queue
+set -x
+poe ./gcm.e -labelio yes > LOG 2>&1
+\end{verbatim}
+\item[$\bullet$] To run the hybrid MPI/OpenMP-parallel {\bf gcm.e} with a job scheduler (different on each machine):
+\begin{verbatim}
+LoadLeveler example (on Gnome):
+# @ job_name = job_mip8
+# standard output file  
+# @ output = job_mpi8.out.$(jobid)
+# standard error file
+# @ error =  job_mpi8.err.$(jobid)
+# job type
+# @ job_type = mpich
+# @ blocking = unlimited
+# time
+# @ class = AP
+# Number of procs 
+# @ total_tasks = 8
+# @ resources=ConsumableCpus(1) ConsumableMemory(5000 mb)
+# @ queue
+set -vx
+export OMP_NUM_THREADS=2 #sinon par defaut, lance 8 threads OpenMP
+export OMP_STACKSIZE=2500MB
+mpirun gcm_32x24x11_phymars_para.e > gcm.out 2>&1
+\end{verbatim}
+IMPORTANT: ConsumableMemory must be equal to OMP\_NUM\_THREADSxOMP\_STACKSIZE.\\
+In this case, we are using 8x2 cores.
+\begin{verbatim}
+LoadLeveler example (on Ada):
+module load intel/2012.0
+# @ output  =  output.$(jobid)
+# @ error = $(output)
+# @ job_type = parallel
+## Number of MPI process
+# @ total_tasks = 8
+## Number of OpenMP tasks attached to each MPI process
+# @ parallel_threads = 2
+## Memory used by each MPI process
+# @ as_limit = 5gb 
+# @ wall_clock_limit=01:00:00
+# @ core_limit = 0
+# @ queue
+set -x
+export OMP_STACKSIZE=2500MB
+poe ./gcm.e -labelio yes > LOG 2>&1
+\end{verbatim}
+IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so $\verb|as_limit|=2 \times 2.5$.
+\end{itemize}
+%**********
 \section{Visualizing the output files}