Getting started on Hydra

This document provides information relevant to users migrating from other HPC systems to the hydra supercomputer. Some special hints for users migrating from the Power6 architecture are given in the last section below..

 

MPI parallel HPC applications

By default, the Intel compilers for Fortran/C/C++ and the IBM MPI implementation are available on hydra.

The MPI wrapper executables are mpiifort, mpiicc and mpiicpc. These wrappers pass include and link information for MPI together with compiler-specific command line flags down to the Intel compiler.

More information and links to the reference guides for the Intel compilers are provided on the following pages:

 

 

Multithreaded (OpenMP) or hybrid (MPI/OpenMP) HPC applications

To compile and link OpenMP applications pass the flag -openmp to the Intel compiler.

In some cases it is necessary to increase the private stack size of the threads at runtime, e.g. when threaded applications exit with segmentation faults. On hydra the thread private stack size is set via the environment variable KMP_STACKSIZE. The default value is 4 megabytes (4m).

For example, to request a stack size of 128 megabytes on hydra, set KMP_STACKSIZE=128m.

For information on compiling applications which use pthreads please consult the Intel C/C++ compiler documentation.

 

Intel Math Kernel Library overview

The Intel Math Kernel Library (MKL) is provided on hydra as the standard high-performance mathematical library. MKL provides highly optimized implementations of (among others)

  • LAPACK/BLAS routines,
  • direct and iterative solvers,
  • FFT routines,
  • ScaLAPACK.

Parts of the library support thread or distributed-memory parallelism.

Extensive information on the features and the usage of MKL is provided by the official Intel MKL documentation.

 

Linking programs with MKL

By default, an MKL environment module is already loaded. The module sets the environment variables MKL_HOME and MKLROOT to the installation directory of MKL. These variables can then be used in makefiles and scripts.

The Intel MKL Link Line Advisor is often useful to obtain information on how to link programs with MKL. For example, to link statically with the threaded version of MKL/11.0 on hydra (Linux, Intel64) using standard 32 bit integers, pass the following command line arguments to the Intel compiler:

-Wl,--start-group
$(MKLROOT)/lib/intel64/libmkl_intel_lp64.a
$(MKLROOT)/lib/intel64/libmkl_intel_thread.a
$(MKLROOT)/lib/intel64/libmkl_core.a 
-Wl,--end-group -lpthread -lm -openmp

Select MPICH2 in the Link Line Advisor in case you intend to use MPI parallel routines from MKL on hydra using IBM MPI.

 

Performance optimization

For guidance on how to achieve good performance on hydra please consult the
IBM Performance Guide for HPC Applications on the IBM System x iDataPlex.

 

Execution of (parallel) programs

Parallel programs on hydra are started with POE, IBM's Parallel Operating Environment. You may find the Overview of IBM's Parallel Environment interesting to read.

Simple interactive example

To run an MPI program interactively (for debug purposes) please follow these steps:

  • Compile your program, e.g.

    mpiifort myprog.f -o myprog

  • Create a file named host.list with a line "localhost" for each processor you intend to use. To use, say, 4 processors of a node, there are four lines with "localhost" (without quotes) in host.list.

  • Run your program on one of the interactive nodes hydra-i.rzg.mpg.de using the command

    poe ./myprog -procs 4
    (poe's options must appear after the name of your binary)

To run a hybrid MPI/OpenMP program interactively set the following environment variables in addition to the above mentioned:

  • export OMP_NUM_THREADS=<nr_of_threads_per_task>
  • export MP_TASK_AFFINITY=core:$OMP_NUM_THREADS

 

Batch jobs

For production runs it is necessary to run the MPI program as a batch job. Please refer to the LoadLeveler batch system page and the sample batch job scripts for further information.

 

Some additional hints for users migrating from the Power6 system vip

To compile and link OpenMP applications pass the flag -openmp to the Intel compiler. It replaces the flag -qsmp used by the XL compilers on vip.

The environment variable KMP_STACKSIZE replaces the variable XLSMPTOPTS known to vip users. For example, to request a stack size of 128 megabytes on hydra, set KMP_STACKSIZE=128m which is equivalent to XLSMPTOPTS="stack=128000000" on vip.

The Intel Math Kernel Library (MKL) is provided on hydra as the standard high-performance mathematical library. For users migrating their code from vip it replaces IBM ESSL and pESSL.

Document Actions