SuperMUC: Changes in the usage of the fat node island
publish at lrz.de
Do Jul 25 17:05:48 CEST 2013
Dear users of the fat node system at LRZ, earlier known as "SuperMIG",
due to the integration of the fat node system into the SuperMUC
infrastructure, there will be some items you will need to take care of
before re-starting your job processing on this part of the system.
1. Login procedure: The name via which the login node is accessed has
changed to supermuc-fat.lrz.de
2. Scratch and Project Data Handling: Instead of the previously used
NAS volumes, the fat nodes are now provisioned with the same GPFS
file system as the thin nodes. The NAS volumes will remain
available in Read-Only mode until mid-November 2013. Please migrate
any still needed data to GPFS by copying them to GPFS on a login
node. For example:
cp -a $SCRATCH_MIGRATION/<dirname> <new_dirname>
cp -a $WORK_MIGRATION/<dirname> <new_dirname>
It is probably a good idea for <new_dirname> to have the same value
as <dirname> (if no name clashes occur). The paths pointed to by
the variables SCRATCH_MIGRATION and WORK_MIGRATION will be
available only on the login nodes.
Also do not forget to adjust your batch scripts if you either have
explicit path names in them, or if your directory structure has
changed through the copying procedure.
3. LoadLeveler job classes and scripts: Because the same instance of
LoadLeveler as for the thin nodes is now used, you will need to
adjust the job class names for scripts that should run on the fat
nodes: Please replace "general" by "fat" and "test" by "fattest".
For jobs in the "special" class, please add the specification
#@ requirements (Island == "island01")
to force them to run on the fat node island.
4. Software updates: A new version of the operating system as well as
the HPC software stack is now deployed. In particular, the
MPICH2-based version of IBM MPI (PE 1.3) replaces the legacy 5.2
version. We consider it advisable to recompile all applications
that previously were built for the fat nodes (see also next
5. Compilation procedure: Due to the different architecture of the
thin and fat nodes, an executable that is compiled with defaults on
the fat node does not take advantage of the AVX instructions on the
thin nodes. Therefore we recommend to compile your code with
"-axAVX -xSSE4.2" to target both types. This option tells the
compiler to generate multiple, processor-specific auto-dispatch
code paths for Intel processors.
For details consult the following documents:
This information is also available on our web server
Mehr Informationen über die Mailingliste aktuell