SuperMUC: Changes in the usage of the fat node island

[LRZ aktuell]

Dear users of the fat node system at LRZ, earlier known as "SuperMIG",
 
 due to the integration of the fat node system into the SuperMUC
 infrastructure, there will be some items you will need to take care of
 before re-starting your job processing on this part of the system.
 
  1. Login procedure: The name via which the login node is accessed has
     changed to supermuc-fat.lrz.de
  2. Scratch and Project Data Handling: Instead of the previously used
     NAS volumes, the fat nodes are now provisioned with the same GPFS
     file system as the thin nodes. The NAS volumes will remain
     available in Read-Only mode until mid-November 2013. Please migrate
     any still needed data to GPFS by copying them to GPFS on a login
     node. For example:
     cd $SCRATCH
     cp -a $SCRATCH_MIGRATION/<dirname>  <new_dirname>
    
     or
    
     cd $WORK
     cp -a $WORK_MIGRATION/<dirname> <new_dirname>
    
     It is probably a good idea for <new_dirname> to have the same value
     as <dirname> (if no name clashes occur). The paths pointed to by
     the variables SCRATCH_MIGRATION and WORK_MIGRATION will be
     available only on the login nodes.
    
     Also do not forget to adjust your batch scripts if you either have
     explicit path names in them, or if your directory structure has
     changed through the copying procedure.
  3. LoadLeveler job classes and scripts: Because the same instance of
     LoadLeveler as for the thin nodes is now used, you will need to
     adjust the job class names for scripts that should run on the fat
     nodes: Please replace "general" by "fat" and "test" by "fattest".
     For jobs in the "special" class, please add the specification
     #@ requirements (Island == "island01")
     to force them to run on the fat node island.
  4. Software updates: A new version of the operating system as well as
     the HPC software stack is now deployed. In particular, the
     MPICH2-based version of IBM MPI (PE 1.3) replaces the legacy 5.2
     version. We consider it advisable to recompile all applications
     that previously were built for the fat nodes (see also next
     section).
  5. Compilation procedure: Due to the different architecture of the
     thin and fat nodes, an executable that is compiled with defaults on
     the fat node does not take advantage of the AVX instructions on the
     thin nodes. Therefore we recommend to compile your code with
     "-axAVX -xSSE4.2" to target both types. This option tells the
     compiler to generate multiple, processor-specific auto-dispatch
     code paths for Intel processors.
    
     For details consult the following documents:
       + http://www.lrz.de/services/compute/supermuc/access_and_login/
       + http://www.lrz.de/services/compute/supermuc/loadleveler/
       + http://www.lrz.de/services/compute/supermuc/filesystems/
 


 This information is also available on our web server
 http://www.lrz-muenchen.de/services/compute/supermuc/aktuell/ali4626/

 Reinhold Bader