Running Chapel Programs on Titan

Last week I had the opportunity to use Titan, #2 supercomputer in the world as of June 2013, as part of the ATPESC program hosted by Argonne labs. Titan is a Cray XK7 machine and its user’s guide is available here.
The following is a quick start to get Chapel programs run on multi-locale with Titan.

• Modules – the default set of modules is all what you need. You may load PrgEnv-cray/4.1.40 instead of the one loaded.
• Environment – you’ll need to have the following variables exported, usually through ~/.bashrc (in my case I used Chapel 1.7.0)

export CHPL_LOC=/tmp/work/csep65/chapel-1.7.0
export CHPL_HOME=$CHPL_LOC
export CHPL_COMM=gasnet
export GASNET_QUIET=yes
export CHPL_HOST_PLATFORM=cray-xk
export CHPL_LAUNCHER=pbs-aprun
export CHPL_LAUNCHER_QUEUE=batch
export CHPL_LAUNCHER_WALLTIME=00:10:00
export CHPL_LAUNCHER_ACCOUNT=TRN001
export CHPL_HOST_COMPILER=gnu
export PATH=${CHPL_LOC}/bin/$CHPL_HOST_PLATFORM:$PATH
Also, note the values in red need to be changed accordingly. The CHPL_LAUNCHER_ACCOUNT will be the account in Titan to get billed. You can find more information on this in the $CHPL_HOME/doc/platforms/README.cray (search for NCCS).

• Launcher tweak – note this fix is avaialble in Chapel code trunk following revision 21702 (http://sourceforge.net/p/chapel/code/21702/tree/trunk/) so will be necessary only if you are using a release 1.7.0 or older. A minor change to Chapel’s PBS launcher is necessary for these versions to work with Titan due to some upgrade in the qsub wrapper used by Titan. The fix is to change the two occurrences of –l size with –l nodes in $CHPL_HOME/runtime/src/launch/pbs-aprun/launch-pbs-aprun.c
• Compile Chapel – once the above steps are completed use the Makefile under $CHPL_HOME to compile Chapel as usual
• Compile your program – use the chpl compiler to compile your program, which will create two outputs, for example a.out and a.out_real. More information on what each of this means is available in the $CHPL_HOME/doc/README.multilocale
• Start your program – you can use, for example ./a.out –nl 4 to launch your program on 4 locales. Chapel launcher will take care of submitting the correct PBS job for Titan’s job scheduler.

This is pretty much what you need to do and hope it helps!