Cluster Usage Instructions

The Computing Cluster at NYSBC is now fully operational. Requests for processing time can be made to infotech@nysbc.org.

Technical Specifications

Compiling Spider Versions

1. export F77=pathf90
2. export F90=pathf90
3. export MPI_ROOT=/opt/hpmpi/bin
4. export PATH=/opt/hpmpi/bin:$PATH
5. export LD_LIBRARY_PATH=/opt/hpmpi/lib/linux_amd64:/opt/hpmpi/include/64
6. export MPI_F90=pathf90
7. export PATHSCALE_SUBSCRIPTION_DAEMON=node0
8. From within the src directory in spider, run this makefile - Makefile_nysbc_cluster

Sample PBS Job File

#!/bin/sh

#PBS -l nodes=4:ncpus=4

/opt/hpmpi/bin/mpirun -np 16 -hostfile $PBS_NODEFILE <path-to-mpi-app>

To run the job, run qsub <job-file>

Using PBS in SPIDER to replace pubsub

• Only multiref alignment in spide rhas been optimised in spider mpi -- other commands are not accelerated
• Additionally, only a subset of commands are mp aware in spider
• Can use pbs to speed up loops on non-mp aware processes
• example of a spider script to speed up a non-linear filter

;========================input
fr l
[tomo]tomo1_crop.spd
;=========output=================
fr l
[filt]filt
;=========params==============
x90=60          ; cycles
x91=0.1         ; delta t
x92=0           ; sigma
;x93=0.05       ; lambda

;=============================
;---------- following set of commands sets the main qsub script for the cluster, one script per loop event
vm
echo "#!/bin/sh" > qsub-header.txt
vm
echo "#PBS -l select=ncpus=1:mem=6GB " >> qsub-header.txt
vm
echo "#PBS -l place=scatter" >> qsub-header.txt
vm
echo "cd /usr/data/asiebert/30_04_08_Rbs/tomo1" >> qsub-header.txt

;-----------------------------------------------

do lb1 x10=1,10                 ; this loop only makes the scripts using a "slave" script as template

  RR x93             ; adjust lambda according to the following sequence
  0.01,0.02,0.05,0.1,0.2,0.5,1,2,5,10                               ;  a nice log sequence

   vm
   echo "x93={**X93}" > filtscript{**x10}.$PRJEXT
   vm
   echo "x10={**X10}" >> filtscript{**x10}.$PRJEXT
   vm
   cat spider_script_filt1_clu_slave.$PRJEXT >> filtscript{**x10}.$PRJEXT
   vm
   cp qsub-header.txt qsub-script{**x10}.txt
   vm
   echo "/usr/apps/spidermp $PRJEXT/$DATEXT @filtscript{**x10}" >> qsub-script{**x10}.txt
   vm
   qsub qsub-script{**x10}.txt
lb1

en      ; end the main script                                                                  
 

• The script which actually does the work:

;========================input
fr l
[tomo]tomo1_crop.spd
;=========output=================
fr l
[filt]filt
;=========params==============
x90=60          ; cycles
x91=0.1         ; delta t
x92=0           ; sigma
;x93=0.05       ; lambda

;=============================

   ce ad
   [tomo]
   [filt]{***x10}
   HEG
   x90
   x91
   x92,x93
en d

User Manuals

1. HP MPI User Manual
2. Userguide for PathScale compiler suite (F77/90/95, C, C++)
3. Manpage for PathScale compiler
4. PBS Pro User Guide

• Set ALLOWTOPICVIEW =

-- DhireshVyas - 26 Feb 2007