Linux Cluster
Using the CCV “ted” Linux cluster for computation
ted.ccv.brown.edu is a dual CPU Pentium III SMP machine which is the front-end for a cluster of 48 dual Xeon nodes (96 CPUs @ 1.7GHz, 2GB memory per node), interconnected by Myrinet and by Gigabit ethernet. All nodes and the front-end run RedHat Linux under the Rocks Cluster Management Software.
Logging in to the Cluster
The front-end, "ted", is used for login and submission of jobs to the compute nodes using the PBS/maui queuing/scheduling system. Login should be through the CCV gateway machine (ssh.ccv.brown.edu) and then to ted as follows:
Login to CCV
The gateway machine is ‘ssh.ccv.brown.edu’. Use ssh:
ssh ssh.ccv.brown.edu -l <userid>
password:
Login to ted:
After connecting to ssh.ccv.brown.edu, login to ted using ssh:
ssh ted
(password is not required)
Setting the Environment
The scripts in /etc/profile.d are automatically executed for logins and subshells. These scripts provide the environment variables and PATH elements needed to use some of the packages on the system. If your home directory startup files explicitly set PATH (or path for tcsh/csh), this setup information will be lost.
If you run csh or tcsh, each script from /etc/profile.d/*.csh is sourced by /etc/csh.cshrc.
If you run ksh or bash, each script from /etc/profile.d/*.sh is sourced by /etc/bashrc.
One of the scripts, ssh-key.sh (or .csh) checks in $HOME/.ssh, looking for identity.pub, identity, and authorized_keys.
If identity.pub is not present, the three files are created.
You have the option of giving a passphrase to protect the identity file, which is probably a good idea for enhanced security. If you DO use a (non-null) passphrase, click here.
Compiling with MPI
Both GNU and Intel compilers are installed on the cluster;
g++, gcc and g77 are in /usr/bin (gcc 2.6)
g++, gcc and g77 are in /opt/gcc32/bin (gcc 3.2)
ifc and icc are in /opt/intel/compiler70/ia32/bin (Intel 7.0)
The compiler scripts that use the installed mpich libraries are in different directories depending on compiler and interconnect:
GNU: /opt/mpich/ethernet/gcc/bin for ethernet
GNU: /opt/mpich/myrinet/gcc/bin for myrinet
Intel: /opt/mpich/Ethernet/icc/bin for Ethernet
Intel: /opt/mpich/Ethernet/icc/bin for ethernet
The scripts are mpicc, mpiCC, mpif77, mpif90
There are some scripts to invoke at login that set up MPI environment.
First you must set, in, say .login:
MPI_FLAVOR = gmpi for GNU compilers
MPI_FLAVOR = impi for Intel compilers
Then source /etc/profile.d/gmpi.csh or /etc/profile.d/impi.csh.
NOTE: at present (9/8/03) these scripts setup Ethernet only.
You must set the environment by hand to use myrinet.
Compilation:
After setting environment as above, run the mpiXX command on your code.
Submitting a Job for Execution
Use PBS/maui queuing/scheduling system to run a job on the compute nodes.
NOTE: man pages are in /opt/OpenPBS/man.
Start with man pbs and man qsub for more detail.
First edit a ‘job file’ to be submitted to the PBS system. This file contains PBS parameters to specify nodes, as well as
commands to run on the nodes. A sample job file looks like this:
----------- Begin Job File (e.g. myjob.pbs)-----------------
## This (or a blank line) is a comment (2 ## at beginning)
## specify a jobname ( #PBS indicates a PBS specification)
#PBS -N my_job_name
## declare job non-rerunnable
#PBS -r n
## specify a name for the output file
#PBS -o my_job_name.out
## join stderr to stdout, so we have just one job log file
#PBS -j oe
## mail to user on a=abort, b=begin, e=end
#PBS -m abe
## specify user for mail
#PBS -M primary_email_address
## specify number of nodes and number of CPUs per node
## this examples allocates 2 CPUs on each of 4 nodes
#PBS -l nodes=4:ppn=2
## specify wallclock time
#PBS -l walltime=hh:mm:ss
## export current environment variables to the jobs nodes
#PBS -V
## PBS specifications are done; now set up the job with shell
## shell commands (no # at beginning) and run your executable
## switch to the job's working directory, provided by PBS
## it's the directory from which you submitted your job.
cd $PBS_O_WORKDIR
## useful info for the log file
echo Master process running on `hostname`
echo Directory is `pwd`
echo PBS has allocated the following nodes:
echo `cat $PBS_NODEFILE`
## put in a timestamp
echo Starting execution at `date`
## PBS has selected the nodes; now set the number of CPUs
set NPROCS=`wc -l < $PBS_NODEFILE`
echo This job has allocated $NPROCS CPUs
## Run the executable
mpirun -np $NPROCS -machinefile $PBS_NODEFILE my_executable
## Another timestamp at end
echo Finished execution at `date`
----------- End Job File ----------------------------------------------
Once the file is edited, submit the program to PBS using qsub: qsub myjob.pbs
Monitor the status of PBS jobs with qstat –a or showq
A web-based graphical monitor is at http://www.ccv.brown.edu/ganglia-webfrontend/
Select ‘Source -> Graphics Cluster’
To query activity on the nodes and execute shell commands, use these two commands that access all nodes sequentially:
cluster-ps <userid> lists processes running for <userid> on all nodes.
cluster-fork <command> executes <command> on all nodes
Disk Storage
When your PBS job starts, your home directory will be auto-mounted on the compute nodes at /users/<userid>. /usr/local and /opt are also mounted on the nodes. The /users filesystem is approximately 500 GB, and programs and data files should be put into your home directory and subdirectories. At this point (9/8/03) we are setting up an additional filesystem called `projects` for data files, but until that is ready you should use your home directory for all files.
NOTE to users who have used the DATA -> /scratch filesystem for large data files in the past: do not use the DATA filesystem when running on ted. It would be best to copy these files to your home directory pending the setup of the `projects` filesystem
