skip navigation

This page is designed for modern browsers. You will have a better experience with a better browser.

Brown Home Brown Home Brown University Brown University Brown University Brown University CCV

IBM SP Cluster

Using the IBM SP cluster

The IBM "SP" is a 148-processor Scalable POWERparallel cluster, and has provided the backbone of CCV's compute power for several years. This document shows how to start using it.

The operating environment

All the machines in the SP run AIX, IBM's proprietary version of the UNIX operating system. Currently (8/2002) they run version 4.3.

For more about AIX and its working environment, see the SP System Software description.

Hardware on the SP

The SP is divided into three computational "pools" based on processor type, as shown below.

("nodes": physically separate computers containing one or more processors)

("FMA": a Floating-Multiply-Add unit, yielding one add, multiply, or add-multiply result per cycle)

CPU type # Nodes CPUs/ node memory/ node FMAs/ CPU Flops/node
332MHz PowerPC 604e 22 4 1 GB 1 2.66 GFlops
200MHz Power3 10 2 2 GB 2 1.76 GFlops
220MHz Power3 5 8 8 Gb 2 7.0 GFlops

Combined, this totals to 111 GFlops on 148 processors.

There is also a server machine, control.cascv.brown.edu. This is a single-processor PowerPC machine with 256 MB of memory. It manages the cluster and the batch jobs which are submitted to it.

Currently, 'control' also serves as the login/development machine (see logging in, below), providing a place to develop and compile software, and to submit jobs. Please note, however, that it is not meant for running production-level jobs or other CPU-intensive tasks.

In addition to using a standard ethernet network, the nodes are interconnected via 160MB/sec TrailBlazer-3 SP switches, allowing fast communications within Parallel programs.

We will soon be connecting the cluster to a high-capacity RAID storage system, which will serve files over the SP switches to the entire network (currently disk sharing is via ethernet).

Note for users: Currently (8/2002) all files on all disks are backed-up. This will be changing when the RAID system comes on-line.

Diagram of the components of the ccv SP system

A graphical depiction of the SP's layout: (the "class" is used in the command file). Click for a larger image.

Logging in via ssh

You need ssh, the 'Secure Shell', to connect to the SP. Unlike telnet or rsh, ssh provides encryption of the entire login session, so that neither passwords nor data are passed through the network as plain-text
If it is not installed on your system, contact your system administrator. More information, including downloads, is available at: All About SSH, The SSH Book, OpenSSH.org (UNIX), ssh.com (Windows, UNIX, free for non-commercial use).

An example of connecting to the SP using ssh:

% ssh control.ccv.brown.edu -l <your userid>
password: <your password> (change this when you first login)

Note also that ftp will not connect to the server. To transfer external files onto the SP, use sftp to connect to the server. sftp provides an ftp-like interface, but uses the encryption provided by ssh

If you wish to run X-windows programs from control, and are using OpenSSH, be sure to use the -X command-line option.

Compiling and running jobs

To run a program on the cluster:

  • Compile the program if necessary (with Fortran compilers or C/C++ compilers). See Third-Party Software for a list of ready-to-run programs supplied and maintained by CCV.
  • Create a command file which specifies how your program will be run The command file specifies which program to run, the input and output sources (if any), which group of machines should be used (the "class" in Figure 1), etc. For the definition of a command file, and examples, look here.
  • Make sure the LoadLeveler commands are in your path. For example, in csh, this is set path = ($path /usr/lpp/LoadL/so/bin).
  • Submit the job to the LoadLeveler, by running llsubmit on the server:

control> llsubmit command_file.cmd

  • If everything has gone well, your job is now in LoadLeveler's queue (prioritized list of user jobs), waiting for the resources specified by your command file to become available.

You may check the status of the cluster and your job using the various loadleveler commands. Some examples:
llq (shows all jobs, their status, and their job ID numbers)
llq -s 5611 (gives extended status for job number 5611)
llcancel 5611 (cancels the given job)
llstatus (gives status of each node in the cluster; 'Busy' means all CPUs are running, 'Run' means some but not all CPUs are running)

  • If the notification option is set to 'complete' or 'always' in the command file (see step two), you will receive an e-mail when the job is finished.

Job priorities and restrictions

Jobs will generally be run on a "first-come first-served" basis, except for the following cases:

  • If there are many jobs within the same 'group' running, the priority will be lowered.
  • If the 'wall_clock_limit' parameter is set for less than 24hrs and/or uses a small number of nodes, the job may be moved up in the queue if LoadLeveler determines it will not have an impact on other jobs (i.e. jobs waiting for a large set of processors).
  • '1n-pwr3' jobs are generally preferred to jobs of class '2n-pwr3' through '5n-pwr3'.

All jobs are limited to 24hrs, or to 12hrs for classes '2n-pwr3' through '5n-pwr3'. This is a hard limit: if a program is still executing after this time limit, it will be terminated.

Thus, if you are not sure how long your job will take, it would be good to either save data intermittently during the run, or to estimate the time by running smaller "benchmark" jobs.