CCV To Host HPC Research Environment

Core System Description:

The nucleus of the new system is a 166-node Linux cluster purchased from IBM. Each node has two Intel Xeon 5540 (2.53 GHz) quad-core Nehalem processors, 24 Gigabytes of DDR-3 memory (1333 GHz) and a 40 GHz Quad-Data-Rate (QDR) Infiniband messaging interface. The nodes are diskless, with I/O provided by the IBM GPFS parallel filesystem accessed through the Infiniband interface. The dual-processor compute nodes are packaged in IBM's iDataPlex system, which is designed for high reliability and reduced power consumption.

In addition to the dual-processor Nehalem nodes, there are 7 quad-processor AMD Opteron nodes with Shanghai-series processors clocked at 2.6 GHz. Six of these nodes have 64 Gigabytes of memory. The seventh has 128 Gigabytes. All have QDR Infiniband interfaces.

Two dual-processor management nodes provide network provisioning, monitoring and other system management utility. Two login nodes provide access to the cluster for application development, debugging and job submission.

Access to the GPFS filesystem is provided by a 12-node service cluster and associated hardware RAID disk storage subsystems totaling roughly 390 Terabytes (30 Terabytes of SAS and 360 Terabytes of SATA disks), and with aggregate performance of more than 10 Gigabytes/s. Each RAID subsystem is connected to two of the service nodes in order to allow configuration for fail-over redundancy as supported in GPFS. The GPFS system is integrated with a Tivoli Storage Manager (TSM) 2 Petabyte tape archival and backup system. Policy-based Hierarchical Storage Management (HSM) facilities control automatic data migration among the SAS, SATA and tape storage pools. In addition, GPFS supports snapshot filesystem imaging to allow easy retrieval of accidentally deleted files.

The compute nodes access GPFS directly via their Infiniband interfaces. Users can also access the filesystem from other campus systems via the CIFS fileservice and through the Web via WebDAV. Each of the service nodes has both QDR Infiniband and 10-Gigabit Ethernet interfaces, and a 10-Gigabit Ethernet switch connects them to the Brown campus networking through a 10-Gigabit Ethernet backbone connection.

The compute and GPFS cluster both run the CentOS build of the RedHat Enterprise Linux (RHEL) operating system, and are provisioned with the xCAT open source cluster management software. Workload management is performed using the Moab Cluster Suite, a commercially supported system for managing queuing and scheduling systems and for user job submission.

Both the computational and disk storage clusters can easily be expanded or "scaled out" with additional nodes.