Rackham

Rackham user guide

The Rackham cluster was introduced in February 2017. Rackham is a SNIC resource and is estimated to be in production until first of January 2023. The major features of Rackham and its storage system Crex is found below. For more technical data please see the end of this article.

Rackham was extended in April 2018.

• Rackham comprises 9720 cores in the form of 486 nodes with two 10-core Intel Xeon V4 CPU's each. There are "fat" nodes,4 with 1TB memory and 32 with 256 GB memory, with the rest having 128 GB.
• Rackham's storage system (named Crex) uses the Lustre file system and provides 6.6PB of storage.
• The interconnect is Infiniband FDR which supports a theoretical bandwidth of 56Gb/s and a latency of 0.7 microseconds [1].

Technical data

Compute and system node hardware

Each compute node consists of two (2) Intel Xeon E5 2630 v4 at 2.20 GHz/core (10 cores, 20 threads, 25 MB LLC, and a bandwidth of 68.3 GB/s) [2]. Each compute node has either 8x16384MB (128GB) or 16x16384MB (256GB) or 16x65536MB (1TB) of ECC 2400MHz DIMM DRAM memory. For local storage each compute node has a 2TB disk at 7200RPM .

Each system node also has two CPUs identical to a compute node. System nodes has 16x16384 (256GB) of ECC 2400MHz DIMM DRAM. For local storage each system node contains 4x600GB 10 000 RPM SAS disks for a total of 2.4 TB.

Crex

Rackham's storage system (named Crex) is a DDN (DataDirect Networks) EXAScaler filesystem based on the ES14KX platform. Crex uses 420 6TB NL-SAS drives and 24 300GB SAS drives for metadata storage. Total storage reserved for SNIC projects is 6.6PB. The filesystem is Lustre, a highly scalable filesystem common in HPC.

Network

The nodes and network uses Infiniband FDR with a maximum bandwidth of 56 Gb/s and a latency of 0.7 microseconds. The topology is fat tree with 14 leaf switches and 6 core switches. Each leaf switch connects 24 servers and reserves 12 ports for uplinks. The subscription rate is 2:1.

Operating system and software

All Rackham nodes uses CentOS 7. We provide most compilers and interpreters and provide requested software from our module system, where you will find build tools, bioinformatics tools and software such as MATLAB, GROMACS, VASP, OpenFOAM, COMSOL Multiphysics, RStudio, and more.

[1] http://www.mellanox.com/page/performance_infiniband

[2] https://ark.intel.com/sv/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz