About the IBM iDataPlex HPC System at MPCDF

In October 2013, the main part of the iDataPlex HPC system HYDRA was installed at the MPCDF with Intel Ivy Bridge processors (~ 3500 nodes with 20 cores @ 2.8 GHz each), in addition to the Intel Sandy Bridge-EP processors (610 nodes with 16 cores @ 2.6 GHz each) that are available since Sept 2012. 350 of the Ivy Bridge nodes are equipped with accelerator cards (338 nodes with 2 NVIDIA K20X GPGPUs each, 12 nodes with 2 Intel Xeon Phi cards each).

Most of the compute nodes have a main memory of 64 GB, 2 x 100 of the Ivy Bridge nodes and 20 of the Sandy Bridge nodes have a main memory of 128 GB.

In total there are ~ 83.000 cores with a main memory of 280 TB and a peak performance of about 1.7 PetaFlop/s. The accelerator part of the HPC cluster has a peak performance of about 1 PetaFlop/s.

In addition to the compute nodes there are 8 login nodes and 26 I/O nodes that serve the 5 PetaByte of disk storage.

The common interconnect is a fast InfiniBand FDR14 network.

The compute nodes are bundled into 5 domains, three domains (including the old Sandy Bridge processor domain) with 628 nodes each, one big domain with more than 1800 nodes, and one domain consisting of the 350 nodes with accelerator cards.
Within one domain, the InfiniBand network topology is a 'fat tree' topology for high efficient communication. The InfiniBand connection between the domains is much weaker, so batch jobs are restricted to a single domain.

Addendum: After 5 years of operation, the 628 Sandy Bridge nodes of the Hydra cluster were decommissioned in September 2017. Another part of the Hydra cluster (628 Ivy Bridge nodes) was taken out of operation in November 2017. The Hydra cluster now has 61.240 cores with a total of 204 TB of main memory.

P6 System at RZG

Document Actions