Skip to main content
PDC Center for High Performance Computing

About the Dardel HPC system

Dardel is an HPE Cray EX supercomputer which features a CPU partition (suitable for a broad range of computational applications) and a GPU partition (intended for the most computationally demanding applications). There is also an associated storage system. This NAISS flagship system was installed at PDC between 2021 and 2023.

Background to Dardel

In 2019, the Swedish National Infrastructure for Computing (SNIC) decided to invest in a new high-performance computing (HPC) system for Swedish researchers, which would be hosted at PDC. The new system was installed at PDC between 2021 and 2023. It is an HPE Cray EX supercomputer, which is called Dardel. After the initial decision, SNIC also funded several expansions to the Dardel system to better serve Swedish research needs over a longer time frame.

At the start of 2023, the National Academic Infrastructure for Supercomputing in Sweden ( NAISS ) took over from SNIC, and PDC became a key associate of NAISS. NAISS is now responsible for the operation of Dardel.

Dardel installation

The Dardel system consists of a CPU partition, a graphics processing unit (GPU) partition and storage which are housed in three rows of cabinets.

  • The row shown on the left in the image above was installed in 2021 and contains most of the CPU partition of the system plus a liquid-based cooling distribution unit. This row is now the middle row of the whole system.
  • The front row (shown on the right in the image above) was installed in 2023 and contains the graphics processing unit (GPU) partition of the system plus additional CPUs and a liquid-based cooling distribution unit.
  • There are three additional cabinets for storage and management located behind these two rows of compute cabinets. These cabinets are air-cooled and contain the parallel (Lustre-based) file storage system. This rear row was installed along with the back row of the computational part of the system in 2021.

Some photos of the process of installing Dardel can be found here .

Performance of Dardel partitions

Current LINPACK (Rmax) performance of Dardel partitions:

The performance of Dardel far exceeds a typical personal computer (PC) or laptop. The table gives some other comparisons between the Dardel CPU partition and a typical personal computer. (If you are not familiar with terms like nodes and cores, you can read the explanation about the parts of a supercomputer .)

Dardel vs a typical PC
  Dardel CPU partition PC
Nodes 1278 1
Cores 163,584 16
Capacity 5055 TFLOPS 0.7 TFLOPS
Memory 497,664 GB 16 GB
Energy consumption 700,000 W

100 W

Note: The capacities listed in this table are based on system-level LINPACK benchmark tests, which give a measure of computing power in terms of floating point operations per second (FLOPS) for solving linear equations.

Dardel system design

 If you are interested in seeing what is inside the HPE Cray EX supercomputer systems in general, you can explore here .

Dardel’s CPU partition

The CPU partition consists of 1278 compute nodes. Each node has two AMD EPYC™ Zen2 2.25 GHz 64-core processors, which means that each compute node has a total of 128 physical CPU cores. The nodes in the CPU partition come in a variety of memory sizes as listed below.

  • 700 × 256 GB (NAISS thin nodes)
  • 268 × 512 GB (NAISS large nodes)
  • 8 × 1024 GB (NAISS huge nodes)
  • 18 × 2048 GB (NAISS giant nodes)
  • 36 × 256 GB (KTH industry/business research nodes)
  • 248 × 512 GB (KTH industry/business research nodes)

This gives a total of 1278 nodes. As each node has 128 physical cores, that means the CPU partition overall has 1278 X 128 = 163,584 physical cores.

These physical CPU cores can execute instructions independently of the other physical cores in the system. Each of the physical CPU cores has hardware support (known as hyperthreading) for two threads - these are software abstractions that can run instructions independently of each other on a single physical core. Each thread can be regarded as being executed on a single non-physical core, usually known as a virtual or logical core.

The two hardware-supported threads that can be used on each physical CPU core mean that each compute node (with its 128 cores) can support 256 hardware threads (virtual cores). Users should note that some Dardel system commands display the number of hardware-supported threads rather than, for example, the number of physical cores. (It is, of course, possible for software to create and run far more than two software threads on a CPU core, however there is only physical hardware support for two such threads.)

Dardel’s GPU partition

The GPU partition comprises 62 GPU nodes, each of which is configured with

  • one AMD EPYC™ processor with 64 cores (which is a special version of a 64-core AMD CPU that is known as the 7A53 (Trento) and only available from HPE),
  • 512 GB of shared fast HBM2E memory (which is cache-coherent and hence simplifies programming), and
  • four AMD Instinct™ MI250X GPU chips, each with two GPU devices (known as graphics compute dies or GCDs), that have an impressive performance of up to 95.7 TFLOPS in double precision when using special matrix operations and are connected by AMD Infinity Fabric® links as shown in the figure below.
Dardel GPU node architecture

Dardel’s interconnect

  • HPE Slingshot (ethernet-based) using Dragonfly topology
  • 200 Gb/s since March 2023
High-speed network Slingshot topology as used in the Dardel system

Usage of Dardel