EXCELLERAT P2

The European Centre of Excellence (CoE) for Engineering Applications, EXCELLERAT, has entered its second phase (known as EXCELLERAT P2) and is continuing its mission of supporting key engineering industries in Europe. This CoE is a conglomerate of business partners and high-performance computing (HPC) centres promoting interaction between academic research and code development with industrial partners.

PDC and the Department of Engineering Mechanics at the KTH Royal Institute of Technology were partners in EXCELLERAT P1 (the first phase of the project) and continue their involvement in this project working in the field of turbulence simulations. During EXCELLERAT P1, the main focus was on the development of a robust adaptive mesh refinement (AMR) workflow and uncertainty quantification (UQ) tools in a computational fluid dynamics (CFD) solver known as Nek5000 (which was originally developed by the Argonne National Laboratory, US).

In the second phase of the project, PDC introduced a new CFD solver, Neko, which is entirely developed at PDC. Both solvers, Nek5000 and Neko, are based on a high-order spectral element method (SEM) and are capable of performing high-fidelity simulations of advection-diffusion problems. They share multiple features and Nek5000 can be seen as Neko’s predecessor. However, unlike Nek5000, Neko is a modern piece of software, which is written mostly in object-oriented Fortran 2008. It supports multiple hardware backends by defining a device abstraction layer and then writing hardware-specific kernels in CUDA, HIP or OpenCL. Neko was tested on multiple systems showing very good strong scaling properties, achieving 80% parallel efficiency when using up to 80% of the GPU partition of the LUMI system based in Finland, which is equivalent to 16,384 AMD MI250X Graphics Compute Dies (GCDs) where each MI250X GPU consists of two GCDs.

In EXCELLERAT P2, KTH continues its previous work on high-order hex-based meshing, AMR workflows, UQ techniques, in situ visualisation and improving the parallel performance of the solver, but this time in the context of Neko. Although this already covers multiple aspects of simulation workflows, there are new features being considered in the second phase. These are in situ data analysis using streaming algorithms and noise prediction for incompressible CFD solvers. The first one is motivated by the wide use of data analysis algorithms – such as proper orthogonal decomposition (POD) or dynamic mode decomposition (DMD) – which rely on large amounts of data being saved to the disk and then analysis being performed at the post-processing step. However, this approach will not be feasible for exascale modelling due to the massive volumes of data and the cost of the I/O operations. That is why KTH is going to implement proper streaming algorithms for POD and DMD in Neko to avoid storing data on the disks and to extract the expected results on the fly during the simulation. In addition, the noise prediction will make it possible to utilise high-quality data and obtain good approximations for the far-field acoustics.

The use case being studied during the second phase of EXCELLERAT is the continuation of one of the use cases from the first phase: “High-fidelity simulation of rotating parts”. The code development enabled us to execute a simulation of a “toy” rotor with the AMR version of Nek5000 at the end of EXCELLERAT P1 (see figure above). Now we are going to extend it to the industrially relevant “Iowa” rotor, which is a double blade, twisted drone rotor with an Eppler E64 wing profile. We are going to demonstrate all of the EXCELLERAT P2 development with this particular use case, which will include performing AMR, UQ, acoustics and POD/DMD in situ.