A New Generation of String Simulations

New physics tells us that cosmic strings – filaments thinner than a hydrogen atom – formed in the early universe. Studying them is beneficial, for example, for predicting future cosmic events. This requires high-resolution simulations. One of the biggest problems with these simulations is their resolution and accuracy. But a team of Portuguese researchers from the University of Porto has set out to find the solution. The computational resources have already allowed them to create the largest cosmic strings simulations to date.

Challenge

Cosmic strings are filament-like objects, predicted by several theories of new physics, which possibly formed in the early Universe. String networks source backgrounds of radiation, possibly detectable by upcoming observational facilities. Understanding the evolution of such non-linear networks and their generated signals involves describing physics both on the scale of the string width (thinner than a Hydrogen atom), and on that of the size of the Universe. To make accurate predictions, one unavoidably needs computationally costly high-resolution simulations, which previously relied on CPUs.

Solution

Our solution was based on a trade-off: use accelerators and their inherently larger memory bandwidth, but deal with having less overall memory available for storing simulation variables. This meant the grid needed to be split into equally sized sub-domains across multiple accelerators, with the necessary boundary terms communicated between neighbouring sub-domains. In order to weakly scale across thousands of GPUs, we overlapped computation and communication. Compared with previous CPU-based simulations, we obtain a speed-up factor of roughly 30. To further complement the simulation’s scientific outputs, we also added in-situ visualization capabilities. At sufficiently large grid size, this reduces the amount of data output by several orders of magnitude: roughly four orders at 4096^3.

Business impact

Given that the current description of such signals relies on a combination of analytic approximations (computationally straightforward but missing some of the relevant physics) and simulations with insufficient resolution to accurately describe the evolution of a string network, we set out to improve the main computational bottleneck of string simulations. Being essentially finite-difference solvers, such simulations are often Input/Output bound (visualization and observables output) and memory bound (evolution). For the past two decades these bottlenecks have limited the accuracy and resolution of simulations, which had not matched the quality of observational data. Results from the scientific exploration of our newly developed codes, coupled with appropriate extreme computing resources, will contribute to directly computing the expected signals from a network of strings with as high an accuracy as possible. This is of utmost importance for comparison with data from several sources such as the Pulsar Timing Array (active), Laser Interferometer Space Antenna (under construction), and Einstein Telescope (planned).

Benefits

Obtain robust predictions for upcoming cosmological experiments
Ensure the resolution of simulations can match or exceed the quality of future data
Simulate increasingly complex and realistic models of string networks (i.e. superstrings)

The assistance of the Portuguese computing infrastructure

The team members are Portuguese and were affiliated with Portuguese institutions, but the resources necessary to address the challenge were beyond those currently existing in Portugal. Nevertheless, local CAUP resources (e.g., an NVIDIA Quadro P5000) were extensively used for code development and validation. Ultimately our production runs used 68 million core hours and 4096 NVIDIA P100 GPUs provided by the Swiss Supercomputing Centre through PRACE. This enabled the largest cosmic string simulations to date. One such simulation takes roughly 33 minutes of wall clock time and would be impossible to fit on a single desktop GPU, but if it did, it would take roughly 94 days to complete.