Aiming exascale at black holes

In 1783, John Michell worked as a rector in northern England, but his scientific work proposed that the mass of a star could reach a point where its gravity prevented the escape of most anything, even light. The same prediction emerged from Albert Einstein’s theory of general relativity. Finally, in 1968, physicist John Wheeler gave such phenomena a name: black holes.

“There was a long debate about whether black holes would actually exist,” says James Stone, professor of astrophysics at the Institute for Advanced Study, home to Einstein from 1933 until his death in 1955. The problem is that an equation for a black hole’s mass goes to infinity in a mathematical representation of spacetime, which models three-dimensional space plus time. That’s called a singularity, and “nature doesn’t like singularities,” Stone notes. “Remarkably, astronomical observations have provided unambiguous proof that black holes do exist.”

And there are lots of them, maybe as many as 40 quintillion, or 40 thousand million billion. “There are both stellar-mass black holes, which are a few times the mass of the sun, and also supermassive black holes, which are millions to billions of times the mass of the sun,” Stone says. “And they are important because matter that falls into black holes doesn’t just disappear quietly.”

Instead, matter turns into plasma, or ionized gas, as it rotates toward a black hole. The ionized particles in the plasma “get caught in the gravitational field of a black hole, and as they are pulled in they release energy,” he says. That process is called accretion, and scientists think the energy released by accretion powers many processes on scales up to the entire galaxy hosting the black hole.

As an example, Stone points to so-called close binary systems. Here, a stellar-mass black hole and a normal star are in a tightly bound orbit. “The two are so close together, that the atmosphere of the normal star is being stripped off and spiraling into the black hole. As it falls in, it emits an enormous amount of radiation.”

To better understand matter in-flow and energy release, Stone turns to simulations. He’s particularly interested in modeling the magnetism that’s crucial for the accretion of plasma by black holes. “When there’s ionized gas,” he says, “you get magnetic fields, and magnetic fields feed back on the motion of charged particles.”

To explore this process, Stone uses what’s called general relativistic radiation magnetohydrodynamics (MHD). To break down this mathematical model, it’s easier to work through it backward. It simulates the plasma dynamics with MDH and the radiation energy from the plasma. Last, being near a black hole requires a relativistic approach because a black hole’s mass puts a substantial curve in spacetime. The gravitational force of any mass — even a person — bends spacetime, but a black hole bends it to infinity.

Plasma could get stuck in orbit around a black hole, just like Earth and the sun. If that plasma loses angular momentum, though, it starts to spiral into the black hole. “We know magnetic fields are absolutely crucial” to this process, Stone says, but the dynamics between magnetic fields and plasma accretion is a mystery — and one Stone wants to reveal.

‘The only way forward to solve these equations is through computation.’

The equations behind general relativistic radiation MHD, though, are not easy to solve. “They’re so complicated that analytic solutions — finding solutions with pencil and paper — is probably impossible,” Stone says. “So really, the only way forward to solve these equations is through computation.” Plus, the plasma flow near black holes gets turbulent, which adds another layer of complexity to the calculations.

With a 2024 INCITE (Innovative and Novel Computational Impact on Theory and Experiment) award, Stone and his colleagues are running models on Department of Energy high-performance computers (HPCs), Argonne Leadership Computing Facility’s Polaris and Oak Ridge Leadership Computing Facility’s Frontier, which has achieved exascale performance capable of a million trillion calculations per second.

Stone is also using the Athena++ code, which has been developed in his lab over the past couple of decades. “It’s the implementation of numerical algorithms for solving the MHD equations,” he says. To make this software capable of running on machines like Frontier, Stone’s team used the Kokkos C++ Performance Portability EcoSystem, which was developed as part of the DOE’s Exascale Computing Project. Stone calls Kokkos “a game changer that allows you to write your code in C++, and then it gets translated into any programming model for any kind of machine,” from a laptop to an exascale platform.

By combining Athena++ with various HPCs, Stone says, “our main goal is to understand how radiation changes black-hole accretion.” Although data exists for slowly accreting black holes, which produce little radiation, the ones that have been observed astronomically are very bright — the brightest things in a universe. That brightness means that those black holes are “accreting very, very rapidly, and they’re releasing enormous amounts of energy. For those systems, we need to incorporate the radiation to understand how it changes the plasma flow.”

The code created by Stone’s team to investigate black-hole accretion can be applied to other astrophysical phenomena. Stone mentions that he “can use the same Athena++ code for MHD simulations to follow the motion of cosmic rays,” high-energy particles also produced by black holes. Scientists study cosmic rays for their role in various galactic phenomena, including star formation.

With another 2024 INCITE award, which also includes time at Argonne and Oak Ridge, Stone says, “we want to see how those cosmic rays propagate through space — how the particles move in this horribly complicated spaghetti of tangled magnetic fields produced by turbulence.”

For black-hole accretion, cosmic rays and more, Stone is quick to note his appreciation of the ECP. When asked if Einstein might have shown equal appreciation of making use of exascale computers, Stone replies: “I think he would have embraced it. He would have understood that it’s just a tool to solve really complicated mathematical systems.”

Often, the predictions from those systems seem like sci-fi to most people — sometimes even to astrophysicists — particularly for black holes. “It’s even bizarre to people like me,” Stone says. “It’s amazing what nature does.”