Toil and trouble

Bubbles could block a promising technology that would separate carbon dioxide from industrial emissions, capturing the greenhouse gas before it contributes to climate change.

A team of researchers with backing from the Department of Energy’s Exascale Computing Project (ECP) is out to burst the barrier, using a code that captures the floating blisters and provides insights to deter them.

Chemical looping reactors (CLRs) combine fuels such as methane with oxygen from metal oxide particles before combustion. The reaction produces water vapor and carbon dioxide, which can easily be separated to create a pure CO₂ stream for sequestration or industrial use. Standard post-combustion separation must pull carbon dioxide from a multigas mixture.

The reduced metal particles then move to an air reactor where they’re oxidized, producing heat that can be used to produce electricity. The replenished carrier particles return to the fuel reactor, completing the loop.

But the fuel reactor’s complex flow behavior typically leads to bubbling, degrading performance and letting unspent fuel contaminate the exhaust. CLRs’ multiphase processes – involving solids, gases and liquids – also are difficult to scale up for industrial use.

“We want the system to perform well, and we can work toward that goal by maximizing contact between the fuel and solid oxygen-carrier,” says Jordan Musser, a physical research scientist at the National Energy Technology Laboratory (NETL). “For this, we need to prevent the formation of large bubbles that inhibit gas-solids mixing. Another concern is understanding how to scale-up units that we design in laboratories, enabling economies of scale to make these reactors beneficial to society.”

Musser is principal investigator for the ECP’s MFIX-Exa, Performance Prediction for Multiphase Energy Conversion Device project. The initiative, which includes collaborators from Lawrence Berkeley National Laboratory and the University of Colorado Boulder, pairs ECP-developed visualization and analytics capabilities with MFIX-Exa, a particle-based multiphase computational fluid dynamics (CFD) simulation capable of running at exascale – a quintillion calculations per second. Oak Ridge Leadership Computing Facility resources contributed to the work.

“MFIX-Exa is an open-source code specifically developed to model reacting gas-solids flows in complex reactor geometries,” Musser says.

The team played a video, produced by Los Alamos National Laboratory (LANL) intern Alexandra Stewart, at last fall’s SC22 conference in Dallas, demonstrating its progress in matching MFIX-Exa with visualization tools. It showed the project can effectively model and intuitively represent bubble formation, an important milestone in optimizing reactor design.

The quest for a model that accurately simulates the CLR environment starts with understanding how two primary components – the fuel reactor and the air reactor – behave. Tracking both produces a computational load-balancing problem that can strangle the efficiency of modeling software.

 Creating a GPU-capable code to simulate reacting gas-solids flows was tough.

MFIX was created in the 1980s to run on standard computer processors. It’s precise but slow. The team rewrote MFIX’s entire particle solver and modernized legacy Fortran code, making it a modern, massively parallel program. MFIX-Exa can run on graphics processing units (GPUs), accelerators that promise to boost its speed for exascale machines. It uses multiple strategies to distribute particle calculations and fluid calculations across GPUs for efficient and constant hardware use.

But creating a GPU-capable code to simulate reacting gas-solids flows was tough. The team tapped the open-source AMReX software framework, designed to enable massively parallel adaptive mesh refinement (AMR) exascale applications. AMR focuses computation on the most important parts of a simulation, capturing detail there while conserving computer power in other parts of the domain. AMReX, developed by teams from Berkeley Lab, the National Renewable Energy Laboratory and Argonne National Laboratory, supports both standard processors and GPUs plus multiple programming models and languages.

Musser and colleagues used MFIX-Exa to model the fuel reactor in NETL’s 50-kW CLR, interactively detecting bubbles and tracking their features, such as volume and velocity. To develop an application-specific scientific workflow, they collaborated with the ALPINE and Cinema ECP software teams.

The ALPINE project, led by LANL ‘s James Ahrens, develops in situ visualization infrastructures and algorithms – ones that run concurrently with MFIX-Exa to detect bubbles and other features. The MFIX-Exa workflow also leverages Catalyst, a library to access and run an in situ statistical feature-detection algorithm. It’s part of Kitware Inc.’s visualization tool suite.

Cinema, a novel visualization tool, processes post-simulation data, interactively exploring and visually analyzing large-scale scientific data. Ahrens developed Cinema, now led by Los Alamos’s David Rogers.

Catalyst’s statistical feature detection algorithm down-samples MFIX-Exa simulation data in situ by more than 300-fold, into a set of functions associating a single number to every point in a space. The algorithm, developed by former LANL computer scientist Soumya Dutta, determines how similar a small section of the simulation is to a target feature – a bubble or void with low particle density. The bubble similarity field calculates the probability that a bubble exists in a particular simulation region. The algorithm also can output other useful fields, such as the particle velocity.

Next, the workflow uses the simulation output fields to create images representing bubble evolution over time. These are linked to a Cinema database and visualized though a specialized viewer, letting domain experts perform post-simulation analyses that focus on specific parameters and then visualize bubble dynamics along those dimensions.

“If you know what visualizations you want while the simulation is running, you could just push out all those images and then you don’t have to render them later,” says Terece Turton, a LANL staff scientist and the ALPINE integration lead. “If you’re looking at a simulation with billions of particles, to save all that to disc and post-process it becomes intractable to some degree. Also, much of the data isn’t terribly useful.”

Cinema’s real-time and post-simulation visualizations let CLR researchers observe such phenomena as fluid velocity changes as bubbles rise, move laterally, merge and split, forming large low-density regions in the reactor’s fluidized beds, possibly decreasing its efficiency. With output from MFIX-Exa, the down-sampled bubble features and Cinema, they can eventually suggest design changes for scaled-up reactor models and assess how they affect bubble formation or other reactor functions.

“The instant visualization is worth every bit of time that we put into it,” Musser says. “There are a lot of diagnostics we can do on our code just by looking at flow patterns. It takes a lot of work off of our team. It’s phenomenal.”