Science Highlight: Direct Data Streaming Cuts Nuclear Physics Processing Time from Hours to Minutes

Facility for Rare Isotope Beams Data Acquisition Scientist Aaron Chester (center) demonstrates real-time machine learning workflows at FRIB using EJFAT and ESnet to SC25 attendees, as with ESnet EJFAT prototype codeveloper Yatish Kumar (left) looks on

The Science

Atomic nuclei in excited states emit radiation to achieve a more stable configuration. Studying the radiation emitted by long-lived excited states, or isomers, yields important insight into the microscopic structure of the nucleus. Such experiments are designed to observe these kinds of events using detectors where these decays would leave a very clear signature. Researchers at the Facility for Rare Isotope Beams (FRIB) at Michigan State University directly observed isomeric decays in real time following the implantation of radioactive ions in an inorganic scintillation detector. The subsequent radioactive decay products were measured using the FRIB Decay Station Initiator (FDSi), a suite of ancillary detection systems for neutron and gamma-ray spectroscopy. 

The Impact

In 2025, an FRIB research team developed and demonstrated an automated high-throughput pipeline, FRIBDAQ (FRIB Data Acquisition), to enable the quick processing of the large raw data sets from FRIB experiments. The researchers had been relying on local compute clusters for computationally intensive data processing, which were in high demand, and then sending the processed files through Globus (via ESnet) for further analysis at High Performance Computing (HPC) facilities. By incorporating the ESnet-JLab FPGA Accelerated Transport (EJFAT) load balancer into the FRIBDAQ software’s workflow, the researchers were able to stream their raw data into the Oak Ridge Leadership Computing Facility in Tennessee at 5 Gbps for processing, which then sent it back to FRIB for analysis by the researcher. They then switched to streaming to the Perlmutter supercomputer at the National Energy Research Scientific Computing Center (NERSC) in Berkeley, California. Fifteen hours of experiment run time produced 615 gigabytes (GBs) of data, which required only 20 minutes and 8 Perlmutter nodes to process using a machine-learning inference model, for an average rate of 525 megabytes per second. By being able to go directly to remote HPC resources and cut their processing time to minutes, the researchers were able to analyze pulse shapes and identify particles in real time for their experiments. 

Screenshot from a video showing plots of analyzed data returned from Perlmutter, along with logs showing 500,000 events’ worth of data from the FRIB DAQ streaming into the EJFAT load balancer at 6 Gbps, with no events lost in the queue or reassembly, and visualization. Left plot: The horizontal bands are the experimental signatures of target isomeric decays. Center plot: Zoomed-in lower-left corner shows another type of isomer signature. Right plot: Particle identification in progress; each “blob” represents a particular isotope delivered to the experiment by the accelerator.

Additional Details 

The EJFAT prototype is designed to seamlessly integrate edge and cluster computing in order to allow data from multiple types of scientific instruments to be streamed and processed in near real time by multiple HPC facilities — and, if needed, to redirect those data streams dynamically. The FRIB researchers appreciated that EJFAT offers low latency as well as flexibility in terms of the data they can send, as multiple input formats are supported. They plan to explore using EJFAT to integrate their workflows with the FABRIC project and other infrastructure that can support outside users, as well as explore data transfers to offsite storage, such as the High Performance Data Facility.

General FRIBDAQ Pipeline Architecture and Dataflow

How EJFAT load balances and sorts events for processing as part of a science workflow.

ESnet Contacts

• Chin Guok, chin@es.net
• Yatish Kumar, yatish@es.net 

Collaborating Institutions

Facility for Rare Isotope Beams, Michigan State University; Department of Chemistry, Michigan State University; Nuclear Science Division, Lawrence Berkeley National Laboratory

Publications

• Chester, A. “Real-Time Machine Learning Workflows at FRIB Using EJFAT and ESnet,” DOE Booth Demonstration, The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC2025). 
• Chester, A.; Cerizza, G.; Crawford, H.; Fox, R.; Liddick, S.; Lubna, R., et al. (2025). High-Throughput Data Processing at FRIB Using ESnet. IEEE Transactions on Nuclear Science, 72(3), 506-509. http://dx.doi.org/10.1109/tns.2024.3483555 
• Sadeghi, B., Scriven, D., Chester, A., Fox, R., Liddick, S., Cerizza, G. (2026) A machine learning framework for accurate and robust analysis of radiation detector pulses. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1082(2), 170971. https://doi.org/10.1016/j.nima.2025.170971