ESnet Completes Requirements Review for High Energy Physics

Trifid and Lagoon Nebulae Finder Chart (annotated; click image to enlarge). Credit: NSF–DOE Vera C. Rubin Observatory 

Key Takeaways:

• The Energy Sciences Network (ESnet) is the U.S. Department of Energy’s “data circulatory system,” transferring DOE-funded scientific research data around the world. To keep up with the needs of technology advances and ever-evolving experiments, ESnet holds regular “Requirements Reviews” with science stakeholders to learn directly from users and make actionable recommendations.
• ESnet is a key part of the Integrated Research Infrastructure program (IRI), an emerging initiative by the U.S. Department of Energy. The findings of Requirements Reviews inform IRI efforts in accelerating scientific discovery, with seamless integration of research instruments, large data sets, modeling and simulation, and high-performance computing resources across an API-driven, high-speed data, and networking substrate. 
• The most recent Requirements Review, for the High Energy Physics (HEP) program, involved 127 authors, editors, and participants. It resulted in a 400-page report featuring 14 research case studies that will help inform ESnet’s strategic planning as well as its current and future collaborations with the HEP community. 

How do you transfer 600 terabytes of data, or 1,172 smart phones’ worth, — across 800 miles — at incredibly high speed from a mile underground in South Dakota? This is just one of many logistical issues that Deep Underground Neutrino Experiment (DUNE) researchers need to solve in order to understand the role neutrino particles play in the universe.

This mission-critical data challenge, along with those posed by thousands of other U.S. Department of Energy (DOE)-funded research activities, is supported by the Energy Sciences Network (ESnet), the DOE user facility that serves as the “data circulatory system” interconnecting tens of thousands of scientific researchers, all 17 national laboratories, all 28 DOE user facilities, and other sites in the U.S. and Europe. ESnet’s network offers speeds of up to 1.2 terabits per second (Tbps) and moves massive quantities of data around the world — a total of 1.8 exabytes in 2024, the equivalent of streaming high-definition movies 24/7 for the next 29,354 years!

DUNE is just one of 14 projects and facilities profiled in the just-released ESnet High Energy Physics 2024 Requirements Review report. The DOE High Energy Physics (HEP) program’s mission is to research matter and energy to gain a better understanding of how the universe works. HEP funds or cofunds, along with the National Science Foundation (NSF) and/or the National Aeronautics and Space Administration, dozens of projects and facilities within six subprograms. ESnet provides the high-performance networking infrastructure that keeps the data flowing for all of the HEP science programs. 

ESnet Requirements Reviews: Generating Feedback and Actions

Categories of findings and actions for the ESnet requirements review

ESnet conducts periodic “requirements reviews” of all six DOE Office of Science programs that ESnet supports: HEP, Advanced Scientific Computing Research (ASCR), Basic Energy Sciences, Biological and Environmental Research, Fusion Energy Sciences, and Nuclear Physics. In a Requirements Review, program managers, researchers, site coordinators and IT staff from each laboratory/facility, and ESnet’s Science Engagement team collaboratively evaluate the current and projected status of experiments, data needs, changing technologies, and the networking needs of DOE/SC mission science to ensure that the network is both technologically up to date and ready to grow with evolving research needs and advances (see diagram, left). Just like the reviews themselves, their implementation and history is iterative. 

“I remember calling the researchers and facilities we were working with and asking them what they needed to do with the network,” recalls Eli Dart, a Network Engineer with ESnet’s Science Engagement group, who led the requirements program from 2007 to 2015. “With that set of science drivers, we were able to identify a common set of services, scaled by bandwidth, that would meet the needs of the science community. This gave us the science mission drivers for the design of the new ESnet network, which would eventually become ESnet4.” (The network is now on its sixth generation, ESnet6). 

This initial exercise in 2006 led to the DOE review committee’s recommendations that helped formalize ESnet’s science requirements gathering efforts. Since then, over 25 Requirements Reviews have been held and have captured the distinct data and network needs of the six Office of Science programs. The resulting reports have provided valuable insights not just to ESnet implementation teams, but also to the Office of Science program managers and other federal research efforts overall. 

“ESnet’s Requirements Review program is instrumental to proactive planning, building, and creating innovative solutions in the network to meet the evolving needs of science workflows for the researchers and organizations using our services,” explains Inder Monga, ESnet’s Executive Director. “The comprehensive case studies encourage alignment and understanding among the scientific collaborators, the various engineers – from lab to computing centers – and DOE program managers. This process enables us to better meet our mission goals of accelerating scientific discovery without being limited by the constraints of geography.”

Integrating the Requirements Reviews into DOE’s Research Infrastructure

Modern science grows more data intensive every day, with ever more sophisticated scientific instruments producing larger and richer data sets, and increasing High Performance Computing (HPC) capabilities for modeling and simulation, data analysis, digital twins, and artificial intelligence and machine learning (AI/ML). ESnet’s Requirements Reviews team has also recently added questions about how research activities align with ASCR’s Integrated Research Infrastructure (IRI) initiative, which seeks “to seamlessly link DOE's experimental and observational scientific user facilities, data assets, and advanced computing resources so that researchers can combine these tools in novel ways that radically accelerate discovery.” As part of its own requirements review process, described in the Architecture Blueprint Activity (ABA) report, the IRI project identified three key “science patterns” that such coordinated efforts must support:

• Time-Sensitive Patterns, which have urgency, requiring real-time or end-to-end performance with high reliability, e.g., for timely decision-making, experiment steering, and virtual proximity.
• Data Integration–Intensive Patterns, which require combining and analyzing data from multiple sources, e.g., sites, experiments, and/or computational runs.
• Long-Term Campaign Patterns, which require sustained access to resources over a long period to accomplish a well-defined objective.

ESnet’s Requirements Review reports have been essential to developing the IRI strategy and identifying IRI pathfinder projects. Dart and Jason Zurawski from ESnet's Science Engagement team completed a companion report to the IRI ABA, ESnet Requirements Review Program Through the IRI Lens, in which they reviewed 74 case studies from 2019-2023 ESnet Requirements Reviews across the Office of Science programs. This meta-analysis showed that the majority aligned with one or more IRI patterns. 

The Requirements Review process now includes an “IRI Readiness Assessment” profile to determine with which IRI patterns a project aligns, as well as action items for further integration into the IRI workflow infrastructure. Because Requirements Reviews evaluate diverse research projects funded through multiple DOE programs, the ESnet team will be in a unique position to identify potential solutions and problem-solving collaborations among programs that wouldn’t ordinarily be interacting with one another. 

Facilitating the HEP 2024 Requirements Review 

Requirements Reviews are time- and labor-intensive, but the effort is worth it. “The process gives us useful and valuable intelligence, and allows us to scope sizing of the network and build services,” explains Zurawski, who has led the process since 2016. “But we’ve also tried to emphasize with researchers that, beyond the deep dives into the case studies, this review mechanism is also a way to interact with your scientific community. We’ve heard from some attendees that the biggest benefit is the interactions with people from their own labs, because the reviews give them an opportunity to network outside of their working environments, and discuss the process of their scientific exploration with like-minded colleagues. There are so many soft benefits that you get by starting a process like this.”

The most recent HEP review began in June 2024, with DOE program officers and Science Engagement staff determining which research projects and facilities should be reviewed. Zurawski mapped out the timeline and action items for the case studies, introductory webinar, and then requested researcher questionnaires. Researchers were also offered the opportunity to have a conference call with the ESnet team in addition to, or in lieu of, completing the questionnaire — a recent shift based on participant feedback.

Prior to the multi-day, in-person review (held September 2024) the ESnet team worked closely with the researchers to develop draft case studies for discussion and review. 

This nearly year-long process, involving 127 authors, editors, and participants, has culminated in this 400-page report. Its list of findings and actions to bring up with the larger ESnet team for investigation, discussion, and/or completion include:

• The need for collaborators across facilities and organizations to have equitable access to data, whether on-site or accessing remotely.
• Scientific data sets continue to grow in size while also needing very fast transfers (up to 600 TB per 100 seconds). 
• The need to store and maintain long-term access to large datasets for all collaborators.
• Prioritization of network traffic requires strategizing of how different sites are identified on the network.
• Some researchers are exploring whether to integrate commercial services into reliable network workflows.
• Design network capacity in response to big data flows, to alleviate time and effort burdens on researchers.
• Assess network resilience to quickly respond to spontaneous events, outages, and natural disasters.

The laboratories and user facilities profiled were National Energy Research Scientific Computing Center, Argonne Leadership Computing Facility, Oak Ridge Leadership Computing Facility, Fermi National Accelerator Laboratory, Brookhaven National Laboratory, SLAC National Accelerator Laboratory, and Argonne National Laboratory.

This review also includes case studies from 14 HEP user facilities, experiments, and joint collaborative efforts. (See graphic, left, and read synopses of the Vera C. Rubin Observatory and Deep Underground Neutrino Experiment (DUNE) case studies from the review.)

ESnet project leads will now become the primary drivers of the action items identified in the HEP report, while the Science Engagement team transitions to a supportive role and begins the next Requirements Review: Fusion Energy Sciences is under way. While each program is reviewed in detail roughly every three years, brief check-ins are held in between reviews. The Science Engagement team is also always available for consultation should a need arise.  

Tailoring Network Solutions 

Science workflows can be unique and bespoke, with many factors determining how a researcher on a project generates, uses, transfers, and stores data — the software and computers they use, the equipment and machinery, the distance and terrain the data needs to travel, and the required turnaround time. (See the HEP report’s case study on Vera Rubin.) The Requirements Reviews are a key tool to uncovering all of the variables and launching ESnet’s “co-design” process, often a multi-year, multi-organization debugging endeavor to unravel end-to-end architecture issues. ESnet engineers are viewed as trusted partners who look forward to utilizing their subject-matter expertise in collaborating to solve each system's distinct equation to extract the most performance from the infrastructure, which allows researchers to prioritize their efforts on scientific discovery. 

Explains Shawn McKee, a University of Michigan research scientist who works on CERN’s ATLAS experiment and participated in the HEP review: “Networking is central to our work. We are trying to prioritize and decrease our time-to-science; how long it takes to turn our raw data into scientific results. The engagement between ESnet and the science communities is a unique opportunity to better understand and optimize the current ways we're using our infrastructure and how we could be using the network in the future. While the review is a lot of work, it is well-spent effort, because it ensures that what ESnet is delivering is usable and relevant for all the science domains that participate in these engagements.”