ESnet Extends 100G Connectivity Across Atlantic
New capabilities improve network speeds between U.S. research sites, European facilities
Jon Bashor, [email protected], 510-486-5849
BERKELEY, Calif.—The Department of Energy’s (DOE’s) Energy Sciences Network, or ESnet, is deploying four new high-speed transatlantic links, giving researchers at America’s national laboratories and universities ultra-fast access to scientific data from the Large Hadron Collider (LHC) and other research sites in Europe.
ESnet’s transatlantic extension will deliver a total capacity of 340 gigabits-per-second (Gbps), and serve dozens of scientific collaborations. To maximize the resiliency of the new infrastructure, ESnet equipment in Europe will be interconnected by dedicated 100 Gbps links from the pan-European networking organization GÉANT.
(See "The Technical Challenges of Transatlantic Connectivity" below.)
Funded by the DOE’s Office of Science and managed by Lawrence Berkeley National Laboratory, ESnet provides advanced networking capabilities and tools to support U.S. national laboratories, experimental facilities and supercomputing centers.
Among the first to benefit from the network extension will be U.S. high energy physicists conducting research at the Large Hadron Collider (LHC), the world’s most powerful particle collider, located near Geneva, Switzerland. DOE’s Brookhaven National Laboratory and Fermi National Accelerator Laboratory—major U.S. computing centers for the LHC’s ATLAS and CMS experiments, respectively—will make use of the links as soon as they are tested and commissioned.
“Particle physicists have been pushing the boundaries of networking technology for decades, and they will make use of our new extension almost immediately,” said ESnet Director Greg Bell. “Very soon, other data-intensive fields will benefit as well. We expect to see significant network traffic across the Atlantic from the astrophysics, materials science, genomics, and climate science communities.”
“The LHC will produce an ever increasing amount of data throughout its life and the ability to distribute this on a global scale is the key to its analysis,” said David Foster, who is deputy head of CERN’s IT Department and is responsible for international network strategy. “Close collaboration between Caltech and CERN, led by Harvey Newman, created the US LHCNet transatlantic networking capabilities for the U.S. physicists since the start of LHC operations and enabled a detailed understanding of the computing models. The European extension of ESnet will continue this work and provide increased levels of capacity to explore new approaches to analyzing the LHC data.”
The global Internet depends on undersea cables for data exchange between continents [seesidebar]. Because most large scientific collaborations are international in scale, such cables are especially important for research networks such as ESnet. In fact, national networks around the world have been collaborating to test high-speed undersea connections for the past 18 months. Last year, six organizations (ESnet, Internet2 and CANARIE in North America, with SURFnet, NORDUnet and GÉANT in Europe) deployed the world’s first 100 Gbps research link across the Atlantic. More recently, New Zealand’s REANNZ deployed an experimental 100 Gbps link across the Pacific.
ESnet’s transatlantic extension builds on the pioneering work of these projects. “Research networks around the world are collaborating to plan a more integrated global network architecture, and to upgrade intercontinental links as rapidly as possible,” said Bell. “We support these goals, and our Atlantic extension is designed in harmony with them.”
ESnet installed its first European network node at CERN (the major laboratory outside Geneva that houses the LHC) in mid-September, and is now deploying other equipment necessary to bring the first link online by October. The plan is for all links to be commissioned and in production by January 2015. The timing is important, because the LHC is undergoing upgrades and expected to resume operations next spring, at which point it will be generating significantly more scientific data every day.
The Technical Challenges of Transatlantic Connectivity
When ESnet engineers began to study the idea of building a new 100 Gbps network between the United States and Europe, a primary concern was ensuring that the service would be robust and built from multiple underlying links—so that if one went down, researchers could still rely on sufficient bandwidth. Based on data collected for years by Caltech physicist and networking pioneer Harvey Newman, the ESnet team understood that multiple cables are sometimes cut simultaneously.
ESnet’s transatlantic extension will use network capacity leased from the owners of four undersea cables. Each cable has several pairs of optical fiber, with one fiber in each pair used for sending data from the U.S. to Europe, and the other used for data moving in the reverse direction. Each fiber is “lit” using dense wavelength division multiplexing (DWDM), a technology that sends data from different sources using different colors of light, allowing between 40 and 80 different signals to be sent on a single fiber simultaneously.
Unfortunately, any undersea or ‘submarine’ cable faces a number of threats, from ship anchors to landslides, according to Joe Metzger, ESnet’s lead engineer for the project.
Ship anchors cause most cable cuts near the coastline. Even though undersea cables are designed with extra armor near land, and are buried up to a meter deep, the anchors dropped by supertankers riding out storms in the North Sea can still damage the fiber optics inside the cables.
Undersea earthquakes pose another hazard. According to Metzger, one such quake triggered an underwater landslide that took out a number of cables all laid through the same channel.
One of the challenges in seeking the safest routes for network links is that while each cable operator knows the locations of its own cables, there’s no accurate database covering all cable paths.
Another major issue is the time needed to repair a break. In its network connecting sites across the terrestrial U.S., ESnet can usually have an engineer at the site within four hours of a problem being reported. And specialized equipment for cable repair is usually in place within eight to 12 hours. In a worst-case scenario, it may take 36 hours to get the link back in service when that link is on land.
With an undersea break, it can take a repair ship up to a week just to find the problem. Once located, the ship uses an underwater robot to cut the cable and affix a buoy to the end, which then raises the end of the cable to the ocean surface. The process is repeated for the other end. The damaged section is removed, a new piece of cable spliced in and the repaired cable then lowered back to the seabed. The procedure can be delayed by storms and the limited number of repair ships. In some cases, it can take four weeks or longer to repair the undersea break.
The challenge for ESnet, then, was to design a transatlantic extension with four separate links to minimize the probability of a complete network outage. Each link needed to follow a separate path. After getting proposals from a number of vendors, ESnet staff wrote a computer program to analyze and rank all of the detailed cable maps to select the four connections that had the least amount of overlap. And to provide an extra measure of safety, ESnet is working with other research networks in North America and Europe to arrange mutual backup in case of catastrophic failure.
Because the scope of the ESnet project breaks new ground for the international networking community, Metzger will give a presentation on the two-year project at the 2014 Technology Exchange conference in late October 2014. The Oct. 26-30 conference is the leading networking conference in the U.S. and draws attendees from around the world.