ESnet’s High-Touch Project

Providing high-fidelity network telemetry and measurement at scale

Overview

ESnet's High-Touch Project uses programmable hardware and software to provide enhanced, high-speed network services and detailed network traffic visibility. The project was begun in 2018 as part of development of the ESnet6 network, the Department of Energy’s high performance network for unclassified scientific data.

Utilizing Field Programmable Gate Arrays (FPGAs) and a containerized software stack, the system is able to observe multiple 100GE / 400GE links without needing to resort to sampling or aggregation. This capability enables High-Touch systems to produce accurate, high-fidelity network telemetry measurements, even in the face of increasing network speeds and traffic volumes. These measurements provide valuable insights into network behavior for troubleshooting, traffic engineering, and cybersecurity. In the future, this data will also enable efforts towards AI/ML-driven network operations.

The High-Touch hardware

The High-Touch Hardware

The FPGA cards used in the system are commercial off-the-shelf devices. Each has two 100GE network ports. Both are connected directly to ports on one of ESnet’s core routers, with the routers configured to mirror the first 128 bytes of every packet into or out of edge-facing router ports. Mirroring only a portion of each packet reduces the risks from handling payload data while still preserving the significant parts of packet headers. It also multiplies the amount of data that can be processed, because the payload data in the original packets do not consume bandwidth on the links between the core router and the FPGA card.

ESnet has deployed more than 80 High-Touch servers (containing Xilinx U280 FPGA cards and a containerized software stack) to 40-plus locations on its network footprint throughout the United States and Europe. The intent of these servers is to observe traffic crossing ESnet’s “edge,” consisting of links to science laboratory sites, R&E network peerings, and commercial peers.

ESnet has deployed more than 80 High-Touch servers in 40-plus locations in the U.S. and Europe. (Click image to enlarge.)

The Software Stack

The High-Touch FPGA cards are supported by a containerized software stack, comprising a mixture of internally developed software and commonly available open-source software. The main functionality of the software is to add metadata annotations to the raw flow- and packet-level measurements from the hardware, and make them available to consumers of the data. The software stack also manages the FPGA cards and makes operational measurements available via Prometheus metrics.

The High-Touch Data

High-Touch offers both flow-level and packet-level telemetry data. Flow-level data provides an overview of traffic. These measurements are similar to those provided by NetFlow and IPFIX, but also include information that requires a complete, unsampled view of traffic, such as packet size histograms and loss/retransmission counts. High-Touch flow data is stored in a ClickHouse database for analysis and retrieval. ClickHouse is a columnar relational database that works well for storing large amounts of invariant data, such as measurement data points. It supports a powerful version of SQL for performing complex queries against the data.

The life of a packet, as it travels through the High-Touch system. (Click image to enlarge.)

Packet-level visibility allows for the capture and saving of headers for all packets in a given flow or flows, configured via Web interface provided by an internal ESnet service known as FUNCAP. The Web interface allows specifying the addresses of interest, to be captured over a given time interval. Matching packets are (by default) captured from wherever they are seen along the network edge. The output of this service is a single, multi-interface pcapNG file that holds the headers of all matching packets.

Current Uses

The High-Touch system is, at the moment, intended for internal ESnet use, with the goal of providing improved network services through better visibility. It is based on a lower layer of functionality that supports running network applications on FPGA hardware. Named ESnet SmartNIC, this lower layer has been released as open source on Github and has been used to create several interesting applications inside and outside of ESnet, including the EJFAT load balancer. ESnet’s long-term goal for the High-Touch system is to make it available as open-source software for the networking community.