******DRAFT******* ESNET IP ROUTING Version 3.2 M. Collins R. Nitzan September 08, 1989 -i- page ---- Contents STATUS ........................................................... 1 ACKNOWLEDGEMENTS ................................................. 1 DEFINITIONS ...................................................... 2 INTRODUCTION ..................................................... 3 TRAFFIC MODEL .................................................... 5 CONDENSED TRAFFIC FLOW MODEL ............................... 7 ROUTE ADVERTISEMENT OVERVIEW ............................... 10 INITIAL ROUTING IMPLEMENTATION PLANS ............................. 16 BACKBONE CONNECTIONS ............................................. 18 REGIONAL NETWORK CONNECTIONS ..................................... 19 CONNECTING TO DOE BACKBONE SITES ................................. 20 APPENDIX A ....................................................... 34 DOE BACKBONE SITE NETWORK NUMBERSAND REGIONAL CONNECTS ..... 34 APPENDIX B ....................................................... 35 CURRENT DOE NON-BACKBONE SITES ............................. 35 APPENDIX C ....................................................... 36 EXPANDED ESNET TRAFFIC FLOW MODEL .......................... 36 DRAFT Routing Spec 09/08/89 Page 1 STATUS ----------------------------------------------------------------------- This document is a DRAFT of the IP routing guideline for ESNET. It is not a routing specification for any one particular site. The document focuses on the issues involved in IP routing. DECNET routing issues are not addressed in this document. This document will constantly evolve. It is intended to accurately reflect the models and traffic flow patterns for ESNET, at any given time. The current version focuses on the "Condensed Traffic Flow Model" which is the model for establishing connectivity among the 19 major DOE laboratories and their collaborators. The "Expanded ESNET Traffic Flow Model", which is anticipated for the future, is included as an appendix. The major difference between the two models is the way DOE non-backbone Site traffic is routed. ACKNOWLEDGEMENTS ----------------------------------------------------------------------- Much appreciation is extended to Hans-Werner Braun (Merit/NSFNET), Scott Brim (Cornell Theory Center), Tony Hain (DOE LLNL/ESNET), Sue Hares (Merit/NSFNET), Jim Leighton (DOE LLNL/ESNET), Milo Medin (NASA) and Jessica Yu (Merit/NSFNET) for their time reviewing this document and providing helpful knowledge regarding routing issues. DRAFT Routing Spec 09/08/89 Page 2 DEFINITIONS ----------------------------------------------------------------------- backbone interconnect A connection between two or more national backbone networks, with EGP peering. DOE Backbone Site: A DOE funded facility designated to be directly connected to the ESNET backbone. DOE Backbone Site network: One or more networks local to a DOE Backbone Site. DOE non-Backbone Site: A DOE funded facility not on the ESNET backbone, but designated for ESNET support. DOE non-Backbone Site network: One or more networks local to a DOE non-Backbone Site ESNET/regional interconnect A connection between ESNET and a regional, with EGP peering. Routes to a regional's site networks are placed into the ESNET backbone at this point. peering: An exchange of Exterior Gateway Protocol (EGP) reachability information between two routers in different administrations, enabled by connections to a common network. This may be done with more advanced reachability protocols as they become available. peer connection: See above regional network: A network within a geographic region under one administration, connecting Universities, Laboratories and other institutions together, with one or more connections to other external networks. For the scope of this document these are typically one of the designated NSF regionals. Unless otherwise specified, all references to regional networks are to those with a DOE Backbone Site connected to it. regional site network: A site network connected to, and affiliated with, a given regional network. DRAFT Routing Spec 09/08/89 Page 3 INTRODUCTION ----------------------------------------------------------------------- The Energy Sciences Network (ESNET) is a nationwide backbone network for the Department of Energy providing T1 service to programs funded by the Office of Energy Research. The five current programs are: 1) Applied Math Sciences 2) Basic Energy Sciences 3) Fusion Energy 4) Health and Environmental Research 5) High Energy and Nuclear Physics ESNET is installed and operated by the National Magnetic Fusion Energy Computer Center (NMFECC) located at Lawrence Livermore National Laboratory. ESNET is intended to provide connectivity between the major DOE laboratories, sub-contractors, and collaborators at various Universities and institutions. The ESNET backbone has multiple protocol routing capabilities, where initially both DOD-IP and DECNET phase IV traffic will be supported. Initially, 19 major DOE funded facilities in the United States will be connected to the ESNET backbone. In addition to the DOE backbone facilities there are a number of other Universities and institutions which collaborate with the DOE backbone sites. These sites are also authorized to use the ESNET backbone for communications with the major laboratories, and between themselves. Appendix B lists the Universities and institutions which currently have access to major DOE laboratories, and each other, using MFEnet I. The number of such sites is expected to increase. At this time, none of these sites are planned to have a T1 connection to the ESNET backbone. The routing issues of ESNET as a national backbone network involve connections to other backbone networks, regional networks and local area networks. Each category involves its own unique set of problems, yet they all interrelate. The Condensed Traffic Flow Model is outlined, followed by details on interconnects to other backbones, regional networks and DOE Backbone Site networks. There are numerous DOE Backbone Site network configurations anticipated, several of which are studied in detail. The other agency backbone networks referred to in this document are: National Science Foundation Network (NSFNET), NASA Science Network (NSN) and the MILNET portion of the Defense Data Network (DDN). DRAFT Routing Spec 09/08/89 Page 4 Planned ESNET Backbone Topology ------------------------------- CERN FRG | / | / -------------------------------FNAL--------------------MIT / ----- PNL / \ / / / / ANL---------- / NYU LBL-----LLNL / \ \ NYC_/ / /| \ / \ \ |\__BNL SLAC / | \ / \ \ | \ AMES | \_________________/___________\_______ \ | \ | | \ \____\PPPL \ | | ORNL /| \ | | \ UM/ | CIT | | \ CEBAF \ | LANL | \ / \ | /\ | \ / UCLA | / \ | \ / \ | / \ SSC-------------------FSU GAC------ \ / UTA/ All lines are full-channel T1, except overseas links to CERN and FRG. DOE Backbone Sites and Inter-Agency Connection Sites ---------------------------------------------------- **AMES NASA AMES Research Center ANL Argonne National Lab BNL Brookhaven National Lab CEBAF Continuous Electron Beam Accelerator Facility CERN Centre Etudes Recherche Nucleaire (near Geneva, Switzerland) CIT California Institute of Technology FNAL Fermi National Accelerator Lab FRG Federal Republic of Germany (Garching) FSU Florida State University GAC General Atomics LANL Los Alamos National Laboratory LBL Lawrence Berkeley Laboratory LLNL Lawrence Livermore National Laboratory MIT Massachusetts Institute of Technology NYC New York City POP NYU New York University ORNL Oak Ridge National Laboratory PNL Pacific Northwestern Laboratory PPPL Princeton Plasma Physics Laboratory SLAC Stanford Linear Accelerator SSC Super Conducting Collider UCLA University of California at Los Angeles **UM University of Maryland at College Park UTA University of Texas at Austin ** indicates Inter-Agency Connection Sites DRAFT Routing Spec 09/08/89 Page 5 TRAFFIC MODEL ----------------------------------------------------------------------- ESNET is a result of DOE's recognition of the importance of networking to its overall mission. ESNET is chartered to provide backbone network services to the five major research programs under the Office of Energy Research. By policy, this support is extended to all facilities which include DOE funding or collaborators thereof. The ESNET Traffic Flow Model presumes that ESNET constitutes the best route for all DOE related traffic. Therefore, the basic goal of the Model is to use ESNET for DOE traffic to the greatest extent possible. Since the ESNET backbone consists of all full-channel T1 circuits, and has a current average hop-count diameter of 2.8, this is probably a correct assumption in most cases, although it is recognized that some exceptions may occur. The Condensed Traffic Model, discussed in the rest of this document, focuses on traffic that is generated by a DOE Backbone Site and/or destined to a DOE Backbone Site. The Expanded Traffic Model, as outlined in Appendix C, will be addressed in the future. | | A Model is used to clearly define the intended traffic flow over | ESNET, as well as to provide a consistent guideline for implementing | route advertisements between ESNET and other networks. The Condensed | Traffic Flow Model is based on the assumption that ESNET is the primary | carrier for the DOE Backbone Site traffic. It is used as the basis for | network advertisements in the rest of the document. | | Adherence to this Model is not mandatory. The Site must choose a | primary backbone which will tell other networks how to route back to | it. This document describes a Model which presumes ESNET to be the | primary backbone for DOE traffic. There are other reasonable | alternatives to this Model that a site may choose, however, | alternatives must be reviewed carefully for they may have impact on | other networks besides the DOE Backbone Site. Some DOE Backbone Sites | will want ESNET as the primary backbone immediately. Other DOE | Backbone Sites may choose, at least initially, to use NSFNET as their | primary carrier. In this case the site would communicate between other | backbone sites using ESNET, and all other traffic will follow current | communication paths. A DOE Backbone Site that does not to conform to | this Model or another reasonable and coordinated solution will increase | the probability of routing loops and asymmetric routes, which | complicates the maintenance and diagnostic services of the national | backbones and other regional networks' staff, as well as causing | additional complexity for other sites establishing traffic routes to | the non-conforming site. Therefore, adherence by a DOE Backbone Site | to this Model or another acceptable variation is vital. | A major effort has been made to design the Model and routing so that asymmetric paths are avoided. This is highly desirable for several reasons. Detecting routing problems becomes easier when traffic uses the same path to get to a destination and to return to its source. Additionally, asymmetric paths may result in traffic using a DRAFT Routing Spec 09/08/89 Page 6 different speed path in each direction resulting in apparent service degradation. Since the granularity of route advertisements between autonomous systems (any point where EGP is used) is restricted to network numbers, this document discusses the Model and route advertisements of networks. Subnet and host routes cannot be supported across the backbone. The Model and network advertisements discussed in this document comprise a simple guideline for policy based routing, based on the selective advertisements of networks. Adherence to this Model may require some facilities to separate into DOE and non-DOE networks with different network addresses. In order to achieve interconnectivity with other networks, ESNET will connect and exchange reachability information with other agencies and regional networks. Exact locations for these connects, as well as which networks are actually advertised between the agencies, will be chosen and monitored carefully. Currently, most DOE Backbone Sites have a regional network connection. The regional network will be used for traffic between that DOE Backbone Site network and any regional site network within that same regional network, except for traffic to another DOE Backbone Site. By peering with the regional network at the DOE Backbone Site, regional site networks can be known in the ESNET backbone routing tables. This will allow traffic between a regional site network and a DOE Backbone Site network (not in that regional network) to traverse the regional to the ESNET/regional interconnect point, and then traverse the ESNET backbone to get to the DOE Backbone Site network. Initially, one ESNET/regional interconnect point is established per regional network. DOE non-Backbone Site networks are treated the same as any other regional site network in the Condensed Traffic Model. In the future some of these site networks may be treated differently. If the hosts that DOE researchers use can be segregated, and a different network number assigned to that group of hosts, they may become DOE backbone Site networks. Alternatively, the new site network could remain a non-Backbone Site network and ESNET could advertise the new site network into other agency backbone networks and regional networks with ESNET connections. The non-Backbone Site network's traffic would then be directed to ESNET by its local regional network. This would allow non-Backbone Site network <-> non-Backbone Site network traffic to use ESNET. These cases are outlined in Appendix C, "Expanded ESNET Traffic Model", however, the details are left for a future version of this document. The Model is a general overview. The issues of other external connections to a DOE Backbone Site network (in addition to ESNET and a regional network) will be addressed on a site specific basis. DRAFT Routing Spec 09/08/89 Page 7 CONDENSED TRAFFIC FLOW MODEL ---------------------------- The initial traffic flow over ESNET, other agency backbones, regional networks and DOE networks is outlined below. 1. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORKS: 1.1 All cases: (DOE Backbone Site network) <-> (ESNET) <-> (DOE Backbone Site network) 2. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORK AND REGIONAL SITE NETWORK: 2.1 DOE Backbone Site network and regional site network within the same regional network: (DOE Backbone Site network) <-> (regional network) <-> (regional site network) 2.2 DOE Backbone Site network and regional site network in different regional networks: 2.2.1 ESNET peering with the regional network: (DOE Backbone Site network) <-> (ESNET) <-> (remote regional network) <-> (remote regional site network) 2.2.2 ESNET not peering with the regional network: (DOE Backbone Site network) <-> (ESNET) <-> (NSFNET) <-> (remote regional network) <-> (remote regional site network) 3. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORK AND OTHER AGENCY SITE NETWORK: 3.1 Interconnect between Agency Backbone and ESNET: (DOE Backbone Site network) <-> (ESNET) <-> (Agency Backbone) <-> (Agency Site Network) 4. TRAFFIC BETWEEN DOE SITE NETWORKS NOT CONNECTED TO ESNET OR TO A REGIONAL NETWORK: 4.1 To be determined on a case-by-case basis The traffic flows are diagramed on the following page. DRAFT Routing Spec 09/08/89 Page 8 NSFNET Backbone --X---------X--------------------------------------------X---------X-- | | (2.2.2) | | | .................................................... | | | : | : | | | : | (2.2.1) : | | | : | .............................. : | | | :............:........ : : | | --X---------|--X-:-------:--X--------------X--:--------:-----------X-- ESNET | | : : | .......... | : : | Backbone | | : : | : (1.1) : | : : | | | : : | : : | : : | R ----X--X-:-- : | : : | : ---:-X------ R E | : | : | : : | : | : | E G | \:/ | : | : : | : | \:/ | G I | .......... | : | : : | : | .......... | I O | :DOE non-: | : | : : | : | :DOE non-: | O N | :Backbone: | : | : : | : | :Backbone: | N A | :net : | : | : : | : | :net : | A L | :........: | : | : : | : | :........: | L | /:\ | : | : : | : | | A --:-X------- : | : : | : ------------ B : | : | : : | : : | : | : : | : (2.1): ------- : | : : | : :........ | : | : : | : \:/| \:/ | \:/ \:/ | \:/ --X-------X-- -----X------ | | | | |DOE Backbone | |DOE Backbone| |Site net D | |Site net E | ------------- ------------ This diagram shows some of the connections relative to descriptions in the configuration sections that follow. The traffic paths shown are from the Condensed Traffic Plan. The numbers in parentheses correspond to the sections of the Condensed Traffic Plan. - 2 locations showing inter-agency backbone connections between ESNET and NSFNET. - A DOE Backbone Site network connected to ESNET and connected to a regional network (DOE Backbone Site network D connects to ESNET and regional network A). - a DOE Backbone Site network where there is no regional network connection to it (DOE Backbone Site network E). - Regional B shows no connection to a DOE Backbone Site network and no ESNET/regional interconnect. Note that there may be more connections where backdoor routes between the regional networks occur, that are not shown, and are not supported by the routing guidelines presented here. DRAFT Routing Spec 09/08/89 Page 9 ESNET | BACKBONE ....................................... | --------------:--X--------------X----------------X--:-------------- | : | | | : | : | | | : | : | | | : | : | -----X------ | : | : | | | | : | : | | REGIONAL | | : | : | ---X X--- | : | : | | | | | | : | : | | ------------ | | : | : | | | | : | \:/ | | | | \:/ | --X----X----- --X-----X--- | | | | | | |DOE Backbone | |DOE Backbone| | | Site | | Site | | ------------- ------------ | | | This diagram shows the traffic flow between two DOE Backbone Sites | that both have connections to a common regional. | DRAFT Routing Spec 09/08/89 Page 10 ROUTE ADVERTISEMENT OVERVIEW ---------------------------- The various agency backbones will advertise EGP reachability information to each other at the interconnect peering locations. Therefore, ESNET will know how to forward NSFNET, NSN and MILNET traffic. Conversely, these backbones will know how to forward ESNET traffic. ESNET will offer to peer with regional networks that contain DOE Backbone Site networks or that contain non-Backbone Sites with appreciable traffic requirements. Initially peering will be done at one location per regional network, even if the opportunity exists in more than one location where a DOE Backbone Site network has external connections to both ESNET and to the regional network. Reachability information about regional site networks with no ESNET/regional connection will be obtained by peering with NSFNET. The following sections outline the traffic flows between, ESNET, other agency backbones, regional site networks and DOE Backbone Site networks. It also are summarize the route advertisements independent of the protocols used to implement them. Secondary route advertisements are not included since the focus is on the intended traffic flow. The issues are examined from the following points of reference: o A DOE Backbone Site network with a connection only to ESNET. o A DOE Backbone Site network with connections to both ESNET and a regional network. o A Regional network with an ESNET/regional interconnect and one or more DOE Backbone Site networks. o A Regional network without an ESNET/regional interconnect or a DOE Backbone Site network, connected to another agency backbone. o ESNET o Other agency backbones (NSFNET, NSN, MILNET) 1. A DOE Backbone Site network with a connection only to ESNET. Traffic Flow ------------ - All traffic goes to/from ESNET. Network Advertisements ---------------------- - Routes for all available networks (or default) are advertised by ESNET to the DOE Backbone Site network. - Route(s) for the DOE Backbone Site network(s) are accepted from the DOE Backbone Site by ESNET. DRAFT Routing Spec 09/08/89 Page 11 2. A DOE Backbone Site network with connections to both ESNET and a regional network, (assume this is an ESNET/regional interconnect). Traffic Flow ------------ - All site traffic goes to ESNET except: Traffic to the regional's site networks goes to the regional network, unless they are another DOE Backbone Site network. Network Advertisements ---------------------- - Routes for all available networks, excluding those that are on the regional network that is connected to the DOE Backbone site network, are advertised by ESNET to the DOE Backbone Site network. It may be preferable to advertise default in some cases. - Routes for regional site networks (excluding DOE Backbone Site networks) are accepted from the regional network by the DOE Backbone Site network. - Route(s) for the DOE Backbone Site network(s) are accepted from the DOE Backbone Site by ESNET and by the regional network. 3. A Regional network with an ESNET/regional interconnect and one or more DOE Backbone Site networks. Traffic Flow ------------ - Traffic destined to DOE Backbone Site networks (within the regional) goes to the site through the regional network. - Traffic destined to other DOE Backbone Site networks (that the regional network is not connected to) goes to the ESNET/regional interconnect. Network Advertisements ---------------------- - Routes for DOE Backbone Site networks (except those directly connected to the regional network) are advertised by ESNET to the regional network at the ESNET/regional interconnect point. - Route(s) for connected DOE Backbone Site network(s) are accepted from the DOE Backbone Site by the regional network. - Routes for other networks are accepted from other agency backbone(s) by the regional network. - Routes for regional site networks (excluding DOE Backbone Site networks) are accepted from the regional network by the DOE Backbone Site network, ESNET (at the ESNET/regional interconnect, point) and other agency backbones. DRAFT Routing Spec 09/08/89 Page 12 5. A Regional network without an ESNET/regional interconnect or a DOE Backbone Site network, connected to another agency backbone. Traffic Flow ------------ - Traffic destined to DOE Backbone Site networks goes to NSFNET or another appropriate backbone which then forwards it to ESNET. Network Advertisements ---------------------- - Routes for DOE Backbone Site networks are advertised by NSFNET to the regional network. - Routes for regional site networks are advertised by the regional network to NSFNET or other appropriate backbone(s). 6. ESNET Traffic Flow ------------ - Traffic to DOE Backbone Site networks goes through ESNET. - Traffic to a regional site network (non-DOE Backbone Site), with an ESNET/regional interconnect somewhere on the regional network, goes to the regional network. - Traffic to a regional site network, with no ESNET/regional interconnect anywhere on the regional network, and traffic to a site on another agency backbone, goes to NSFNET or another appropriate backbone. Network Advertisements ---------------------- - Routes for regional site networks (non-DOE Backbone Sites), where the regional network has an ESNET/regional interconnect peering point, are accepted from the regional network by ESNET. - Route(s) for DOE Backbone Site network(s) are accepted from the DOE Backbone Site network(s) by ESNET. - Routes excluding the above are accepted from other agency backbones by ESNET. - Routes for DOE Backbone Site networks are advertised by ESNET to other DOE Backbone Site networks. - Routes for DOE Backbone Site networks are advertised by ESNET to other agency backbones. - Routes for DOE Backbone Site networks are advertised by ESNET to regional networks, (except when the DOE Backbone Site network also has a connection to that regional network). Note: When two backbones are peering, one or both backbones should filter out routes that are actually part of a third backbone, where the third backbone is also peering with one or both of them as well. DRAFT Routing Spec 09/08/89 Page 13 7. Other Agency Backbones (NSFNET, NSN, MILNET) Traffic Flow ------------ - Traffic to DOE Backbone Site networks use ESNET as the primary path. Network Advertisements ---------------------- - Routes to DOE Backbone Site networks are advertised by ESNET to the other agency backbones. - Routes to other agency networks are accepted from the appropriate backbone by ESNET DRAFT Routing Spec 09/08/89 Page 14 ROUTE ADVERTISEMENT FLOW AND FILTERING ESNET Site Regional NSFNET ----- ----- ----- ----- | | | | | | | | | S - - - -X- - - -> S ---------------> S - - - -X- - - > S | | <--------------- <- - - -X- - - - <- - - -X- - - - | | | | | | | | | | | | | | | | | | R - - - -X- - - -> R - - - -X- - - > R ---------------> R | | <- - - -X- - - - <-------------- <- - - - X - - - | | | | | | | | | | | | | | | | | | E ---------------> E - - - - X - - -> E - - - - X - - -> E | | <- - - -X- - - - <- - - -X- - - - <- - - - X - - - | | | | | | | | | | | | | | | | | | N ---------------> N - - - - X - - -> N - - - - X - - -> N | | <- - - -X- - - - <- - - - X - - - <-------------- | | | | | | | | | | | ----- | | | | | | | | | | | S - - - - - - - - -X- - - - - - - - > S | | | | <- - - - - - - - -X- - - - - - - - | | | | | | | | | | | | | | | | R - - - - - - - - -X- - - - - - - - > R | | | | <---------------------------------- | | | | | | | | | | | | | | | | E ----------------------------------> E | | | | <- - - - - - - - -X- - - - - - - - - | | | | | | | | | | | | | | | | N - - - - - - - - -X- - - - - - - - > N | | | | <- - - - - - - - -X- - - - - - - - - | | | | | | | | | | | ----- | | | S -----------------------------------------------------> S | | < - - - - - - - - - - - - X - - - - - - - - - - - - - | | | | | | | | | | R - - - - - - - - - - - - - X - - - - - - - - - - - - -> R | | <- - - - - - - - - - - - - X - - - - - - - - - - - - - | | | | | | | | | | E -----------------------------------------------------> E | | <- - - - - - - - - - - - - X - - - - - - - - - - - - - | | | | | | | | | | N - - - - - - - - - - - - - X - - - - - - - - - - - - -> N | | <----------------------------------------------------- | | | | | ----- ----- DRAFT Routing Spec 09/08/89 Page 15 The diagram on the preceeding page shows the routing information | flow for the model presented in this document. There are thus four | autonomous entities: the DOE Backbone Site, the regional network, the | ESNET backbone and the NSFNET backbone. | | The four entities give rise to four disjoint sets of routing | information: | | o A DOE Backbone Site's net(s), denoted by S; | o Regional's nets (excluding any DOE Backbone Site nets in the | regional), R; | o Networks connected to the ESNET backbone, excluding the Site, E; | o Everything else on the NSFNET, N. | | | Because of the multiple connections, all four entities will try to | supply all four sets of routes to everyone unless some filter | explicitly prevents this. Arrows indicate all possible routing | information exchanges. The arrow is broken with an X if there is a | routing filter that prevents the flow of some set of information. | Filters can be located at either the source or destination of the | routing information. An X marks possibly the most typical locations of | the filters. For example, since ESNET will never tell a DOE Backbone | Site about that site's networks, the S arrow from ESNET to Site is | broken with an X on the arrow indicates that the ESNET router will | never supply this information An arrow is solid if routing information | is exchanged. | | If the topology is to be loop free, no box can have two incoming | solid lines with the same label. For example, the Regional cannot | learn N from both the NSFNET and ESNET. | DRAFT Routing Spec 09/08/89 Page 16 INITIAL ROUTING IMPLEMENTATION PLANS ----------------------------------------------------------------------- The initial implementation of the backbone will be done in several phases. When the backbone is first installed, it is desirable to have a safe environment with possible areas of breakage reduced to a minimum. PHASE 1 ------- GOAL: o Provide connectivity between the 19 DOE Backbone Site networks. FEATURES: o No EGP peering with other agency backbones or regional networks o All DOE Backbone Site networks route statically to the ESNET router. Current connections between a DOE Backbone Site network and a regional network should still be used for non-DOE Backbone Site traffic. o The ESNET backbone will use dynamic IGP routing. PHASE 2 ------- GOAL: o Connect to other agency backbones at one (possibly two) locations. FEATURES: o ESNET will advertise, with EGP, all DOE Backbone Site networks with a different AS number at each of these backbone peering points. o ESNET will accept, with EGP, network reachability information from other agency backbones. o The routes learned from other agency backbones are transferred to the IGP within ESNET's backbone. o The DOE Backbone Site networks may also route non-DOE Backbone Site traffic to the ESNET router, as ESNET now knows how to route it to other agency backbones, and they know how to route it back to ESNET. DRAFT Routing Spec 09/08/89 Page 17 PHASE 3 ------- GOAL: o Connect to the regional networks that have DOE Backbone Site network connections. FEATURES: o ESNET will advertise, with EGP, all DOE Backbone Site networks (not in the regional network) to that regional network. o ESNET will accept, with EGP, network reachability information about regional site networks from regional networks. DRAFT Routing Spec 09/08/89 Page 18 BACKBONE CONNECTIONS ----------------------------------------------------------------------- Several agencies (DOE, NASA, NSF, DCA) are proposing at least 2 major sites across the continental US for inter-agency backbone connections. The backbones, in addition to ESNET are: NSFNET, NSN and MILNET. The locations are to be geographically dispersed (east coast, central, west coast and possibly the pacific) and meet certain requirements, some of which are: (1) Have UPS power (2) 24 hour, 7 days a week operator services (3) Easy access for maintenance purposes (which automatically excludes any classified locations as possible sites) Two site locations have been agreed upon: the west coast connection at NASA Ames Research Center and the East coast connection at the University of Maryland, College Park. The plan is to form EGP peer connections to the backbones with which ESNET is interested in communicating and advertising the DOE Backbone Site networks. By establishing one EGP peer session, with NSFNET for example, this could indirectly imply EGP peering with those that peer with NSFNET on that same wire (NSN, MILNET). For robustness, however, ESNET will probably peer directly with the other backbones. Each backbone connection will EGP peer with a different AS number. The reachability information received through EGP will be transferred into the IGP running in the backbone. As a safety measure, filters will be used in case other backbones advertise the DOE Backbone Site networks that ESNET carries. ESNET has been assigned the following Autonomous System numbers: 291, 292 and 293. DRAFT Routing Spec 09/08/89 Page 19 REGIONAL NETWORK CONNECTIONS ----------------------------------------------------------------------- Interconnections between ESNET and a regional network are possible when an ESNET router and a regional network router are co-located on the same physical network. Typically they will exchange reachability information with EGP. ESNET will only accept advertisements from the regional network for regional site networks that are part of that regional network. ESNET will only advertise the DOE Backbone Site | networks to the regional network. If these regional networks are also | advertised by NSFNET, to ESNET, they will be filtered out at the | ESNET/NSFNET peering point. Similarly, NSFNET advertisements of DOE | Backbone Site networks to the regional network should be filtered out | by either NSFNET or the regional network. | Most regional networks have more than one DOE Backbone Site network connected to it. Initially, the ESNET backbone will establish an EGP peering interconnect point with the regional network at only one of these DOE Backbone Site network locations. Therefore, even if an ESNET router and a regional network router are co-located, this does not necessarily mean that they will form an EGP interconnect peering point. Traffic between a DOE Backbone Site network and a local regional site network traverses the regional network. Traffic from a regional site network destined to a remote DOE Backbone Site network will traverse the regional network to the one ESNET/regional interconnect point in order to be carried to its destination by the ESNET backbone. It is strongly recommended that all DOE Backbone Site networks with a regional network router, (or any other external path router), co-locate that router with the ESNET router. DRAFT Routing Spec 09/08/89 Page 20 CONNECTING TO DOE BACKBONE SITES ----------------------------------------------------------------------- This section of the routing guideline examines various possible local site configurations with their external connections. It is not intended to be a site specific guide. The following configurations assume co-location of the ESNET and regional network routers. | | ESNET will coordinate with the DOE Backbone Site, regional network | and other backbones, prior to implementing or significantly changing | the advertising parts of the Model. The DOE Backbone Site will choose | which of the backbones will be the primary carrier for its network(s), | and thus where other networks will look to find them. Specific | procedures are beyond the scope of this document, yet the intent is to | make sure all the people involved are informed and that changes are | synchronized appropriately. | ESNET's connection to the site may also involve a peering with the regional network to acquire routes for regional site networks in the ESNET backbone routing tables (an ESNET/regional interconnect). If the site is not already connected to a regional network, its traffic to NSFNET and other agency backbones will be routed by ESNET, since ESNET knows these routes by peering with other agency backbones. Most of the configurations shown have an external connection to a regional network as well as to ESNET. Advertisement issues that are explored cover the site as if it were the ESNET/regional interconnect point (since its more interesting). However, because this is not always the case, there is a brief comment on the implications of not peering for each of the configurations. Some configurations are such that peering is a requirement and some configurations may peer optionally. In the following discussions IGP is used as a generic term to mean any one of several Interior Gateway Protocols. Since different router vendors implement different IGPs, some sites may not fit into the configurations below because the IGPs may be incompatible. This type of problem will be addressed as a site-specific routing plan. The network advertisements outlined in the following configuration sections are meant to show how routing can be accomplished with various hardware configurations. The feasibility of a specific configuration will depend on the type of routers. It is highly desirable to have as much filtering as possible done in the ESNET routers. Filtering is typically more flexible for EGP than an IGP, since all vendors will allow filtering of EGP routes when redistributed into the IGP. Configuration 3 should work for most situations. In some of the following cases using ESNET and the regional network as a backup for each other is possible. This is not addressed in detail in the following descriptions. Usage and implementation of backdoor and sidedoor routes must be carefully coordinated, as they are tricky. They are not supported by the routing guidelines in this DRAFT Routing Spec 09/08/89 Page 21 document, since they are very case specific. ESNET routers will be configured to carry DECNET traffic, and may be enabled to the site network if the DECNET addressing restrictions are adhered to. The basic hardware requirements are a router with 2 T1 interfaces and 1 Ethernet interface. | | | | | EXAMPLE OF A LOCAL SITE'S EXTERNAL CONNECTIONS | ---------------------------------------------- | | | | | | | | | | | | | | ------------ ------------ ------------ | | ESNET | | REGIONAL | | Other | | | | | | | External | | | Router | | Router | | Router | | ------------ ------------ ------------ | |Default |Regional | Other nets | | || | || nets | || | | \/ | \/ | \/ | | | | | ---------------------------------------------------------- | Class C Net Private | | Ethernet | | | /\ | | || Local site networks | ------------ | | Local site | | | | | | Router | | ------------ | | || | | \/ Default | | | | | ---------------------------------------------------------------------- | Local Site Ethernet | | | | This diagram shows an example of how a local site may connect with | several external routers. The external routers, shown above the local | site router, are co-located on a stub Ethernet. The arrows show an | example of how routing updates may be sent. With this configuration | the routing updates generated by the external routers are kept off the | local site Ethernet. This cuts down on traffic and processing time on | the local Ethernet and allows a site to cut all external paths at one | point. | DRAFT Routing Spec 09/08/89 Page 22 CONFIGURATION ONE ----------------- -------- | ESNET | | ROUTER | -------- | Site IGP/Default | ------------------------------- <-- Ethernet Local DOE Backbone Site network EGP REACHABILITY ---------------- None. IGP ROUTING ----------- ESNET sends a default route into the local site with an IGP. ESNET accepts routing information about DOE Backbone Site networks advertised by the local site with the site IGP. ESNET distributes information about these networks into the ESNET backbone, as well as externally with EGP to other backbones and regional networks. NO ESNET/REGIONAL INTERCONNECT ------------------------------ Not feasible. ADDITIONAL HARDWARE ------------------- None. DECNET CONSIDERATIONS --------------------- Since the ESNET router is directly connected to the site Ethernet, any DECNET traffic (within ESNET's known addressing space) may reach the local site. DRAFT Routing Spec 09/08/89 Page 23 CONFIGURATION TWO ----------------- OPTION A -------- --------- | ESNET |......EGP......| REGIONAL| | ROUTER | | ROUTER | -------- --------- Site IGP/ | | Site IGP Default | | ----------------------------- <-- Ethernet Local DOE Backbone Site network An EGP peer relationship is formed between the regional network and ESNET to exchange routes. With an IGP, the ESNET router advertises default and the regional network router advertises regional site networks to the local site. Filtering should be done in the regional network router. For this particular configuration it is recommended that it be converted to configuration 3. The scheme outlined below is only feasible if sufficient filtering capabilities are available in the routers. EGP REACHABILITY ---------------- ESNET sends reachability information to the regional network about all DOE Backbone Site networks using EGP. ESNET accepts reachability information from the regional network about all regional site networks. ESNET distributes these to its backbone with an IGP, but not externally to other backbones with EGP. Exception: Remote DOE backbone Site networks that are also connected to the regional network will neither be advertised by ESNET to the regional network, nor will they be accepted by ESNET as advertisements from the regional network, via EGP. IGP ROUTING ----------- ESNET sends routing information into the local site with an IGP. Either complete routes or a (preferable) default route may be sent. The regional network sends routing information about the regional site networks into the local site with an IGP, filtering out routes carried by ESNET. DRAFT Routing Spec 09/08/89 Page 24 Routes that should not be accepted from the regional network are other DOE Backbone Site networks either on that regional network or learned from NSFNET. Doing so may override a more direct, high bandwidth path through the ESNET backbone. In order to ensure the appropriate traffic uses the ESNET router, in case the regional accidentally advertises non-regional site networks, it may be necessary to: 1) Have the ESNET router advertise all its routes with a lower metric than the regional router (this is a problem if both the regional network and ESNET routers use the same low metric and the filtering capabilities in the routers are insufficient). 2) Use option B. 3) Adjust the physical configuration. ESNET and the regional network accept routing information about DOE Backbone Site networks from the local site with the site IGP. ESNET distributes information about these networks into the ESNET backbone with an IGP, as well as externally with EGP to other agency backbones. The regional network distributes these networks within its internal IGP, but not externally to other backbones with EGP. NO ESNET/REGIONAL INTERCONNECT ------------------------------ This configuration is also feasible if there is no ESNET/regional interconnect with peering. There is no EGP reachability information exchanged between the ESNET and regional network routers. ADDITIONAL HARDWARE ------------------- None. DECNET CONSIDERATIONS --------------------- Since the ESNET router is directly connected to the site Ethernet, any DECNET traffic (within ESNET's known addressing space) may reach the local site. DRAFT Routing Spec 09/08/89 Page 25 OPTION B -------- --------- | ESNET |......EGP......| REGIONAL| | ROUTER | | ROUTER | -------- --------- | | | | --------------------------------- <-- Ethernet Local DOE Backbone Site network This option has a variation in the routing, using the same hardware configuration as Configuration Two, Option A. The difference between Options A and B is that one of the routers does not advertise its routes to the DOE Backbone Site. Routes are exchanged between ESNET and the regional network via EGP. Traffic is directed to one router, and if appropriate, redirected to the other router. Due to the route exchanges between ESNET and the regional network, this configuration implies an ESNET/regional interconnect with peering. Using the ESNET router as the default router, redirecting to the regional network router, involves small EGP updates. ESNET sends reachability information about the DOE Backbone Site networks to the regional network. Reachability information accepted from the regional network, by ESNET, is for the regional site networks (non-DOE Backbone Site networks). Using the regional network router as the default router, redirecting to ESNET if necessary, involves ESNET advertising all its routes to the regional network router. This is because ESNET is the presumed carrier for the bulk of the remote traffic. In this case the EGP updates would be large. DRAFT Routing Spec 09/08/89 Page 26 CONFIGURATION THREE ------------------- -------- --------- | ESNET | | REGIONAL| | ROUTER | | ROUTER | -------- --------- | | | | | EGP | Site IGP/| --------------------- <--Ethernet or serial Default | | -------------------------------- <--Ethernet Local DOE Backbone Site network In this configuration all route filtering is done by ESNET. EGP REACHABILITY ---------------- ESNET sends reachability information about all DOE Backbone Site networks, including the local DOE Backbone Site network, to the regional network with EGP. ESNET accepts reachability information about regional site networks from the regional network with EGP. ESNET distributes these to its backbone with an IGP, but not externally to other agency backbones with EGP. Exception: Remote DOE backbone Site networks that are also connected to the regional network will neither be advertised by ESNET to the regional network, nor will they be accepted by ESNET as advertisements from the regional network, via EGP. IGP ROUTING ----------- ESNET sends routing information into the local site with an IGP. Either complete routes or a (preferable) default route may be sent. ESNET accepts routing information about DOE Backbone Site networks from the local site with the site IGP. ESNET distributes reachability information about these networks to the regional network with EGP. ESNET distributes information about these networks into the ESNET backbone with an IGP, as well as externally with EGP to other agency backbones. DRAFT Routing Spec 09/08/89 Page 27 NO ESNET/REGIONAL INTERCONNECT ------------------------------ ESNET sends reachability information about the local DOE Backbone Site network to the regional network with EGP. ESNET accepts reachability information about regional site networks from the regional network with EGP and distribute these routes to the DOE Backbone Site network, with the site IGP, but not to the ESNET backbone. Exception: Remote DOE backbone Site networks that are also connected to the regional network will not be accepted by ESNET as advertisements from the regional network, via EGP. ADDITIONAL HARDWARE ------------------- An Ethernet or serial connection for the ESNET to regional connect. A serial connection requires the ESNET and regional routers to be from the same vendor. DECNET CONSIDERATIONS --------------------- Since the ESNET router is directly connected to the site Ethernet, any DECNET traffic (within ESNET's planned DECNET addressing space) may reach the local site. DRAFT Routing Spec 09/08/89 Page 28 CONFIGURATION FOUR ------------------ -------- -------- | ESNET | | REGONAL| | ROUTER | | ROUTER | -------- -------- | | | | | EGP | | --------------------- <--Ethernet or Serial Site IGP/ | Default |- - - - - - - - - <--Ethernet | | -------- -------- | SITE | | DECNET | | ROUTER | | ROUTER | -------- -------- | | | Site IGP | | | ------------------------------ <--Ethernet Local DOE Backbone Site network This differs from configuration 3 in that there is a local router between ESNET and the DOE Backbone Site network. All route filtering can be done by the ESNET router. In addition, the site may enable filtering capabilities as well. EGP REACHABILITY ---------------- ESNET sends reachability information about all DOE Backbone Site networks, including the local DOE Backbone Site networks, to the regional network with EGP. ESNET accepts reachability information about the regional site networks from the regional network with EGP. ESNET distributes these networks within its backbone with an IGP, but not externally to other agency backbones with EGP. Exception: Remote DOE backbone Site networks that are also connected to the regional network will neither be advertised by ESNET to the regional network, nor will they be accepted by ESNET as advertisements from the regional network, via EGP. DRAFT Routing Spec 09/08/89 Page 29 IGP ROUTING ----------- ESNET sends routing information into the local site with an IGP. Either complete routes or a (preferable) default route may be sent. ESNET accepts routing information about DOE Backbone Site networks from the local site with the site IGP. ESNET distributes reachability information about these networks to the regional network with EGP. ESNET distributes information about these networks within the ESNET backbone with an IGP, as well as externally with EGP to other agency backbones. NO ESNET/REGIONAL INTERCONNECT ------------------------------ ESNET sends reachability information about the local DOE Backbone Site network to the regional network with EGP. ESNET accepts reachability information about regional site networks from the regional network with EGP and distribute these routes to the DOE Backbone Site network, with the site IGP, but not to the ESNET backbone. Exception: Remote DOE backbone Site networks that are also connected to the regional network will not be accepted by ESNET as advertisements from the regional network, via EGP. ADDITIONAL HARDWARE ------------------- An Ethernet interface for the ESNET router. DECNET CONSIDERATIONS --------------------- A DECNET router or additional Ethernet interface may be needed to route DECNET traffic if that capability is not available in the site router. DRAFT Routing Spec 09/08/89 Page 30 CONFIGURATION FIVE ------------------ Option A -------- --------- | ESNET | EGP | REGIONAL| | ROUTER |...............| ROUTER | -------- --------- | | Site IGP/ | | Site IGP Default | | ------------------------- <--Ethernet | | Site IGP | ------- | SITE | | ROUTER| ------- | | ------------------------------ <--Ethernet Local DOE Backbone Site network The difference between Options A and B is the protocol used between the local site router and the external network routers (ESNET and regional). In Option A an IGP is used and in Option B EGP is used. If the regional router uses an IGP then Option A is required. If both the ESNET and regional network routers agree to use EGP then Option B may be preferable. EGP REACHABILITY ---------------- ESNET sends reachability information about all DOE Backbone Site networks to the regional network with EGP. ESNET accepts reachability information from the regional network, for regional site networks, with EGP. ESNET distributes information about these networks into the ESNET backbone with an IGP, but not externally with EGP to other agency backbones. Exception: DOE backbone Site networks that are also regional site networks will neither be advertised by ESNET to the regional network, nor will they be accepted by ESNET as advertisements from the regional network. DRAFT Routing Spec 09/08/89 Page 31 IGP ROUTING ----------- ESNET sends routing information into the local site router with an IGP. Either complete routes or a (preferable) default route may be sent. The local site router should accept regional site network routes from the regional network with an IGP. Routing information about other DOE Backbone Site networks (in that regional) and non-regional site networks, should not be accepted from the regional network by the local site router. Setting a static default route to the ESNET router in the local site router, coupled with input route filtering in the local site router, could handle the case where the regional router does not have sufficient IGP filtering capabilities. ESNET and the regional network accept routing information about DOE Backbone Site networks from the local site with the site IGP. ESNET distributes information about these networks into the ESNET backbone with an IGP, as well as externally with EGP to other agency backbones. The regional network distributes these networks with its internal IGP but not externally to other agency backbones with EGP. NO ESNET/REGIONAL INTERCONNECT ------------------------------ This configuration is also feasible if there is no ESNET/regional interconnect with peering. There is no EGP reachability information exchanged between the ESNET router and regional network router. Option B: In this option EGP is used instead of an IGP between the external routers and the local site router. Both the ESnet router and the regional router peer with the local site router. The local site router directs traffic to the appropriate external router. ADDITIONAL HARDWARE ------------------- None. DECNET CONSIDERATIONS --------------------- A DECNET router or additional Ethernet interface may be needed to route DECNET traffic if that capability is not available in the site router. DRAFT Routing Spec 09/08/89 Page 32 CONFIGURATION SIX ----------------- -------- --------- | ESNET | | REGIONAL| | ROUTER | | ROUTER | -------- --------- | | | | Site IGP Site IGP/ | ------- Default | | SITE | | | ROUTER| | ------- | | Site IGP | | -------------------------------- <--Ethernet Local DOE Backbone Site network This configuration presents problems and complications that are not easily resolved. The scheme outlined below is only feasible if sufficient filtering capabilities are available in the routers. The problem is how to receive regional site network routes via the local site router. In general, ESNET and regional site network routes are preferably exchanged with EGP. EGP REACHABILITY ---------------- None. IGP ROUTING ----------- ESNET sends routing information into the local site with an IGP. Either complete routes or a (preferable) default route may be sent. The local site router should accept regional site network routes from the regional network with an IGP. Routing information about other DOE Backbone Site networks in that regional network and non-regional site networks should not be accepted from the regional network by the local site router. ESNET and the regional network accept routing information about DOE Backbone Site networks from the local site with the site IGP. ESNET distributes information about these networks into the ESNET backbone, as well as externally with EGP to other backbones. The regional network distributes these networks with its internal IGP, but not externally to other agency backbones with EGP. ESNET will accept information about regional site networks with an IGP. This may be DRAFT Routing Spec 09/08/89 Page 33 distributed into the ESNET backbone with an IGP, but not externally with EGP. Exceeding a hop count limit in the IGP may be an issue. NO ESNET/REGIONAL INTERCONNECT ------------------------------ This configuration does not have a direct connect between the ESNET router and the regional network router. However, with the appropriate filtering as described above, it is an indirect interconnect that may be used to pass traffic between ESNET and the regional network. If an interconnect is explicitly not desired (which is preferable), then ESNET does not accept regional network IGP routes and the regional network does not accept ESNET IGP routes. ADDITIONAL HARDWARE ------------------- None. DECNET CONSIDERATIONS --------------------- Since the ESNET router is directly connected to the site Ethernet, any DECNET traffic (within ESNET's planned DECNET addressing space) may reach the local site. DRAFT Routing Spec 09/08/89 Page 34 APPENDIX A DOE BACKBONE SITE NETWORK NUMBERSAND REGIONAL CONNECTS ------------------------------------------------------ DOE BACKBONE SITE NETWORK NUMBER REGIONAL CONNECTION --------------------------------------------------------------------- ANL 192.5.(170-175, UIUC/NCSA 195-197) 130.202 CICNET BNL 192.12.15 NYSERNET 130.199 CEBAF 129.57 SURANET CIT 131.215, SDSCNET FNAL 131.225 UIUC/NCSA FSU 128.186 SURANET GAC -none- LANL 128.165 LOSALOMOS LBL 128.3 BARRNET LLNL 128.115 BARRNET MIT 18 JVNCNET NYC NYSERNET NYU 128.122 NYSERNET ORNL SURANET PNL 130.20. NORTHWESTNET PPPL 192.55.13 -none- SLAC 134.79 BARRNET SSC UCLA SDSCNET UTA 128.83 SESQUINET BARRNET LBL, LLNL, SLAC CICNET ANL JVNCNET MIT NORTHWEST PNL NYSERNET BNL, NYU, NYC SDSCNET CIT, UCLA SESQUINET UTA SURANET CEBAF, FSU, ORNL UIUC/NCSA ANL, FNAL no connections PPPL, GAC note: NYC is actually a router located at a Point of Presence. Its physical location is in NYSERNET's area. DRAFT Routing Spec 09/08/89 Page 35 APPENDIX B CURRENT DOE NON-BACKBONE SITES ------------------------------ ABN -none- University of Auburn, Alabama CMR NYSERNET Columbia University COL WESTNET University of Colorado at Boulder CUI CORNELL Cornell University at Ithaca CYR -none- Cray Research Inc. (Livermore) DOE -none- Department of Energy at Germantown, Maryland GRM -none- Grumman Aerospace GTR SURANET Georgia Tech. ISU MIDNET Iowa State University (Ames) KSV MIDNET Kansas State University MDD -none- McDonnell-Douglas MSV -none- MacNeal-Schwendler NCV NCAR National Center for Atmospheric Research (NCAR) SAB -none- Sandia National Lab - Albuquerque SAV SDSCNET Science Applications Inc. SER WESTNET Solar Energy Research Institute (SERI) at Golden TRW -none- TRW, Los Angeles UCB BARRNET University of California at Berkeley UIU UIUC/NCSA University of Illinois at Urbana UMR SURANET University of Maryland UWM CICNET University of Wisconsin at Madison UWS NORTHWESTNET University of Washington at Seattle BARRNET UCB CICNET UWM CORNELL CUI MIDNET ISU, KSV NCAR NCV NORTHWEST UWS NYSERNET CMR SDSCNET SAV SURANET UMR, GTR UIUC/NCSA UIU WESTNET COL, SER DRAFT Routing Spec 09/08/89 Page 36 APPENDIX C EXPANDED ESNET TRAFFIC FLOW MODEL --------------------------------- 1. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORKS; 1.1 All cases: (DOE Backbone Site network)<->(ESNET)<-> (DOE Backbone Site network) 2. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORK AND DOE NON-BACKBONE SITE NETWORK: 2.1 DOE Backbone Site network and DOE non-Backbone Site network within the same regional network: (DOE Backbone Site network)<->(regional network)<-> (DOE non-Backbone Site network) 2.2 DOE Backbone Site network and DOE non-Backbone Site network in different regional networks; 2.2.1 ESNET peer-connected to non-DOE Backbone Site network's regional network: (DOE Backbone Site network)<->(ESNET)<->(regional network)<-> (DOE non-Backbone Site network) 2.2.2 ESNET not peer-connected to regional network: (DOE Backbone Site network)<->(ESNET)<->(NSFNET)<-> (regional network)<-> (DOE non-Backbone Site network) 2.2.3 DOE non-Backbone Site network not a member of a regional network: (DOE Backbone Site network)<->(ESNET)<-> (non-Backbone Link)<-> (DOE non-Backbone Site network) 3. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORK AND NON-DOE REGIONAL NETWORK; 3.1 DOE Backbone Site network and regional site network within same regional: (DOE Backbone Site network)<->(regional network)<-> (regional site network) DRAFT Routing Spec 09/08/89 Page 37 3.2 DOE Backbone Site network and regional site network in different regional networks; 3.2.1 ESNET peer-connected with regional site network's regional network: (DOE Backbone Site network)<->(ESNET)<->(regional network)<-> (regional site network) 3.2.2 ESNET not peer-connected to regional site network's regional network: (DOE Backbone Site network)<->(ESNET)<->(NSFNET)<-> (regional network)<->(regional site network) 4. TRAFFIC BETWEEN TWO DOE NON-BACKBONE SITE NETWORKS; 4.1 DOE non-Backbone Site network within same regional network: (DOE non-Backbone Site network)<->(regional network)<-> (DOE non-Backbone Site network) 4.2 DOE non-Backbone Site network in different regional networks; 4.2.1 ESNET peer-connected with both DOE and non-DOE Network's regional network: (DOE non-Backbone Site network)<->(regional network)<-> (ESNET)<->(regional network)<-> (DOE non-Backbone Site network) 4.2.2 ESNET peer-connected to only one DOE non-Backbone Site network regional network: (DOE non-Backbone Site network)<->(regional network)<-> (ESNET)<->(NSFNET)<-> (regional network)<-> (DOE non-Backbone Site network) 4.4.3 ESNET peer-connected to neither DOE non-Backbone Site network's regional network: (DOE non-Backbone Site network)<->(regional network)<-> (NSFNET)<-> (regional network)<-> (DOE non-Backbone Site network) 4.4.4 Both DOE Non-Backbone Site Networks directly linked to ESNET: (DOE non-Backbone Site network)<->(non-Backbone Link)<-> (ESNET)<-> (non-Backbone Link)<-> (DOE non-Backbone Site network) DRAFT Routing Spec 09/08/89 Page 38 5. TRAFFIC BETWEEN DOE BACKBONE SITE NETWORK AND OTHER AGENCY SITE; 5.1 Cross-connect between Agency Backbone and ESNET: (DOE Backbone Site network)<->(ESNET)<-> (Agency Backbone)<->(Agency Site) 6. TRAFFIC BETWEEN DOE SITE NETWORKS NOT CONNECTED TO ESNET OR A REGIONAL NETWORK 6.1 To be determined on case-by-case basis