ESNET DECNET ROUTING Version 2.1 (draft 2) for the DOE Energy Sciences Network (ESnet) prepared by the DECnet Working Group (EDWG) June 7, 1991 DRAFT DECnet Routing 6/7/91 page i User's Guide To This Document This document has been written with the purpose of defining a DECnet Phase IV routing policy for the ESnet-DECnet, and detailing DECnet addressing and circuit cost assignments for implementation of that policy. The authors of the document, however, recognized that such an undertaking could not be limited to simply a definition of DECnet routing policies and a list of assigned addresses and circuit costs. In order to make the document complete, sections dealing with network management issues and local facility connection issues were included. It was also deemed necessary to include a section providing a basic tutorial on DECnet routing for those individuals not familiar with the ins-and-outs of DECnet routing. Additional appendices were added to provide a list of ESnet-DECnet contacts, the results of a DECnet routing simulator program illustrating DECnet routing for every ESnet site, and a technical discussion on appropriate selection of designated routers. As a result, the document has become quite large, and somewhat diverse in the specifics of what it covers. It is expected that there will be a diversity of readers of this document, with differing objectives in what they desire to achieve from it. The following can serve as a general guide for readers: 1. READERS UNFAMILIAR WITH DECNET ROUTING - Appendix H (page 43) provides a basic tutorial into DECnet routing and should be read first. 2. READERS CONCERNED WITH ADDRESSING/ROUTING WITHIN ESNET - Sections II (page 4) & III (page 8) contain ESnet DECnet addressing and circuit cost assignments, respectively; as well as addressing and circuit cost policies in general. Appendix C (page 24) contains the DECnet routing as seen at each ESnet site. 3. READERS CONCERNED WITH LOCAL ISSUES RELATED TO DECNET ROUTING/ ADDRESSING - Section V (page 18) deals with designated routers; Appendix G (page 42) is a technical discussion on designated routing in respect to system type. 4. READERS CONCERNED WITH NETWORK MANAGEMENT ISSUES - Section IV (page 14) describes the management structure for DECnet routing/addressing issues, and how one can interact with that management structure. 5. READERS JUST WANTING TO KNOW WHO TO CALL - Appendix E (page 35) contains a list of names, e-mail addresses, etc. of individuals directly involved in management and operation of the ESnet DECnet. 6. READERS WITH A STRONG HEART & LOTS OF SPARE TIME - Read it cover-to-cover; you'll be a better person for it. DRAFT DECnet Routing 6/7/91 page ii Contents page ------- Status ....................................... 1 Acknowledgements.............................. 1 I. Introduction.................................. 2 II. ESnet DECnet Addressing....................... 4 2.1 ESnet Address Constraints......... 4 2.2 ESnet Address Assignments......... 5 2.3 Site Responsibility Toward Tail Circuit Addresses............ 7 III. ESnet DECnet Circuit Costs.................... 8 3.1 Circuit Cost Symmetry............. 8 3.2 Circuit Cost Constraints.......... 8 3.3 ESnet DECnet Routing Objectives... 10 3.4 ESnet Circuit Cost Assignments.... 11 3.5 Supplemental Layered Protocol DECnet-over-IP Circuits....... 12 3.6 Tail Circuit & Internetwork Link Costs.................... 13 IV. Ongoing DECnet Management..................... 14 4.1 Management Structure & Responsibilities 14 4.2 Address & Circuit Cost Changes.... 15 4.2.1 Delegation Of Circuit Cost Assignments to ESnet Staff 16 4.2.2 Review by EDWG.............. 16 4.2.3 Implementation ............. 16 4.2.4 Resolution of Implementation Problems/Changes.......... 16 4.3 Network Topology Modification, & Additions.............. 17 4.3.1 Access to Routing Node Parameters & Routing Tables. 17 V. Designated Router Issues & Facility LANs...... 18 5.1 Designated Router Issues.......... 18 5.1.1 Designated Level-1 Router... 18 5.1.2 Designated Level-2 Router... 19 5.2 Facility LAN DECnet Environments.. 19 5.3 ROUTER PRIORITY Recommendations... 20 5.4 Level-2 DECnet Address Guidelines. 20 References.................................... 21 Appendix A. DOE Backbone Sites................ 22 Appendix B. Glossary.......................... 23 Appendix C. Routing between ESnet Site Pairs.. 24 Appendix D. "HEPnet/SPAN Network Circuit Cost Plan..................... 31 Appendix E. Directory of Contacts............. 35 Appendix F. Derivation of ESnet Backbone Circuit Costs................. 40 Appendix G. Procedure for Selecting Designated Router............. 42 Appendix H. DECnet Routing Concepts........... 43 DRAFT DECnet Routing 6/7/91 page 1 STATUS This document is a DRAFT of the DECnet routing guideline for ESnet. This document has been prepared by the ESnet DECnet Working Group (EDWG) for implementation within ESnet. The scope of this document is DECnet Phase IV routing within the ESnet-DECnet. The document is NOT intended as a guideline for DECnet routing in general; rather, it is written to specifically address DECnet routing issues within the ESnet network environment. The primary focus of the document will be directed toward routing and addressing across the ESnet backbone. The document will detail the AREA number assignments and associated DECnet circuit costs for the ESnet routers and connections. It will also describe the basis for those AREA number and circuit costs assignments, in order to enable additions or modifications to ESnet in a manner consistent with the addresses and costs already in place. The document will also include specifications for those facilities and networks that have DECnet connections to ESnet sites. This is necessary because of the "boundlessness" of Phase IV DECnet. While interconnected DECnet networks may be treated as administratively independent entities, they must be treated as a single DECnet network from an addressing and routing perspective. Thus, there is a need to specify a DECnet routing and addressing policy that encompasses external DECnet connections into ESnet sites, as well as the ESnet backbone. It is anticipated that this document will be constantly evolving. The current version of the document describes the DECnet addressing and circuit costing of the ESnet backbone in its initial implementation. Since the document is intended to accurately reflect the current DECnet routing implementation on ESnet, as DECnet addresses or circuit costs are added or modified, the document is to be similarly modified to reflect those changes. ACKNOWLEDGEMENTS This DECnet Routing document is based on the many years of experience in constructing, maintaining, operating, inter-operating and managing a very large global DECnet entity, the DECnet portion of the network infrastructure known as HEPnet. The DECnet portion of HEPnet served the High Energy Physics community prior to the ESnet. The DECnet portion of HEPnet, sometimes called PHYSnet, is now an integral part of the ESnet structure and will serve, when completed, the WAN needs of all of the DOE/OER associated programs. To those distributed management entities, distributed world-wide, who founded the DECnet network and kept it operating for many years to the benefit of the research programs that used it, we owe a debt of gratitude. The "hints and kinks" of the vagaries of DECnettery, routing and management included, are a part of this document. Primary authorship of this document is attributed to P. DeMar and C.E. Bemis, Jr. with substantial input from the EDWG membership. Additional acknowledgement is extended to R. Nitzan, who initiated the project of a DECnet routing document, and from who's original document certain segments of this document have been extracted. DRAFT DECnet Routing 6/7/91 page 2 I. INTRODUCTION The Energy Sciences Network (ESnet) is a nationwide backbone network for the Department of Energy, initially providing T1 service to programs funded by the Office of Energy Research. The five current programs are: 1). Applied Math Sciences 2). Basic Energy Sciences 3). Fusion Energy 4). Health and Environmental Research 5). High Energy and Nuclear Research ESnet is intended to provide connectivity between the major DOE laboratories, sub-contractors, and collaborators at various Universities and institutions. The ESnet backbone is installed and operated by the National Magnetic Fusion Energy Computer Center (NMFECC) located at the Lawrence Livermore National Laboratory. The backbone will have multiple protocol routing capabilities, with DECnet Phase IV, DOD-IP, and X.25 traffic initially supported. Nineteen major DOE funded facilities in the United States will be connected to the ESnet backbone. The following diagram reflects the topology of the ESnet T1 backbone. All links are full channel T1 except the DOE link and the overseas links to CERN and FRG. The T1 links to the non-DOE sites (NASA Ames Research Center/AMES & Univ. of Maryland/UMD) are to provide access to other major research networks, such as the Internet/NSFnet and the NASA networks, SPAN/NSN. CERN | | -------------------------------FNAL--------------------MIT / ----- PNL / \ / / / / ANL---------- / NYU LBL-----LLNL / / | \ NYC_/ / /| \ / ISU | \ |\__BNL SLAC / | \ / | \ | \ AMES | ----------------- / ----- | ---------- \ | \ | | | \____\PPPL----FRG \ | | | |\_UMD \ | | | DOE\| CIT | | ---ORNL---------------CEBAF \ | LANL | / \ \ | /\ | / \ UCLA | / \ | / \ \ | / UTA----SSC-----------FSU GAC------ In addition to the DOE backbone facilities, there are a number of other Universities and institutions which participate in DOE-funded energy research. These sites are authorized to use the ESnet backbone to facilitate their work in energy research. Access to ESnet is usually via a leased line to an ESnet backbone site. DRAFT DECnet Routing 6/7/91 page 3 The focus of this document is on ESnet DECnet routing issues. The projected AREA number assignments and circuit costs for the ESnet backbone (which defines the routing and traffic flow) will be outlined. However, the ESnet backbone will be providing a higher level of DECnet service for an existing, operating, production, wide-area DECnet network (called the ESnet-DECnet), which presently serves DOE-funded energy research. In light of the existing network that is currently supporting most of the ESnet backbone sites and is already using coordinated DECnet addresses, ESnet backbone addressing will be consistent with that existing address structure. Additionally, the ESnet-DECnet is itself only a subset of a much larger DECnet network that transcends program, agency, and national boundaries. Agreements on AREA number usage and circuit costs are in place between the DECnet networks that make up this "multi-network" DECnet. All ESnet address and circuit cost assignments will be consistent with those agreements. Thus, this document will cover ESnet backbone DECnet address and circuit cost assignments in the context of consistency with existing DECnet address and circuit cost usage, both within the ESnet community, as well as within the larger global DECnet community. This document also outlines a proposed management structure for the ESnet-DECnet that will allow for orderly planning, implementation and operation in-so-far as these aspects relate to the primary topic of this document, that is DECnet routing and addressing. DRAFT DECnet Routing 6/7/91 II. ESNET DECNET ADDRESSING page 4 2.1 ESnet Address Constraints: While DECnet address space allows for use of AREA numbers 1-63 (each capable of supporting 1023 nodes), the selection of AREA/node numbers for ESnet backbone routers is extremely constrained by the fact that most ESnet sites are already part of the ESnet-DECnet network (HEPnet). Thus, most ESnet backbone sites already have their local systems within the coordinated DECnet address space (AREAs 1-46) that the ESnet-DECnet shares with the NASA Space Physics Analysis Network (SPAN), the European HEPnet network, the European SPAN network, Japanese DECnet networks, and other national and international entities. The following factors mandate continued use of those existing addresses (ie. AREAs): a. Many facilities have a programmatic need for DECnet communications with other entities within this DECnet. HEP requirements for access to European & Japanese DECnets would be a primary example. A site with access requirements to NASA's Space Physics Analysis Network might be another. In order to maintain their present level of connectivity with those other facilities, it is necessary to remain in the coordinated address space to which those entities belong. b. Facilities that are presently using a given DECnet address space locally, have a very strong bias toward continuing that use. Address changes on a facility scale are exceptionally disruptive, both in terms of the coordination necessary to implement the change and in terms of the inevitable access problems while the rest of the network "catches up" to the address changes. c. There are no additional AREA numbers presently available to the ESnet-DECnet within the existing coordinated address space. All AREA numbers have been either allocated, reserved for local use, or set aside to facilitate the transition to DECnet Phase V. NOTE: "Reserved for local use" refers to DECnet areas 47-63, which are available for facilities to use locally, but which will have no wide-area access capability. WAN DECnet backbones limit the exchange of routing information to DECnet AREAs 1-46 via the routing node executor parameter, MAX AREA, which is set to 46. As a consequence, local sites are free to use DECnet AREAs 47-63 for local needs without conflict with another local site using AREAs 47-63. d. Coordinated addressing results from an agreement in the "global" DECnet Internet*, agreements that preserve and conserve DECnet address space in DECnet Phase-IV. Conservation is required because of the limited number of unique addresses (64449 total). The agreement reserves DECnet AREAs 1-46 inclusive, for WAN addresses in the coordinated address space. * In this paper, the term "Global DECnet Internet" refers to the collection of DECnet networks (ESnet-DECnet, SPAN, European HEPnet, European SPAN, etc.) that share a common DECnet address space due to interconnection. DRAFT DECnet Routing 6/7/91 page 5 2.2 ESnet Address Assignment: Given the constraint of remaining consistent with the existing address structure that the ESnet-DECnet shares with SPAN, European HEPnet, European SPAN, etc., AREA number assignment for ESnet backbone sites becomes rather straightforward. Most sites are already utilizing an appropriate AREA number, and thus need do little more than bring up their ESnet backbone router within their own AREA. These sites which aren't presently a part of the ESnet-DECnet address space will adopt the AREA number of an adjacent ESnet backbone site facility. Utilization of adjacent facility's AREA number is mandated by the requirement for all Level-1 nodes to be contiguous. Since each ESnet backbone site has multiple site adjacencies, the choice of which AREA number to adopt for sites not currently using coordinated AREA numbers will be primarily dictated by the desirability to minimize the "diameter" of all DECnet AREAs. It should be noted that several ESnet sites (FNAL, GAC, & SLAC) will utilize a different area number for their ESnet backbone router than is utilized for their local systems and tail circuit support. The ESnet backbone routers at those sites have been assigned a different area number in order to facilitate DECnet routing and reduce vulnerability to area partitioning within the ESnet backbone. AREA number assignments for ESnet sites are as follows: BACKBONE AREA SITE ASSIGNMENT ANL 46 BNL 43 CEBAF 43 CIT 41 DOE 43 FNAL 46/42 * FSU 45 GAC 41/27 * ISU 46 LANL 1 LBL 41 LLNL 41 MIT 43 NYC 43 NYU 43 ORNL 45 PNL 36 PPPL 43 SLAC 41/44 * SSC 45 UCLA 41 UTA 25 * The first area number is the assignment for the ESnet backbone router, while the second area number is the one utilized by local systems and local routers supporting tail circuits. DRAFT DECnet Routing 6/7/91 page 6 The DECnet AREA address structure for the ESnet backbone sites is depicted below. (Dots outline AREAs and dashed lines indicate T1 links; AREAs are denoted as Axx) ..................... ................. : A46: : A43: -----------------------------FNAL---------------------MIT : / ............. : /| : : | : / : A36: : / | : : | : / : PNL : : / | : : / : ....../........:../........: : / | : : / : : / / : : / ANL------------\: / NYU : : LBL--------LLNL : : / / | : \ NYC__/ : : | ....../|\ : : | ISU | : :\ |\ : : | :A24 /:| \ : : | | : : \ | \__BNL : : | : AMES:| .\........: :.|......|..........: : \ | : : | :.....:| : \ | | : \ | : : SLAC | : ---------------|------|-------------------PPPL : : \ A41 | : | | : | : : \ | : .......... ..|......|........... : DOE | : : CIT | : :A1 : : | | A45: : \| : : \ | : : LANL : : | --ORNL------------------CEBAF : : \ | : : /\ : : | / \ : :...............: : UCLA | : :../..\..: : | / \ : : \ | : / : UTA-----SSC------FSU : : \ | : / : : :...................: : \| : / :A25 : : GAC----- :....: :................: DRAFT DECnet Routing 6/7/91 page 7 2.3 Site Responsibility Toward Tail Circuit Addresses and Propagation of Routing Information Each ESnet backbone site, in utilizing the coordinated address space of the U.S. & European HEP/SPAN DECnet network, becomes responsible for ensuring that all its own connections conform to that coordinated address space. Backbone sites must ensure that none of their local DECnet connections, or their remote DECnet connections, introduce routing information into the ESnet backbone that isn't part of the ESnet-DECnet/SPAN coordinated address space. This means that an ESnet backbone site must NOT pass: a. Routing information on DECnet AREAs 1-46, unless those AREAs have been approved by the HEP/SPAN Technical Coordination Group (see Section 3.6) for a specific network's use. This information must not be transmitted to the ESnet backbone router. That is, the backbone router itself will not be configured to filter out that routing information. b. Routing information on DECnet AREAs 47-63. ESnet backbone routers will have their MAXIMUM AREA parameter set to 46. This also applies to routers that support special circuits used for backup purposes, such as leased lines and layered DECnet-over-DOD_IP circuits (see Section 3.5), that may be established on an ESnet site local network. c. Routing information on nodes with addresses that have not been allocated by the AREA manager (Section 4.1) for the DECnet AREA utilized by that ESnet router. As in part 'a', this routing information is not to be transmitted to the ESnet backbone router; if present, that routing information must be filtered out by the facility's own routers. In the event it is determined that unapproved DECnet routing information is introduced into the ESnet backbone, the site responsible for introducing that routing information will take immediate steps to eliminate it. Normally, this would involve turning off the tail circuit that is responsible for introducing the inappropriate routing information. In the event that this is not feasible (or the routing information is introduced by systems on the local LAN), the backbone site will then contact ESnet operations to have the DECnet circuit connecting the ESnet backbone to the facility LAN turned off. In either case, the DECnet circuit will remain turned off until it can be turned on without propagating the inappropriate routing information. DRAFT DECnet Routing 6/7/91 page 8 III. ESNET DECNET CIRCUIT COSTS 3.1 Circuit Cost Symmetry The DECnet circuit is a system construct; it is not a "network" construct. That is, each system maintains its own DECnet circuit information for a given communications link. Thus, the assignment of DECnet circuit costs is at the system level. There is no technical reason that two systems with a direct DECnet connection between each other can not assign different DECnet circuit costs to that link. However, such "mismatched" assignment of circuit costs to the same physical connection enormously complicates routing management, from a network perspective. Therefore, circuit costs on the ESnet backbone links will be assigned equal value in opposite directions. When a circuit cost assignment is made, it implies assignment of that value by both systems supporting that particular physical connection. 3.2 Circuit Cost Constraints The assignment of circuit costs within the ESnet-DECnet must also be made in the context of its position as a part of a much larger and more complex DECnet environment. There are several aspects of this complex network environment that must be considered in any ESnet scheme for assigning circuit costs. These aspects are as follows: a. There is an existing circuit cost agreement with the other members of the global DECnet Internet. That agreement is for all interconnected networks to utilize a common circuit cost structure. The circuit cost structure is based on the concept of classifying network links into one of three catagories, then assigning circuit costs accordingly. The circuit types are listed below, followed by an illustration depicting the relationship between the three types: 1. Network backbone links - those links installed to support traffic on a network-wide basis; 2. Tail circuits - those links installed to support a specific site's traffic into and out of a network backbone; 3. Internetwork links - those links specifically installed to support traffic between two different networks. (Internetwork links may be multihop, such as in the UMD link). ------------ ESnet ----------- (tail circuit) | ESnet Site |##############| ESnet Site| . . . . ------------ backbone ----------- . Inter- | link . ------------ network --> | | University | link | . ------------ ----------- SPAN ----------- . | SPAN RC |###############| SPAN RC | . . . . ----------- backbone link ----------- (tail circuit) DRAFT DECnet Routing 6/7/91 page 9 (3.2 Circuit Cost Constraints, cont'd) In the illustration on the previous page, the backbone links have a clearly defined function to transport traffic between backbone sites. The internetwork link has the function to transport traffic between network backbones. The tail circuits purpose is to transport traffic between a tail site (ie., university) and a network backbone. The tail circuit is specifically NOT intended to transport traffic between two network backbones (ie., through the university). Since network backbone links are installed with the specific purpose of supporting network-wide traffic, circuit costs across an identifiable network backbone are to be kept as low as possible, with a value of '1' as the default. A circuit cost "range" of 14-17 is specified for internetwork links (ie., between network backbones). The ESnet backbone and internetwork link circuit cost assignments need to conform to this circuit cost structure if the existing internetwork traffic flow is to be maintained. (See Reference 8; also included as Appendix D, "HEPnet/SPAN Network Circuit Cost Plan"; adopted by HEPnet and SPAN, agreed to in principle by European HEP & European SPAN). b. There are also a large number of indirect links between WAN DECnet networks, called backdoor links. These links are the result of facilities (universities) installing tail circuits to two different DECnet network backbones. An example of a potential backdoor connection would be the two university tail circuits to ESnet and SPAN in the illustration on the previous page. Such links are intended to only support traffic between the local facility and the DECnet networks that its leased lines connect to. However, improper circuit cost assignments may result in general internetwork traffic traversing these tail circuit lines. In order to discourage internetwork traffic from traversing these links (as well as the facility's local campus network), maximum values for circuit costs (ie. 25) need to be assigned to these tail circuits. c. The AREA number assignments for the ESnet backbone leave the potential for AREA partitioning at some backbone sites, should there be a link failure. In the current T1 topology, and with the constraints of the ESnet-DECnet addressing scheme, there are a number of possibilities for DECnet AREA partitioning problems, with varying degrees of impact. One means of "reducing" the AREA partition potential is to define and install DECnet-over-DOD_IP circuits as backup DECnet circuits. In the event of a circuit failure that would normally cause an AREA partition, the backup DECnet-over-DOD_IP circuit keeps the AREA contiguous. However, the layering of DECnet over DOD_IP significantly reduces the efficiency of the DECnet circuit. Thus, the ESnet circuit cost structure needs to allow for DECnet-over-DOD_IP circuits as backup links for prevention of AREA partitioning, and in addition, the cost structure needs to ensure that such links are not utilized unless there is a link failure on the backbone. DECnet-over-DOD_IP links, and the capability to properly initiate and define them, are a part of the MultiNet TCP/IP software package for VMS hosts. Some background information on "layered" DECnet-over-DOD_IP circuits may be found in Ref. 9. DRAFT DECnet Routing 6/7/91 page 10 3.3 ESnet DECnet Routing Objectives: Ideally, routing would be structured to minimize hops while distributing the traffic load. Unfortunately, given the nature of DECnet routing (least cost path to an AREA), coupled with necessity of "sharing" AREAs among multiple backbone sites, it is not possible for all sites to have "optimal" routing. Decisions must be made as to how traffic will flow between AREAs, given the potential for different paths. Therefore, routing objectives are required to provide direction in the assignment of circuit costs where different degrees of "sub-optimal" routing for different sites are going to be the result. Objectives in assigning circuit costs, in order of priority, should be: a. Keep intra-agency/inter-site DECnet traffic on the backbone; b. Route inter-agency & international traffic across designated circuits; c. Insure tail sites are NOT used as transit paths; d. Maintain conformity with existing Internetwork circuit cost agreements; e. Initially preserve the SLAC-LBL-FNAL-BNL backbone structure of the existing ESnet DECnet. The largest number of ESnet-DECnet tail circuits currently radiate from these sites, thus direct routing should be preserved from an optimization perspective; f. Minimize the number of hops between sites where possible; g. Minimize asymmetric routing where possible; Note: Objectives 'f' and 'g' are considered to be of equivalent priority. Decisions on assignment of circuit costs that may meet one objective ('f' or 'g') at the expense of the other will be done on a case-by-case basis. DRAFT DECnet Routing 6/7/91 page 11 3.4 ESnet Circuit Cost Assignments Given the ESnet DECnet Routing objectives, together with the constraints in assigning circuit costs, the designation of circuit costs for the ESnet backbone then becomes a somewhat subjective assignment of circuit cost values (values range from 1-25) that best appears to meet those objectives and constraints. In accord with routing objective 'a', the lowest values that seem appropriate are assigned to the ESnet backbone. However, in order to meet the routing objectives 'b', 'c', and 'd', certain ESnet backbone links must be assigned higher circuit cost values than the minimal cost generically assigned to backbone circuits. The current ESnet backbone circuit cost assignments are depicted in the diagram given below. Circuit costs are symmetric and must be defined on both ends of a given T1 link in order to achieve the ESnet-DECnet routing objectives. Appendix F (page 40) contains a description of the derivation and justification of the initial circuit cost assignments. Some of the initial values were later modified based upon operational experience. ..................... ................. 1 : A46: : A43: -----------------------------FNAL---------------------MIT : / ............. : /| : 1 : | : / : A36: : / | : : | 2 : / : PNL : : 1 / |1 : : / : ....../........:../........: : / | : 2 : / : : / 1 /25 : : / ANL------------\: / NYU : : LBL--------LLNL : : / 25/ | : \ NYC__/25 : : | ....14/|\ : : | ISU | : :\ |\ : : 1| :A24 /:| \ : : | | : : \ | \1_BNL : : | : AMES:| .\........: :.|......|..........: : \ |2 : : | :.....:| : \ | | 2 : \ | : : SLAC | : ---------------|------|-------------------PPPL : : \ A41 | : | | 1 : | : : 1 \ 1 | : .......... ..|......|........... : DOE | : : CIT | : :A1 : : | | A45: : 25\| : : \ | : : LANL : : | --ORNL------------------CEBAF : : 1 \ | : : /\3 : : | /1 \ : 1 :...............: : UCLA | : :../..\..: : | / 1 \ : : \ | : / : UTA-----SSC------FSU : : 1 \ | : /4 : : 1: 1 : : \| : / :A25 : :...................: : GAC----- :....: :................: DRAFT DECnet Routing 6/7/91 page 12 3.5 Supplemental Layered Protocol DECnet-over-IP Circuits for ESnet Backbone Backup The AREA number and circuit cost assignments are made under the assumption that all ESnet backbone T1 circuits are up and functioning. There is, however, some level of vulnerability to functional DECnet routing in the event that certain circuits fail. It is proposed that layered protocol DECnet circuits (DECnet-over-IP) using the MultiNet software package be installed between a select number of sites to minimize the potential for routing problems due to link failure (Ref.9). Multinet circuits would be assigned high (22) circuit cost values as a default, since they are only intended as a fail-over mechanism. The value 22 is selected instead of 25 to allow flexibility in dealing with the lower speed connections between ESnet backbone sites that may remain in place for testing purposes and for DECnet Phase-V migration purposes. a. The greatest vulnerability is for AREA fragmentation (Section 2.2). AREA fragmentation occurs when a link outage fragments a single AREA into two separate AREAs, both obviously using the same AREA number. AREA fragmentation by a single line outage can be avoided by insuring that every ESnet backbone site has an alternate path to the other nodes within its own AREA. To achieve this objective, the following layered DECnet-over-IP MultiNet circuit, where the underlying IP network structure is that of the ESnet, is required: AREA 43: MIT-CEBAF Other layered DECnet circuits may also be of value and will be considered on a case-by-case basis. b. To a lesser extent, there is a potential problem should there be a loss of both links between the West Coast sites and the East Coast/Midwest sites. The higher circuit costs associated with the Southern tier sites could effectively result in traffic between West Coast and East Coast AREAs traversing the internetwork DECnet links and SPAN, rather than via the Southern tier T1s (which are of higher cost). A single inter-AREA Multinet DECnet link between LLNL and PPPL with a cost of 22 would keep the DECnet traffic on the Southern tier T1s (albeit enveloped inside TCPIP packets). c. The use of layered protocol DECnet-over-IP circuits with the MultiNet software package requires careful attention to detail when authorized circuits are "inserted" into the ESnet-DECnet structure. For these circuits to function properly, they must be brought up on VMS routers on the local sites network. Router parameters, such as the executor MAX AREA parameter, must be in conformance with the overall ESnet DECnet backbone environment to preclude the introduction of erroneous routing information into the network. Only approved DECnet-over-IP circuits, under the coordination of ESnet management are allowed in the ESnet network structure. ESnet facilities are individually responsible for ensuring that DECnet-over-IP circuits are not installed without prior coordination and approval. DRAFT DECnet Routing 6/7/91 page 13 3.6 Tail Circuit & Internetwork Link Costs The term 'tail circuit' refers to any DECnet circuit installed to a remote site solely to support DECnet access for that remote site. It is independent of the underlying hardware and software used to support the circuit. Existing DECnet tail circuits and any future DECnet tail circuits are to be set to a default circuit cost of 25. If there are specific topological considerations that might be invoked to justify a lower value, then approval for implementing a lower circuit cost is required from the AREA manager for that particular AREA. If the circuit crosses an AREA boundary, then both AREA managers involved must approve the lower cost. It is not expected that a tail circuit cost would ever be less than 22. Assignment of a tail circuit cost value less than 22 will require approval of the ESnet management authority for DECnet addressing and routing issues. Similarly, internetwork links have a default cost range of 14-17. However, such links require joint coordination with the other network entity supporting the link. Therefore, no DECnet internetwork circuit costs will be assigned in context of this document. Circuit cost assignments will be left to the DECnet management authorities of the respective networks (EDWG in the case of ESnet), and are subject to approval of the ESnet-DECnet/SPAN Technical Coordination Group. (the official name of this coordination group is The HEP/SPAN Technical Coordination Group, HSTCG, where HEP refers to a network affiliation now incorporated in the ESnet-DECnet) DRAFT DECnet Routing 6/7/91 page 14 IV. Ongoing DECnet Routing/Addressing Management 4.1 DECnet Management Responsibilities (In context of ESnet-DECnet routing) There are three management entities that have differing areas of responsibility for DECnet routing within the ESnet-DECnet environment: a. The ESnet implementers, the NMFECC staff at LLNL, have overall responsibility for the implementation and management of the ESnet backbone network structure and associated hardware. This includes responsibility for maintaining DECnet support on the backbone routers. The ESnet staff also monitor the backbone routers to ensure that current DECnet routing is the expected DECnet routing. b. The ESnet DECnet Working Group (EDWG) has the charter to "provide planning, documentation, review, and site-level management for DECnet issues, including addressing, performance, security, and Phase V transition planning". Within the boundaries of this charter, the EDWG has the responsibility for circuit cost and address assignments within the ESnet-DECnet. Additionally, the EDWG has the responsibility to coordinate with other DECnet networks on those DECnet issues that require inter-network coordination. AREA number usage and internetwork circuit cost agreements are primary examples. c. Area managers are designated individuals to which the EDWG has delegated the authority to handle the day-to-day issues involving individual DECnet AREAs. An Area manager allocates and manages DECnet addresses within the AREA he is responsible for, as well as stipulating circuit costs for tail circuits that utilize those AREA/node numbers at ESnet sites. Area managers monitor tail site connections within their area for conformance to prescribed ESnet-DECnet address and circuit cost assignment. DRAFT DECnet Routing 6/7/91 page 15 (4.1 DECnet Management Structure,--- cont'd) The interaction between these three entities ('a', 'b', and 'c') on DECnet routing and addressing issues occur at two different levels: 1. Circuit cost strategies and area number assignments are discussed, reviewed, and adopted by the EDWG, in the course of its regular meetings. The ESnet staff participates in this activity through its representative on the EDWG. Similarly, most AREA managers are also EDWG representatives. Those that are not EDWG representatives will be advised of EDWG routing and addressing directives by the EDWG Chairman. 2. Responsibility for routing and addressing issues that arise in the course of normal day-to-day operations are normally handled by the ESnet staff and the area managers. Such day-to-day issues are to be handled in accordance with established EDWG policies and this DECnet Routing document. The EDWG will designate one member to serve as a day-to-day contact with the ESnet staff and area managers, in the event that issues extending beyond (or requiring deviation from) the contents of this routing document should arise. Area managers and ESnet staff are expected to confer with and receive specific approval from that individual for address or circuit cost assignments that deviate from the specifications of this document. While no formal mechanism for interaction between area managers and the ESnet staff is specified, informal channels of communication are encouraged for those day-to-day operational issues which might arise. 4.2 DECnet Address & Circuit Cost Assignment Changes There is also a need to provide a mechanism for modifying the circuit cost and AREA number assignments specified in this document. Reasons for initiating such changes include (but are not restricted to): 1. Changes and modifications in Internetwork agreements that impact AREA number allocation or circuit cost strategies; 2. Refinement of existing routing structure to accommodate specific requests by individual ESnet sites (site 'A' desires a different default path to site 'B', which can be accommodated without negatively impacting other sites); 3. More desirable DECnet routing based on long term traffic utilization statistics. DRAFT DECnet Routing 6/7/91 page 16 (4.2 DECnet Management Structure,--- cont'd) 4.2.1 Delegation Of Circuit Cost Assignments to ESnet Staff The EDWG delegates responsibility for the ESnet backbone and internetwork circuit cost adjustments to the ESnet staff. The ESnet staff will ensure that any changes in backbone or internetwork circuit costs will remain consistent with the circuit cost architecture specified in this document, and in conformance to internetwork agreements also described herein. The ESnet staff will notify the EDWG of any circuit cost assignment changes, and will update this document to reflect those changes. 4.2.2 Review by the EDWG The EDWG will regularly review DECnet circuit cost assignments within the ESnet-DECnet. Additionally, the EDWG will review suggestions for circuit cost modifications from members of the ESnet community at large. Changes or modifications which are adopted will be passed on to the appropriate AREA manager (or ESnet staff, in the case of the ESnet backbone) for implementation; and notification of the change will be passed on to the ESCC. 4.2.3 Implementation AREA managers, when notified of approved changes will assume responsibility for seeing that the changes are implemented in a timely and non-disruptive manner. Note that an AREA manager will not have operational control over all DECnet systems within his AREA; however, he shall coordinate such changes, and report problems in getting changes implemented on systems not directly under his control. In the case of changes affecting the ESnet backbone, the ESnet staff will assume responsibility for implementing those changes in a timely and nondisruptive manner. 4.2.4 Resolution of Implementation Problems/Changes In the event that there is an apparently unresolvable problem within the EDWG in arriving at a consensus on a proposed address or circuit cost change, the issue will be brought to the attention of the ESnet Site Coordinator Committee (ESCC) Chairman. Similarly, in the event that there is a problem with the implementation of an approved change, the matter will be brought to the attention of the ESCC Chairman. In the event that the EDWG declines to approve, or fails to review in a timely fashion, a request for a change in AREA number usage or circuit cost assignment, the initiator of that request may petition the ESCC Chairman for review by the ESCC. DRAFT DECnet Routing 6/7/91 page 17 4.3 Network Topology Modification, Additions, and Management Responsibilities New circuits and internetwork links, foreign connects and the like, may be implemented from time to time as programmatic needs require them and as financial constraints allow them to be implemented. To meet the goals, objectives and routing structure outlined in this document, the DECnet issues shall be coordinated with the EDWG when changes are contemplated or planned that will affect the topology of ESnet, or its connectivity to other DECnet networks. Specifically, the changes are: a. When new circuits/links are added to the ESnet structure; b. When new international links are contemplated. 4.3.1 Access to Routing Node Parameters & Routing Tables Network management functions to insure compliance with this routing document require "read" access to ALL ESnet-DECnet routing nodes. This is not only to ensure that routing nodes within the network can be monitored for compliance with the specifications of this document, but also to facilitate trouble-shooting when problems do occur. "Read" access will mean that routing nodes are to allow general network access to their routing parameters and current routing tables. A router will be considered compliant with providing "read" access if it meets one of the following conditions: 1. Supports the DEC protocol Network Management Listener (NML) and allows open network access to the NML DECnet object. NML is supported on DEC's own router products. 2. Provides an account within which a user can logon and examine DECnet routing parameters and tables. The username/password combination for that account is to made available to valid ESnet users at their request. Multiprotocol routers, such as the ESnet backbone routers, provide this type of access via TCPIP Telnet. DRAFT DECnet Routing 6/7/91 page 18 V. Designated Router Issues & Facility LANs The incorporation of an ESnet supplied dual-protocol router into a local sites network topology is worthy of some considerable contemplation. Proper attention must be given to local topology, local routing issues, both for the DECnet and the DOD_IP suites, and local connectivity such as extended ethernet LAN issues. The various ESnet backbone site LAN issues are different, no two sites are identical, and no one single solution is possible for all ESnet backbone sites. Some of the aspects to be considered for proper local DECnet routing in the presence of the ESnet WAN router are presented in this Section. The DECnet Working Group, the EDWG, if asked, will provide consultation and assistance in addressing the various aspects given below. 5.1 Designated Router Issues: The installation of any DECnet router at a facility must be done with a careful understanding of the possible effects that the DECnet router may have on the local DECnet environment at that facility. The major concern is the effect that the router may have on DECnet routing within the local LAN environment. Any router can affect the local LAN routing in two ways: a. First, as a Level-1 routing node, the router has the potential for becoming the designated router for all the non-routing nodes in the same AREA on its ethernet. Great care should be exercised in determining which router serves the designated router function for an ethernet; b. Second, as a Level-2 (AREA) routing node, the router may become the designated Level-2 router for all of the local Level-1 routing nodes in the same DECnet AREA on its ethernet. Generally, the Level-2 router bearing the largest amount of inter-AREA traffic should serve as the designated Level-2 router for the LAN. 5.1.1 Designated Level-1, Intra-AREA, Router: As noted in section 2.2, there are three types of DECnet nodes. The non-routing nodes maintain no routing information, relying on a routing node to perform this function for them. All non-routing nodes on the same ethernet depend on the same intra-AREA Level-1 routing node for routing. This node is referred to as the designated router for the ethernet for that particular DECnet AREA. The designation of a specific node as the designated router is accomplished using the ROUTER PRIORITY parameter assigned to the ethernet interface circuit on each router. The ROUTER PRIORITY has a valid range of 0-127, with a default of 64. All routing nodes (Level-1 & Level-2 routers) send out router hello packets which advertise their router priority. Non-routing nodes then select the intra-AREA router with the highest ROUTER PRIORITY value as their designated router. When two or more routers "tie" for highest ROUTER PRIORITY, the router with the highest DECnet address is selected. If the router with the highest ROUTER PRIORITY goes down, its absence is noted by the lack of hello packets from that router, and the non-routing nodes will select the highest ROUTER PRIORITY from the remaining routers which are still communicating on the ethernet. DRAFT DECnet Routing 6/7/91 page 19 5.1.2 Designated Level-2 Router: Intra-AREA Level-1 routing nodes must similarly select an intra-AREA Level-2 routing node through which to route traffic bound for other AREAs. This is simply done by the selection of the "closest" Level-2 router, in terms of circuit costs. In cases where multiple Level-2 routers are of equal cost away, the Level-2 router with the highest DECnet address is selected. On an ethernet LAN, all other nodes are always of equal cost from any specific node. Thus, all ethernet-adjacent Level-2 routers will be of equal cost from any Level-1 router on an ethernet LAN. This makes the selection of DECnet address (and which Level-2 router is assigned the highest DECnet address) very significant. Note that the ROUTER PRIORITY parameter is not a factor in the selection of the designated Level-2 router. However, since Level-2 routers also perform the Level-1 routing function, a Level-2 router with the highest ROUTER PRIORITY parameter will become the designated router for all the non-routing nodes on the ethernet. As the designated router, it will perform both Level-1 and Level-2 routing for the non-routing nodes. Thus, if a Level-2 router has the highest ROUTER PRIORITY value ( a Level-1 function) and, it has a lower DECnet address than other Level-2 routers, non-routing nodes will utilize one Level-2 router for inter-AREA routing and Level-1 routing nodes will use another. Generally, this is not advisable. 5.2 Facility LAN DECnet Environments: Generally, there are two basic DECnet environments that are found within ESnet: a. facilities that only utilize DECnet AREAs within the AREA number range 1-46, b. and facilities that utilize DECnet AREA numbers in the range 1-46, AND, in the range 47-63. These latter AREAs are termed "high number" AREAs. As noted in section 2.1 (d), the high number AREAs are available for local use by a facility. Systems within high number AREAs do not have any direct access with the wide-AREA DECnet, due to the MAXIMUM AREA parameter value of 46 assigned to all backbone routers; however, they do maintain full access to all systems on their own site, including those using addresses in AREAs 1-46. (See reference 7). The absence or existence of high-number AREAs will impact the appropriate selection of the proper DECnet address for the ESnet backbone router at a particular site. DRAFT DECnet Routing 6/7/91 page 20 5.3 ROUTER PRIORITY Recommendations: The present recommendation for ALL ESnet facilities, regardless of whether high-number AREAs are present, is to assign the ESnet backbone router a low ROUTER PRIORITY so that it will not become the designated Level-1 router at that local site. The source for concern is the potential performance effect on the router's synchronous traffic by CPU interrupts for local ethernet routing to the detriment of WAN DECnet routing on the ESnet backbone. The assignment of a low ROUTER PRIORITY to the ESnet router also avoids the potential problem noted above where non-routing nodes use the ESnet backbone router for inter-AREA traffic while Level-1 routing nodes use another Level-2 router. As a general rule, Level-2 routers should be assigned low ROUTER PRIORITY values, but this is only a recommendation; backbone facilities do have the flexibility to assign that parameter as they deem appropriate. 5.4 Level-2 DECnet Address Guidelines: a. Facilities without high-number AREAs: Facilities without on-site high-number AREAs are to assign a higher DECnet address to the ESnet backbone router than are presently utilized by any other Level-2 router on that particular ethernet. It is presumed that the largest amount of off-site inter-AREA traffic will be routed via the ESnet backbone. Assignment of the highest DECnet address among Level-2 routers to the ESnet backbone router will minimize the number of hops for that traffic. b. Facilities with high-number AREAs: Facilities with on-site high-number AREAs MUST assign a lower DECnet address for the ESnet router than for other Level-2 routers on that same ethernet (and within that same AREA). The ESnet backbone router MUST have a MAXIMUM AREA parameter of 46 (Section 2.1 (d)). In order for the local facility's low-number and high-number AREAs to communicate, their inter-AREA traffic must utilize other Level-2 routers, since the ESnet backbone router will not pass traffic to high number AREAs by virtue of its MAXIMUM AREA parameter. This will add an extra hop for traffic bound out of the site for the ESnet backbone, but that is a necessary trade-off in order to maintain local connectivity. DRAFT DECnet Routing 6/7/91 page 21 REFERENCES 1. DECnet Digital Network Architecture, Phase-IV, General Description, Order # AA-N149A-TC, May 1982, Digital Equipment Corporation, Maynard, MA 2. DECnet Digital Network Architecture, Phase-IV, Routing Layer Functional Specification, Version 2.0, Order # AA-X435A-TK, 1983, Digital Equipment Corporation, Maynard, MA 3. Guide to DECnet-VAX Networking, VMS Version 5.0 Order # AA-LA47A-TE, 1988 Digital Equipment Corporation, Maynard, MA 4. VMS Networking Manual, VMS Version 5.0 Order # AA-LA48A-TE, 1988 Digital Equipment Corporation, Maynard, MA 5. VMS Network Control Program Manual, VMS Version 5.0 Order # AA-LA50A-TE, 1988 Digital Equipment Corporation, Maynard, MA 6. The MAX AREA Parameter Filter Concept, by P. DeMar, Fermi National Accelerator Laboratory, July 1986, (unpublished, but available from author) 7. Managing AREA Numbers in Multi-Organizational DECnet Networks, P. DeMar, Fermi National Accelerator Laboratory, July 1987, (unpublished, but available from author) 8. HEPnet/SPAN Network Circuit Cost Plan, P. Demar, in Energy Research Management Workshop, Lawrence Berkeley Laboratory Report, LBL-25343, June 1988, pp. 333-337 9. The Efficacy and Efficiency of "Layered" DECnet-over-IP for Backup DECnet Circuits in ESnet, C. E. Bemis, Jr., October 12, 1989 (unpublished, but available from author) 10. Routing and Networking Overview, Order No. AA-HS15A-TK, December 1986 Digital Equipment Corporation, Maynard, MA DRAFT DECnet Routing 6/7/91 page 22 APPENDIX A DOE Backbone Sites, Inter-Agency & International Connection Sites ----------------------------------------------------------------- ANL Argonne National Laboratory BNL Brookhaven National Laboratory CEBAF Continuous Electron Beam Accelerator Facility CIT California Institute of Technology DOE DOE Office of Energy Research, Washington D.C. FNAL Fermi National Accelerator Laboratory FSU Florida State University (Supercomputer Computation Research Institute) GAC General Atomics Corporation ISU Iowa State University (Ames Laboratory) LANL Los Alamos National Laboratory LBL Lawrence Berkeley Laboratory LLNL Lawrence Livermore National Laboratory MIT Massachusetts Institute of Technology NYC New York City point of presence NYU New York University ORNL Oak Ridge National Laboratory PNL Pacific Northwestern Laboratory PPPL Princeton Plasma Physics Laboratory SLAC Stanford Linear Accelerator Center SSC Superconducting SuperCollider UCLA University of California at Los Angeles UTA University of Texas at Austin Inter-Agency Connection Sites AMES NASA AMES Research Center (FIX-WEST) UMD University of Maryland at College Park (FIX-EAST) International Connection Sites CERN Organisation Europeene pour la Recherche Nucleaire (European Organization for Nuclear Research) Geneva FRG Federal Republic of Germany (Garching) DRAFT DECnet Routing 6/7/91 APPENDIX B. GLOSSARY page 23 Adjacency: A concept in DECnet routing that indicates nodes located one logical DECnet hop away. Adjacencies for end-nodes are the designated router, and cached ethernet nodes; for Level-1 routers, all intra-AREA end nodes, all other intra-AREA Level-1 routers and the nearest intra-AREA Level-2 router; for Level-2 routers, all Level-2 routers in any AREA, all intra-AREA Level-1 routers and all intra-AREA end nodes. Adjacencies are one hop. Aggregate cost: The sum of the circuit cost values for all links along a given path between source and destination nodes. AREA: A logical grouping of DECnet addresses. DECnet address space is divided into 63 logical groupings (AREAs), each containing 1023 nodes. AREAs are designated by integer value. DECnet routing is based on hierarchical routing by DECnet AREA. Back Door Connection: Any connection between two administratively separate DECnet networks that occurs when both networks support tail circuits to the same facility. Internetwork traffic is not intended to traverse back door connections. Circuit cost: A routing metric that is used to determine appropriate routing paths. Circuit cost is a system value assigned to each DECnet circuit on a particular system. It is an integer value ranging between 1-25. Although a circuit cost is a system parameter, in context of this document, it implies assignment of the same value on both systems for their common link. Designated router: The router known to an end node, that is adjacent to it, and is able to perform the Level-1 routing function on its behalf. ESnet Backbone Site: A DOE funded facility designated to be directly connected to the ESnet backbone. ESnet Backbone Site Network: The local network infrastructure and topology at an ESnet backbone site. This includes one side of the ESnet supplied dual protocol Cisco router. To insure local functionality, the local network must be considered carefully and integrated with the DECnet, and DOD_IP, aspects of the ESnet router. Internetwork link: A connection installed with the purpose of supporting access between two administratively different wide-area DECnet networks. The ESnet and SPAN are examples of administratively different DECnets. Path: An available route that traffic may traverse between two nodes, which may or may not traverse through other intermediate nodes. DRAFT DECnet Routing 6/7/91 page 24 APPENDIX C Given below are the "routes" between ESnet site pairs that DECnet traffic would flow over. The routes are not always symmetrical and the return path should also be examined for completeness. The DECnet routing paths shown here result from the Circuit Costs and Addresses outlined in this document. The output is derived from the DECnet Trace Route (DNTR) Utility developed at LLNL. ames->anl :ames->llnl->lbl->fnal->anl ames->bnl :ames->llnl->pppl->nyc->bnl ames->cebaf :ames->llnl->pppl->cebaf ames->cit :ames->llnl->lbl->slac->cit ames->doe :ames->llnl->pppl->cebaf->doe ames->fnal :ames->llnl->lbl->fnal ames->fsu :ames->llnl->pppl->cebaf->ornl->fsu ames->gac :ames->llnl->gac ames->isu :ames->llnl->lbl->fnal->anl->isu ames->lanl :ames->llnl->gac->lanl ames->lbl :ames->llnl->lbl ames->llnl :ames->llnl ames->mit :ames->llnl->pppl->nyc->mit ames->ornl :ames->llnl->pppl->cebaf->ornl ames->pppl :ames->llnl->pppl ames->slac :ames->llnl->lbl->slac ames->ssc :ames->llnl->pppl->cebaf->ornl->ssc ames->ucla :ames->llnl->gac->ucla ames->uta :ames-> anl->ames :anl->fnal->lbl->llnl->ames anl->bnl :anl->fnal->mit->nyc->bnl anl->cebaf :anl->fnal->mit->nyc->pppl->cebaf anl->cit :anl->fnal->lbl->slac->cit anl->doe :anl->fnal->mit->nyc->pppl->cebaf->doe anl->fnal :anl->fnal anl->fsu :anl->fnal->ssc->fsu anl->gac :anl->fnal->lbl->llnl->gac anl->isu :anl->isu anl->lanl :anl->fnal->ssc->uta->lanl anl->lbl :anl->fnal->lbl anl->llnl :anl->fnal->lbl->llnl anl->mit :anl->fnal->mit anl->ornl :anl->fnal->ssc->ornl anl->pppl :anl->fnal->mit->nyc->pppl anl->slac :anl->fnal->lbl->slac anl->ssc :anl->fnal->ssc anl->ucla :anl->fnal->lbl->slac->cit->ucla anl->uta :anl->fnal->ssc->uta DRAFT DECnet Routing 6/7/91 page 25 bnl->ames :bnl->nyc->pppl->llnl->ames bnl->anl :bnl->nyc->mit->fnal->anl bnl->cebaf :bnl->nyc->pppl->cebaf bnl->cit :bnl->nyc->pppl->llnl->lbl->slac->cit bnl->doe :bnl->nyc->pppl->cebaf->doe bnl->fnal :bnl->nyc->mit->fnal bnl->fsu :bnl->nyc->pppl->cebaf->ornl->fsu bnl->gac :bnl->nyc->pppl->llnl->gac bnl->isu :bnl->nyc->mit->fnal->anl->isu bnl->lanl :bnl->nyc->pppl->llnl->gac->lanl bnl->lbl :bnl->nyc->pppl->llnl->lbl bnl->llnl :bnl->nyc->pppl->llnl bnl->mit :bnl->nyc->mit bnl->ornl :bnl->nyc->pppl->cebaf->ornl bnl->pppl :bnl->nyc->pppl bnl->slac :bnl->nyc->pppl->llnl->lbl->slac bnl->ssc :bnl->nyc->pppl->cebaf->ornl->ssc bnl->ucla :bnl->nyc->pppl->llnl->gac->ucla bnl->uta :bnl->nyc->mit->fnal->ssc->uta cebaf->ames :cebaf->pppl->llnl->ames cebaf->anl :cebaf->ornl->anl cebaf->bnl :cebaf->pppl->nyc->bnl cebaf->cit :cebaf->pppl->llnl->lbl->slac->cit cebaf->doe :cebaf->doe cebaf->fnal :cebaf->ornl->anl->fnal cebaf->fsu :cebaf->ornl->fsu cebaf->gac :cebaf->pppl->llnl->gac cebaf->isu :cebaf->ornl->anl->isu cebaf->lanl :cebaf->ornl->ssc->uta->lanl cebaf->lbl :cebaf->pppl->llnl->lbl cebaf->llnl :cebaf->pppl->llnl cebaf->mit :cebaf->pppl->nyc->mit cebaf->ornl :cebaf->ornl cebaf->pppl :cebaf->pppl cebaf->slac :cebaf->pppl->llnl->lbl->slac cebaf->ssc :cebaf->ornl->ssc cebaf->ucla :cebaf->pppl->llnl->gac->ucla cebaf->uta :cebaf->ornl->ssc->uta cit->ames :cit->ucla->gac->llnl->ames cit->anl :cit->slac->lbl->fnal->anl cit->bnl :cit->ucla->gac->llnl->pppl->nyc->bnl cit->cebaf :cit->ucla->gac->llnl->pppl->cebaf cit->doe :cit->ucla->gac->llnl->pppl->cebaf->doe cit->fnal :cit->slac->lbl->fnal cit->fsu :cit->slac->lbl->fnal->ssc->fsu cit->gac :cit->ucla->gac cit->isu :cit->slac->lbl->fnal->anl->isu cit->lanl :cit->ucla->gac->lanl cit->lbl :cit->slac->lbl cit->llnl :cit->ucla->gac->llnl cit->mit :cit->ucla->gac->llnl->pppl->nyc->mit cit->ornl :cit->slac->lbl->fnal->ssc->ornl cit->pppl :cit->ucla->gac->llnl->pppl cit->slac :cit->slac cit->ssc :cit->slac->lbl->fnal->ssc cit->ucla :cit->ucla cit->uta :cit->slac->lbl->fnal->ssc->uta DRAFT DECnet Routing 6/7/91 page 26 doe->ames :doe->cebaf->pppl->llnl->ames doe->anl :doe->cebaf->ornl->anl doe->bnl :doe->cebaf->pppl->nyc->bnl doe->cebaf :doe->cebaf doe->cit :doe->cebaf->pppl->llnl->lbl->slac->cit doe->fnal :doe->cebaf->ornl->anl->fnal doe->fsu :doe->cebaf->ornl->fsu doe->gac :doe->cebaf->pppl->llnl->gac doe->isu :doe->cebaf->ornl->anl->isu doe->lanl :doe->cebaf->ornl->ssc->uta->lanl doe->lbl :doe->cebaf->pppl->llnl->lbl doe->llnl :doe->cebaf->pppl->llnl doe->mit :doe->cebaf->pppl->nyc->mit doe->ornl :doe->cebaf->ornl doe->pppl :doe->cebaf->pppl doe->slac :doe->cebaf->pppl->llnl->lbl->slac doe->ssc :doe->cebaf->ornl->ssc doe->ucla :doe->cebaf->pppl->llnl->gac->ucla doe->uta :doe->cebaf->ornl->ssc->uta fnal->ames :fnal->lbl->llnl->ames fnal->anl :fnal->anl fnal->bnl :fnal->mit->nyc->bnl fnal->cebaf :fnal->mit->nyc->pppl->cebaf fnal->cit :fnal->lbl->slac->cit fnal->doe :fnal->mit->nyc->pppl->cebaf->doe fnal->fsu :fnal->ssc->fsu fnal->gac :fnal->lbl->llnl->gac fnal->isu :fnal->anl->isu fnal->lanl :fnal->ssc->uta->lanl fnal->lbl :fnal->lbl fnal->llnl :fnal->lbl->llnl fnal->mit :fnal->mit fnal->ornl :fnal->ssc->ornl fnal->pppl :fnal->mit->nyc->pppl fnal->slac :fnal->lbl->slac fnal->ssc :fnal->ssc fnal->ucla :fnal->lbl->slac->cit->ucla fnal->uta :fnal->ssc->uta fsu->ames :fsu->ornl->cebaf->pppl->llnl->ames fsu->anl :fsu->ssc->fnal->anl fsu->bnl :fsu->ornl->cebaf->pppl->nyc->bnl fsu->cebaf :fsu->ornl->cebaf fsu->cit :fsu->ssc->fnal->lbl->slac->cit fsu->doe :fsu->ornl->cebaf->doe fsu->fnal :fsu->ssc->fnal fsu->gac :fsu->ssc->fnal->lbl->llnl->gac fsu->isu :fsu->ssc->fnal->anl->isu fsu->lanl :fsu->ssc->uta->lanl fsu->lbl :fsu->ssc->fnal->lbl fsu->llnl :fsu->ssc->fnal->lbl->llnl fsu->mit :fsu->ornl->cebaf->pppl->nyc->mit fsu->ornl :fsu->ornl fsu->pppl :fsu->ornl->cebaf->pppl fsu->slac :fsu->ssc->fnal->lbl->slac fsu->ssc :fsu->ssc fsu->ucla :fsu->ssc->fnal->lbl->slac->cit->ucla fsu->uta :fsu->ssc->uta DRAFT DECnet Routing 6/7/91 page 27 gac->ames :gac->llnl->ames gac->anl :gac->llnl->lbl->fnal->anl gac->bnl :gac->llnl->pppl->nyc->bnl gac->cebaf :gac->llnl->pppl->cebaf gac->cit :gac->ucla->cit gac->doe :gac->llnl->pppl->cebaf->doe gac->fnal :gac->llnl->lbl->fnal gac->fsu :gac->llnl->pppl->cebaf->ornl->fsu gac->isu :gac->llnl->lbl->fnal->anl->isu gac->lanl :gac->lanl gac->lbl :gac->llnl->lbl gac->llnl :gac->llnl gac->mit :gac->llnl->pppl->nyc->mit gac->ornl :gac->llnl->pppl->cebaf->ornl gac->pppl :gac->llnl->pppl gac->slac :gac->ucla->cit->slac gac->ssc :gac->llnl->pppl->cebaf->ornl->ssc gac->ucla :gac->ucla gac->uta :gac->llnl->lbl->fnal->ssc->uta isu->ames :isu->anl->fnal->lbl->llnl->ames isu->anl :isu->anl isu->bnl :isu->anl->fnal->mit->nyc->bnl isu->cebaf :isu->anl->fnal->mit->nyc->pppl->cebaf isu->cit :isu->anl->fnal->lbl->slac->cit isu->doe :isu->anl->fnal->mit->nyc->pppl->cebaf->doe isu->fnal :isu->anl->fnal isu->fsu :isu->anl->fnal->ssc->fsu isu->gac :isu->anl->fnal->lbl->llnl->gac isu->lanl :isu->anl->fnal->ssc->uta->lanl isu->lbl :isu->anl->fnal->lbl isu->llnl :isu->anl->fnal->lbl->llnl isu->mit :isu->anl->fnal->mit isu->ornl :isu->anl->fnal->ssc->ornl isu->pppl :isu->anl->fnal->mit->nyc->pppl isu->slac :isu->anl->fnal->lbl->slac isu->ssc :isu->anl->fnal->ssc isu->ucla :isu->anl->fnal->lbl->slac->cit->ucla isu->uta :isu->anl->fnal->ssc->uta lanl->ames :lanl->gac->llnl->ames lanl->anl :lanl->uta->ssc->fnal->anl lanl->bnl :lanl->gac->llnl->pppl->nyc->bnl lanl->cebaf :lanl->gac->llnl->pppl->cebaf lanl->cit :lanl->gac->ucla->cit lanl->doe :lanl->gac->llnl->pppl->cebaf->doe lanl->fnal :lanl->uta->ssc->fnal lanl->fsu :lanl->uta->ssc->fsu lanl->gac :lanl->gac lanl->isu :lanl->uta->ssc->fnal->anl lanl->lbl :lanl->gac->llnl->lbl lanl->llnl :lanl->gac->llnl lanl->mit :lanl->gac->llnl->pppl->nyc->mit lanl->ornl :lanl->uta->ssc->ornl lanl->pppl :lanl->gac->llnl->pppl lanl->slac :lanl->gac->ucla->cit->slac lanl->ssc :lanl->uta->ssc lanl->ucla :lanl->gac->ucla lanl->uta :lanl->uta DRAFT DECnet Routing 6/7/91 page 28 lbl->ames :lbl->llnl->ames lbl->anl :lbl->fnal->anl lbl->bnl :lbl->fnal->mit->nyc->bnl lbl->cebaf :lbl->fnal->mit->nyc->pppl->cebaf lbl->cit :lbl->slac->cit lbl->doe :lbl->fnal->mit->nyc->pppl->cebaf->doe lbl->fnal :lbl->fnal lbl->fsu :lbl->fnal->ssc->fsu lbl->gac :lbl->llnl->gac lbl->isu :lbl->fnal->anl->isu lbl->lanl :lbl->fnal->ssc->uta->lanl lbl->llnl :lbl->llnl lbl->mit :lbl->fnal->mit lbl->ornl :lbl->fnal->ssc->ornl lbl->pppl :lbl->fnal->mit->nyc->pppl lbl->slac :lbl->slac lbl->ssc :lbl->fnal->ssc lbl->ucla :lbl->slac->cit->ucla lbl->uta :lbl->fnal->ssc->uta llnl->ames :llnl->ames llnl->anl :llnl->lbl->fnal->anl llnl->bnl :llnl->pppl->nyc->bnl llnl->cebaf :llnl->pppl->cebaf llnl->cit :llnl->lbl->slac->cit llnl->doe :llnl->pppl->cebaf->doe llnl->fnal :llnl->lbl->fnal llnl->fsu :llnl->pppl->cebaf->ornl->fsu llnl->gac :llnl->gac llnl->isu :llnl->lbl->fnal->anl->isu llnl->lanl :llnl->gac->lanl llnl->lbl :llnl->lbl llnl->mit :llnl->pppl->nyc->mit llnl->ornl :llnl->pppl->cebaf->ornl llnl->pppl :llnl->pppl llnl->slac :llnl->lbl->slac llnl->ssc :llnl->pppl->cebaf->ornl->ssc llnl->ucla :llnl->gac->ucla llnl->uta :llnl->lbl->fnal->ssc->uta mit->ames :mit->fnal->lbl->llnl->ames mit->anl :mit->fnal->anl mit->bnl :mit->nyc->bnl mit->cebaf :mit->nyc->pppl->cebaf mit->cit :mit->fnal->lbl->slac->cit mit->doe :mit->nyc->pppl->cebaf->doe mit->fnal :mit->fnal mit->fsu :mit->fnal->ssc->fsu mit->gac :mit->fnal->lbl->llnl->gac mit->isu :mit->fnal->anl->isu mit->lanl :mit->fnal->ssc->uta->lanl mit->lbl :mit->fnal->lbl mit->llnl :mit->fnal->lbl->llnl mit->ornl :mit->fnal->ssc->ornl mit->pppl :mit->nyc->pppl mit->slac :mit->fnal->lbl->slac mit->ssc :mit->fnal->ssc mit->ucla :mit->fnal->lbl->slac->cit->ucla mit->uta :mit->fnal->ssc->uta DRAFT DECnet Routing 6/7/91 page 29 ornl->ames :ornl->cebaf->pppl->llnl->ames ornl->anl :ornl->anl ornl->bnl :ornl->cebaf->pppl->nyc->bnl ornl->cebaf :ornl->cebaf ornl->cit :ornl->ssc->fnal->lbl->slac->cit ornl->doe :ornl->cebaf->doe ornl->fnal :ornl->anl->fnal ornl->fsu :ornl->fsu ornl->gac :ornl->ssc->fnal->lbl->llnl->gac ornl->isu :ornl->anl->isu ornl->lanl :ornl->ssc->uta->lanl ornl->lbl :ornl->ssc->fnal->lbl ornl->llnl :ornl->ssc->fnal->lbl->llnl ornl->mit :ornl->cebaf->pppl->nyc->mit ornl->pppl :ornl->cebaf->pppl ornl->slac :ornl->ssc->fnal->lbl->slac ornl->ssc :ornl->ssc ornl->ucla :ornl->ssc->fnal->lbl->slac->cit->ucla ornl->uta :ornl->ssc->uta pppl->ames :pppl->llnl->ames pppl->anl :pppl->anl pppl->bnl :pppl->nyc->bnl pppl->cebaf :pppl->cebaf pppl->cit :pppl->llnl->lbl->slac->cit pppl->doe :pppl->cebaf->doe pppl->fnal :pppl->anl->fnal pppl->fsu :pppl->cebaf->ornl->fsu pppl->gac :pppl->llnl->gac pppl->isu :pppl->anl->isu pppl->lanl :pppl->llnl->gac->lanl pppl->lbl :pppl->llnl->lbl pppl->llnl :pppl->llnl pppl->mit :pppl->nyc->mit pppl->ornl :pppl->cebaf->ornl pppl->slac :pppl->llnl->lbl->slac pppl->ssc :pppl->cebaf->ornl->ssc pppl->ucla :pppl->llnl->gac->ucla pppl->uta :pppl->cebaf->ornl->ssc->uta slac->ames :slac->lbl->llnl->ames slac->anl :slac->lbl->fnal->anl slac->bnl :slac->lbl->fnal->mit->nyc->bnl slac->cebaf :slac->lbl->fnal->mit->nyc->pppl->cebaf slac->cit :slac->cit slac->doe :slac->lbl->fnal->mit->nyc->pppl->cebaf->doe slac->fnal :slac->lbl->fnal slac->fsu :slac->lbl->fnal->ssc->fsu slac->gac :slac->lbl->llnl->gac slac->isu :slac->lbl->fnal->anl->isu slac->lanl :slac->lbl->fnal->ssc->uta->lanl slac->lbl :slac->lbl slac->llnl :slac->lbl->llnl slac->mit :slac->lbl->fnal->mit slac->ornl :slac->lbl->fnal->ssc->ornl slac->pppl :slac->lbl->fnal->mit->nyc->pppl slac->ssc :slac->lbl->fnal->ssc slac->ucla :slac->cit->ucla slac->uta :slac->lbl->fnal->ssc->uta DRAFT DECnet Routing 6/7/91 page 30 ssc->ames :ssc->fnal->lbl->llnl->ames ssc->anl :ssc->fnal->anl ssc->bnl :ssc->fnal->mit->nyc->bnl ssc->cebaf :ssc->fnal->mit->nyc->pppl->cebaf ssc->cit :ssc->fnal->lbl->slac->cit ssc->doe :ssc->fnal->mit->nyc->pppl->cebaf->doe ssc->fnal :ssc->fnal ssc->fsu :ssc->fsu ssc->gac :ssc->fnal->lbl->llnl->gac ssc->isu :ssc->fnal->anl->isu ssc->lanl :ssc->uta->lanl ssc->lbl :ssc->fnal->lbl ssc->llnl :ssc->fnal->lbl->llnl ssc->mit :ssc->fnal->mit ssc->ornl :ssc->ornl ssc->pppl :ssc->fnal->mit->nyc->pppl ssc->slac :ssc->fnal->lbl->slac ssc->ucla :ssc->fnal->lbl->slac->cit->ucla ssc->uta :ssc->uta ucla->ames :ucla->gac->llnl->ames ucla->anl :ucla->cit->slac->lbl->fnal->anl ucla->bnl :ucla->gac->llnl->pppl->nyc->bnl ucla->cebaf :ucla->gac->llnl->pppl->cebaf ucla->cit :ucla->cit ucla->doe :ucla->gac->llnl->pppl->cebaf->doe ucla->fnal :ucla->cit->slac->lbl->fnal ucla->fsu :ucla->cit->slac->lbl->fnal->ssc->fsu ucla->gac :ucla->gac ucla->isu :ucla->cit->slac->lbl->fnal->anl->isu ucla->lanl :ucla->gac->lanl ucla->lbl :ucla->cit->slac->lbl ucla->llnl :ucla->gac->llnl ucla->mit :ucla->gac->llnl->pppl->nyc->mit ucla->ornl :ucla->cit->slac->lbl->fnal->ssc->ornl ucla->pppl :ucla->gac->llnl->pppl ucla->slac :ucla->cit->slac ucla->ssc :ucla->cit->slac->lbl->fnal->ssc ucla->uta :ucla->cit->slac->lbl->fnal->ssc->uta uta->ames :uta->ssc->fnal->lbl->llnl->ames uta->anl :uta->ssc->fnal->anl uta->bnl :uta->ssc->fnal->mit->nyc->bnl uta->cebaf :uta->ssc->fnal->mit->nyc->pppl->cebaf uta->cit :uta->ssc->fnal->lbl->slac->cit uta->doe :uta->ssc->fnal->mit->nyc->pppl->cebaf->doe uta->fnal :uta->ssc->fnal uta->fsu :uta->ssc->fsu uta->gac :uta->ssc->fnal->lbl->llnl->gac uta->isu :uta->ssc->fnal->anl->isu uta->lanl :uta->lanl uta->lbl :uta->ssc->fnal->lbl uta->llnl :uta->ssc->fnal->lbl->llnl uta->mit :uta->ssc->fnal->mit uta->ornl :uta->ssc->ornl uta->pppl :uta->ssc->fnal->mit->nyc->pppl uta->slac :uta->ssc->fnal->lbl->slac uta->ssc :uta->ssc uta->ucla :uta->ssc->fnal->lbl->slac->cit->ucla DRAFT DECnet Routing 6/7/91 page 31 APPENDIX D HEPnet/SPAN NETWORK CIRCUIT COST PLAN Phil DeMar Fermilab March 4, 1988 This document outlines a method for regulating DECnet circuit costs. in a DECnet network made up of multiple individual DECnet networks, independently administered, but connected to each other. This plan has been formulated by network representatives of the Space Physics Analysis Network (SPAN), and the U.S. High Energy Physics DECnet network (HEPnet). SPAN and U.S. HEPnet make up a major part of a very large international DECnet network that supports energy and space research in North America, Europe, and Japan. This plan has been designed to control the internetwork traffic between SPAN and U.S HEPnet, as well as provide a frame-work for controlling internetwork traffic for other entities connecting to either network. Until now, such traffic was largely uncontrolled and not monitored as to the path from source to destination. In order to balance the load or force traffic to follow along a preferred path, this plan outlines a circuit cost structure to be followed which would facilitate preferred traffic flow on a global basis. This internetwork Plan has been adopted (but not yet implemented) by U.S. HEPnet, and is presently pending formal adoption by SPAN. It has also been reviewed by the European DECnet community (HEP and European Space), which characterized it as an appropriate model to follow, pending further study. I. BASIC STRUCTURE OF THE CIRCUIT COST PLAN The plan's guidelines are structured to force an individual network's inter-network traffic to remain on the network backbone, and isolate intra-network traffic on specific links, where it can be monitored and controlled. The basic structure of the plan is to classify DECnet circuits into one of three categories, backbone circuit, tail (or rib) circuit, and internetwork connection. Circuit costs are then assigned according to the category any particular circuit falls within. II. CIRCUIT CLASSIFICATION Backbone Circuits: The backbone circuits for a network are those circuits that have been established for the purpose of supporting intra-network traffic. Since traffic is intended to be supported by and directed down a network backbone, circuit costs along the backbone are to be kept at an absolute minimum. The plan specifies that backbone circuit costs will be set at 1. Additionally, the aggregate circuit cost across a network backbone is to be kept to a maximum of 5. The underlying concept is that it is in fact these network backbones that need to be clearly identified and isolated from each other. DRAFT DECnet Routing 6/7/91 page 32 (Appendix D, cont'd) Tail (Rib) Circuits: Circuits attached to the backbone in order to provide network access for a specific facility (or facilities) are identified as tail (rib) circuits. The distinction between a backbone circuit and a tail circuit is that a tail circuit is established to provide access for a specific facility or facilities, while a backbone circuit is established to provide network-wide access. Unlike the tail circuit, the backbone circuit's use is not facility specific. Since tail circuits are intended only to support traffic originating from or bound to a specific facility or group of facilities, they are to be assigned a high cost. The maximum circuit cost value assignable in DECnet is 25, and that value will be assigned to all tail circuits. It should be noted that ALL tail circuits utilizing Level-2 routing should default to circuit cost of 25, even if the tail circuit does not form part of a connection to another network. Internetwork Connections: The final entity is identified as the internetwork connection, one or more DECnet circuits established specifically to provide inter-network access. By definition, internetwork connections connect two network backbones, since the intended traffic is network specific, not facility specific. The cost of an internetwork connection is assigned no specific value, but specified to fall between 6 and 22. The concept here is that since backbone and tail circuit costs are basically fixed, the internetwork connection costs should be assigned on an individual basis, adjusting the costs to establish the desired inter-network traffic flows. III. GUIDELINES AND SPECIFICATIONS FOR CIRCUIT COSTS A. General Guidelines For Implementation 1. All DECnet circuits between Level-2 (area) routing nodes will be identified as either backbone circuits, tail circuits, or internetwork connections. 2. An internetwork circuit cost authority is required to set circuit costs for internetwork connections, as well as approve deviations from the default values for backbone and tail circuit costs. 3. Level 1 routing costs are not an issue with internetwork traffic and should be left to the discretion of local network managers. However, Level-1 and non-routing nodes will not be changed to Level-2 (area) routing nodes without prior approval of the appropriate internetwork circuit cost authority. 4. All DECnet circuit costs will be given the same value in both directions for every circuit. DRAFT DECnet Routing 6/7/91 page 33 (Appendix D. cont'd) B. Specifications For Circuit Cost Assignment 1. Each backbone circuit segment will be assigned to a minimum value of 1 or 2, with 1 as a default. a. The aggregate circuit cost within the backbone will be kept to a minimum, and will not exceed a value of 5. 2. The DECnet circuit cost of a rib or tail circuit will be set to a default value of 25, although values as low as 22 may be utilized to accommodate individual network requirements. 3. Any local network connecting two or more backbones which is not intended to be an internetwork connection will have an aggregate path cost between the backbones of 51 or greater. 4. Internetwork connection circuit costs will be set by the internetwork circuit cost authority, and will be set to a value lower than that of a tail circuit (22-25), thus forcing traffic to follow a preferred path. a. This allows the modeling of the remaining network where the internetwork costs are less than 45. b. The total cost along an internetwork connection will fall between a value of 6 and 21 inclusive. c. Values between 22 and 34 inclusive are usable but could cause area fragmentation. DRAFT DECnet Routing 6/7/91 page 34 (Appendix D, cont'd) G L O S S A R Y Backbone Circuit: One or more closely connected routing nodes which intentionally support traffic for a network. Rib or Tail Circuit: One circuit connecting the backbone to a local network supporting traffic that terminates at the local network Local Network: A user site containing one or more nodes. These nodes may be connected via rib circuits from one or more backbones. Internetwork Circuit: A circuit that supports the traffic between the backbones consisting of one or more circuit segments. Circuit Segment: A logical portion of a rib or tail circuit which is a connection between logically adjacent nodes; A DECnet circuit. Area Boundary: The circuit segment that connects two DECnet areas. Area Fragmentation: A situation which occurs when level-2 (AREA) routers within the same DECnet AREA lose contiguity. The result is that two separate DECnet AREAs, both using the same AREA number, exist within the network. Communications between those AREAs and the rest of the network will partially (or totally) break down. Aggregate Path: The total cost on all circuit segments which establish connections between any two nodes. DRAFT DECnet Routing 6/7/91 page 35 APPENDIX E DIRECTORY OF CONTACTS I. ESnet DECnet Working Group (EDWG): A. Chairman: Fermi National Accelerator Laboratory: Philip DeMar External Network Manager; HEPnet Manager Fermilab P.O. 500, M.S. 234 Batavia, Illinois 60510 (708) 840-3678, DEMAR@FNAL.BITNET, FNAL::DEMAR B. Membership: Argonne National Laboratory: Robert J. McMahon Computer Scientist Argonne National Laboratory 9700 South Cass Ave Argonne, Il. 60439 (708) 972-7270, B17385@ANLVM.BITNET Brookhaven National Laboratory: Frank Lepera Computer Analyst Computing and Communications Division Brookhaven National Laboratory Upton, New York 11973 (516) 282-4183, HEPnet (BNLCL1::FLEP) BITnet (FLEP@BNLCL1) ESnet Staff (NERSC, Lawrence Livermore Nat. Laboratory): Tony Hain Associate Network Manager ESnet / NERSC P.O. Box 5509 L-561 Livermore, Ca. 94550 (510) 422-4200, HAIN@EAGLE.ES.NET, 42155::EAGLE::HAIN Fusion Research Center (Univ. of Texas): Alan B Macmahon Research Project Manager Fusion Research Center 11.222 RLM Hall University of Texas Austin Tx, 78712 (512) 471-3182, macmahon@fusion.ph.utexas.edu, FRCIFS::ABM DRAFT DECnet Routing 6/7/91 page 36 (Appendix E, cont'd) General Atomics: David Caruso System Analyst General Atomics 13/511 PO Box 85608 San Diego, CA 92138 (619)455-3659; SDSC::CARUSOD or 27652::"caruso@vusc.span" Lawrence Berkeley Laboratory: Robert Fink Head, Networking Systems Group Mail Stop 50F Lawrence Berkeley Labs Berkeley, CA 94720 (510) 486-5692, RLFINK@UX3.LBL.GOV Lawrence Livermore National Laboratory: R. Kevin Oberman Engineering Network Manager Lawrence Livermore National Laboratory P.O. Box 808, L-156 Livermore, CA. 94550 (510) 422-6955, oberman@icdc.llnl.gov Los Alamos National Laboratory: Steve E. Turpin MS B255, C-5 Los Alamos National Laboratory Los Alamos, NM 87545 (505) 667-0750, steve%beta@lanl.gov Massachusetts Institute of Technology: Donald R. Nelson Systems Manager MIT Plasma Fusion Center 175 Albany Street, NW17-248 Cambridge, MA 02139 (617) 253-7616, mitpfc::nelson, nelson@pfc.mit.edu Oak Ridge National Laboratory Curtis E. Bemis, Jr. Senior Research Staff Physics Division, Bldg. 6000,MS-6371 Oak Ridge National Laboratory P.O. Box 2008 Oak Ridge, TN 37831-6371 (615) 574-4769, ORPH01::BEMIS or BEMIS@ORPH01.BITnet DRAFT DECnet Routing 6/7/91 page 37 (Appendix E, cont'd) Stanford Linear Accelerator Center: Charles Granieri Computer Systems Specialist Stanford Linear Accelerator Center Mail Stop 97 2575 Sand Hill Road Menlo Park, CA 94025 (415) 926-2844, CXG@SLACVM.SLAC.STANFORD.EDU, SLACVX::CXG Supercomputer Computations Research Institute (Florida St. Univ.): Douglas Lee, System Manager Supercomputer Computations Research Institute Florida State University 400 Science Center Library Tallahassee, FL 32306-4052 (904) 644-4275; SCRI::DOUG or DOUG@SCRI1.SCRI.FSU.EDU Superconducting Supercollider Laboratory (SSC) Brenda Ramsey SSC Laboratory 2550 Beckleymeade Ave., Suite 260 Dallas, Texas 75237 (214) 708-6052, sscvx1::sys_ramsey II. ESnet Site Coordinators Committee Chairman: R. Roy Whitney CEBAF 12000 Jefferson Ave. Newport News, VA 23606 (804) 249-7536, 6414::Whitney, Whitney@SURAGATE.CEBAF.GOV III. ESnet Operations Staff (NERSC, LLNL): Mike Collins Staff Member, ESnet NERSC P.O. Box 5509 L-561 Livermore, Ca. 94550 (510) 422-4018, COLLINS@CCC.NERSC.GOV DRAFT DECnet Routing 6/7/91 page 38 (Appendix E, cont'd) IV. DECnet Area Managers: Area 1: Glenn Michel Staff Member, Network Operations Center Los Alamos National Laboratory Los Alamos, NM 87545 (505) 667-1845 or FTS 843-1845, GYM@LANL.GOV Area 36: G. R. "Jerry" Johnson Pacific Northwest Laboratory Richland, Washington (509)375-2097, D37885@PNL.MFENET Area 41: William Jacquith Staff Scientist Mail Stop 50F Lawrence Berkeley Labs Berkeley, CA 94720 (510) 486-4388, LBL::JAKE, JAKE@LBL.BITNET, JAKE@LBL.GOV Russell Wright (Alternate) Staff Scientist Mail Stop 50B-2258 Lawrence Berkeley Labs Berkeley, CA 94720 (510) 486-6965, LBL::WRIGHT, WRIGHT@LBL.GOV Area 42: Philip DeMar External Network Manager Fermilab P.O. 500, M.S. 234 Batavia, Illinois 60510 (708) 840-3678, DEMAR@FNALF.FNAL.GOV, FNAL::DEMAR Vyto Grigaliunas (Alternate) Network Analyst Fermilab P.O. 500, M.S. 234 Batavia, Illinois 60510 (708) 840-2539, VYTO@FNAL.GOV, FNAL::VYTO DRAFT DECnet Routing 6/7/91 page 39 (Appendix E, cont'd) Area 43: Frank Lepera Computer Analyst Computing and Communications Division Brookhaven National Laboratory Upton, New York 11973 (516) 282-4183, HEPnet (BNLCL1::FLEP) BITnet (FLEP@BNLCL1) Donald R. Nelson (Alternate) Systems Manager MIT Plasma Fusion Center 175 Albany Street, NW17-248 Cambridge, MA 02139 (617) 253-7616, mitpfc::nelson, nelson@pfc.mit.edu Area 44: Charles Granieri Computer Systems Specialist Stanford Linear Accelerator Center Mail Stop 97 2575 Sand Hill Road Menlo Park, CA 94025 (415) 926-2844, CXG@SLACVM.SLAC.STANFORD.EDU, SLACVX::CXG Michael Sullenberger (Alternate) Systems Programmer Stanford Linear Accelerator Center Mail Stop 97 2575 Sand Hill Road Menlo Park, CA 94025 (415) 926-2294, SLACVX::MLS, MLS@SLACVX.SLAC.STANFORD.EDU Area 46: Vyto Grigaliunas Network Analyst Fermilab P.O. 500, M.S. 234 Batavia, Illinois 60510 (708) 840-2539, VYTO@FNAL.GOV, FNAL::VYTO Philip DeMar (Alternate) External Network Manager Fermilab P.O. 500, M.S. 234 Batavia, Illinois 60510 (708) 840-3678, DEMAR@FNALF.FNAL.GOV, FNAL::DEMAR DRAFT DECnet Routing 6/7/91 page 40 APPENDIX F DERIVATION OF ESNET BACKBONE CIRCUIT COSTS (Some costs were later modified based upon operational experience.) The proposed circuit costs for the ESnet backbone were derived by using the step-by step logical approach outlined below. The process was based on building a circuit cost framework anchored around the backbone links identified as the most desirable least-cost paths within the network. Those particular links were assigned minimal circuit cost values. Circuits were added to the framework in accordance with their ability to be assigned minimal circuit costs. Circuits connecting sites that were potential inadvertent internetwork connections (backdoor links) were the last ones to be added to the framework, and assigned costs consistent with existing circuit cost agreements requiring higher circuit costs. In cases where high circuit costs are mandated by circuit cost agreements, an attempt was made to implement those higher circuit costs within the facility LAN environment, thus avoiding high circuit costs across the ESnet backbone. The step-by step process is outlined below: Step 1: In accord with routing objective 'c.', a circuit cost of 1 should be assigned to the SLAC-LBL, LBL-FNAL, FNAL-MIT, MIT-NYC, and NYC-BNL links. This, in point of fact, now forms a logical backbone for the exchange of traffic between AREAs 41, 43, and 46. Step 2: It would seem desirable, however, not to force all traffic between these AREAs down the same links realizing that the LLNL-PPPL link is also available to support traffic between the West Coast AREA, (41), and the East Coast, AREA (43). Thus, one now also assigns a cost of 1 to the LBL-LLNL, LLNL-PPPL, and PPPL-NYC links. In order to direct traffic from LBL to the East Coast AREAs via the LLNL-PPPL link, the FNAL-MIT link is modified to a cost of 2. Step 3: The ANL-PPPL link is assigned a cost of 3. This is necessary to keep the BNL-FNAL traffic on the FNAL-MIT link (least cost path from BNL to AREA 46 should be to FNAL, not ANL). The FNAL-ANL link is assigned a cost of 2. At this point, there is a basic routing structure for data flow between AREAs 41, 43, and 46, following the LBL-FNAL-MIT and LLNL-PPPL "backbones". The remaining ESnet T1 circuits comprise what has been termed the "Southern tier" of ESnet. Many, but not all, of the sites situated on this Southern tier have backdoor connections to SPAN. The DECnet internetwork circuit cost agreements specify that these sites should have a high aggregate cost from the network backbone. For most facilities, the high aggregate cost has been implemented within the local network environment. This allows assignment of minimum circuit costs on the ESnet backbone links. However, one site within the Southern tier does have a connection to SPAN and does not have high circuit costs implemented within its internal network environment. The ESnet backbone links into that site (UTA) must carry a sufficiently high cost to conform to the internetwork circuit cost agreements. DRAFT DECnet Routing 6/7/91 page 41 ( Appendix F. cont'd) Step 4: CIT, UCLA, GAC, and LANL all support backdoor paths to SPAN. All, however, are able to install the high circuit costs within their local environment, hence the ESnet backbone links can be assigned a minimum circuit cost. Thus, the SLAC-CIT, CIT-UCLA, UCLA-GAC, GAC-LLNL, and GAC-LANL links are assigned a cost of 1. Step 5: The SSC-FNAL and ANL-ORNL links are assigned a cost of 1, while the SSC-ORNL, ORNL-CEBAF, AND CEBAF-PPPL links are assigned a cost of 2. The latter assignment is an attempt to reduce somewhat the routing asymmetry that occurs for the sites along these circuits. However, the wide "diameter" of AREAs 43 and 46, coupled with the multiple connecting T1 links between them, makes for an inordinate, but unavoidable amount of routing asymmetry. Examples of the routing asymmetry may be discovered in Appendix C. Step 6: The LANL-UTA and SSC-UTA links must be assigned high circuit costs in order to discourage internetwork traffic from traversing the UTA system. However, since the backdoor to SPAN is via another network (THEnet), the links are assigned Internetwork circuit costs (17), rather than tail circuit costs (25). Note that assigning these circuit costs to the LANL-UTA and SSC-UTA links discourages DECnet traffic from following a SSC-UTA-LANL path due to the high cost. It remains an objective to transfer the high costs on these links into the UTA network environment, thus enabling assignment of low circuit costs to the UTA ESnet backbone links. Step 7: FSU also has a backdoor connection to SPAN. However, FSU is be able to implement high circuit costs within its local environment. Thus, the SSC-FSU and ORNL-FSU links can also carry minimum costs, and are assigned circuit costs of 1. Step 8: The two remaining circuits (LLNL-PNL & NYC-NYU) are ESnet tail circuits. The LLNL-PNL link is set to a high cost (25) to discourage intra-ESnet traffic from traversing an LANL-PNL back-door. The cost of the NYC-NYU link is also set to a cost of 25 as a safety measure; there are,apparently, no other external considerations at NYU. Step 9: Using the circuit costs derived in the above steps, the "model" ESnet-DECnet network was carefully examined, and verified to meet the routing objectives, with the aid of a DECnet routing simulator. The simulator performs the complete Level-1 and Level-2 routing algorithm for each defined node in the backbone network. The routing at each of the ESnet backbone sites is given in Appendix C. DRAFT DECnet Routing 6/7/91 page 42 APPENDIX G PROCEDURE FOR SELECTING DESIGNATED ROUTER The following procedure for selecting an appropriate DESIGNATED ROUTER on an ethernet was drafted by Bill Duane of DEC. The EDWG would like to thank him for this specific addition to the document, and for his general comments on the content of the document. ------------------------------------------------------------------------------ NOTE: A designated router is selected for each AREA on a multi-area LAN. So, if two AREAs exist on a multi-area LAN, two designated routers will be selected (the one with the highest router priority in each AREA). The designated router (for each area) should be selected as follows: i) Determine the maximum Packet per Second (PPS) rate which each router can provide for LAN to LAN forwarding. Examples of this are about 2000 PPS for the DEMSA based DECrouter 2000, about 350 PPS per VUP for DECnet-VAX host based routers, and about 140 PPS for the DECSA based DECnet Router. Note that the DECnet-VAX number assumes that the node is dedicated to routing (no applications). ii) Subtract the PPS rate overhead for each synch and/or async line from the maximum PPS rate determined above. If you have PPS rates from line counters then use these. For a worst case analysis use: (((Line Speed in bits per sec*2 for FDX)/8 bits per byte)/128 bytes per packet) = max 128 byte packets per second from this line. So for a 64Kbps line, this works out to about 125 PPS. If the router was a DECrouter 2000 with two 64Kbps lines, then the remaining PPS is (2000PPS - 2*(125 PPS))=1750 PPS. iii) List all the routers in the area by their remaining PPS and assign the router priority with those routers with the highest remaining PPS getting the highest Routing Priority. ------------------------------------------------------------------------------ DRAFT DECnet Routing 6/7/91 APPENDIX H page 43 DECNET ROUTING CONCEPTS A general overview of the address and routing structure for DECnet Phase IV is presented in this Section as an aid to understanding ESnet DECnet routing I. DECnet Addressing: DECnet Phase IV (the current release of the product) provides for 2 bytes (16 bits) of address space. That address space is divided into logical groups of addresses, called AREAs. The most significant six bits of DECnet address space identify the DECnet AREA, while the least significant 10 bits identify the individual nodes within an AREA. This provides for 63 different AREAs, each capable of supporting 1023 nodes. The syntax for expressing a DECnet address is typically 'xx.yyy', where 'xx' would denote a specific DECnet AREA, and 'yyy' would specify a particular node within AREA 'xx'. All nodes within a particular DECnet AREA must be contiguous. That is, there must be a path between any two nodes in the same AREA such that all intermediate nodes along that path are also within that same AREA. This is required because DECnet routing uses its hierarchical address structure to determine routing path (ie., intra-AREA nodes are reachable only via intra-AREA connections). Failure to keep all the nodes in the same AREA contiguous will result in a breakdown of the routing function, and loss of access for those detached nodes. II. DECnet System Types: There are three types of DECnet nodes. The majority of DECnet nodes are NON-ROUTING nodes. They maintain no routing information on other nodes or AREAs. They will cache the identity of an adjacent routing node, and use that routing node to send network traffic. In addition to NON-ROUTING nodes, there are two types of routing nodes. Level-1 routing nodes maintain routing information on all nodes within their own AREA; however, they maintain no information on the other DECnet AREAs in the network. Level-2 (or AREA) routers maintain information on paths to other DECnet AREAs, as well as routing information on all the nodes within their own AREA. It should be noted that the Level-2 routers maintain information on paths to other AREAs, and not information on specific nodes in other AREAs. All Level-2 routers within a DECnet network must be contiguous. There must be a path between any two Level-2 routers within the network such that all intermediate nodes along that path are also Level-2 routers. The Level-2 routers, in effect, form their own network "backbone" for the exchange of data across AREA boundaries. Failure to keep intra-AREA Level-2 routers contiguous results in what is referred to as "AREA partitioning", and will cause disrupted communications for the fragmented AREAs. An AREA Partition may totally prevent ESnet off-site DECnet connections, depending on the resultant routing topology. Connection requests/acknowledgements intended for a system in one fragment of a partitioned AREA may be routed to the other fragment of that partitioned AREA (where the intended system does NOT reside). In that situation, the connection would "time-out" and fail. DRAFT DECnet Routing 6/7/91 page 44 (Appendix G. con't.) III. DECnet Routing: DECnet uses hierarchical routing based on the logical division of its address space into AREAs. Level-1 routing is routing between nodes in the same AREA (intra-area routing). Level-2 routing is routing between different DECnet AREAs (inter-area routing). These are two logically separate functions. However, level-2 routing nodes also perform the level-1 routing function for their own area, and hence are level-1 routers as well. Traffic bound for a node within the same AREA will be forwarded via Level-1 routing, and will not cross into another AREA. Traffic bound for a node in another AREA will be sent via Level-1 intra-AREA routing to an intra-AREA Level-2 router, then forwarded to the destination AREA via Level-2 routing (via the Level-2 "backbone"). Once the packet arrives at a node within the destination AREA, it will then be forwarded via Level-1 routing to the destination node. Note that level-1 routing nodes must maintain routing information on the 'nearest' (in terms of aggregate circuit cost) level-2 router within their own area, in order to forward inter-area traffic on to the level-2 routing network. The routing algorithm itself uses 'least cost paths' to determine the routing path. Every DECnet circuit is administratively assigned a static value called a circuit cost. The allowable range for that circuit cost parameter is 1-25. (Note: The DECrouter 2000 is an exception and has a circuit cost range of 1-63.) The least cost path is then the path between source and destination node that has the lowest end-to-end total (or aggregate) cost. Level-1 routers maintain least cost path routing information on all nodes within their AREA, and keep the information current by periodically exchanging and processing routing information with adjacent routing nodes. Level-2 routing nodes must additionally maintain routing information on least cost path to all other AREAs in the network. That information is similarly kept current by a periodic exchange of routing information between adjacent Level-2 routers. While circuit costs themselves are static values, DECnet routing is dynamic. As the topology of the network changes (ie., a circuit goes down, or comes back up), the routing information kept at routing nodes is updated to reflect the new topology, and new 'least-cost-path' tables are generated. In instances where paths are of equal costs, the following situation occurs: 1. The selection of a default path becomes vendor specific. DEC chooses to select the path through the 'next node' with the highest DECnet address. Another common designation of a default path is by a 'least hops' selection. The important point is that designation of a default path in situations where multiple equal cost paths exist is NOT part of the Digital Network Architecture (ie., DECnet), and is a vendor-specific implementation issue. DRAFT DECnet Routing 6/7/91 page 45 (Appendix G. con't.) 2. DECnet optionally supports the concept of path splitting in situations of multiple equal cost paths. If the path splitting option is 'enabled', both paths will be utilized, with alternate DECnet packets being sent down different paths. Path splitting will NOT be enabled within the ESnet-DECnet network. There are two fundamental aspects of DECnet routing that become major factors in assigning DECnet circuit costs to the ESnet backbone: a. Since Level-2 routing is based on least cost path to an AREA, in instances where there are multiple paths to another AREA, traffic will be forwarded only along the path of lowest cost into that AREA. Note that one of the alternate paths may actually be of lower aggregate cost to the destination NODE; however, the routing criteria is lowest cost to the destination AREA, and not to the destination NODE. b. Routing, and hence the resultant traffic flow, between two NODEs may traverse different links in opposite directions. This phenomenon is called asymmetric routing, and becomes increasingly more probable for routing between AREAs when there are multiple paths available for traffic flow between the different AREAs. Asymmetric routing in the DECnet protocol, while not esthetically pleasing and more difficult to understand and manage, is not necessarily a detractor of network performance, provided the available bandwidth in the split paths is similar. However, minimizing the occurrences of asymmetric routing across the network is an objective in an initial circuit cost assignment strategy.