Experience with a Class Based Queuing Demonstration
R. Nitzan
February 2, 1998
TABLE OF CONTENTS
Section 1 INTRODUCTION ................................ 1
Section 2 DEMO SETUP .................................. 2
Section 3 GOTCHAS ..................................... 4
Section 4 NEXT STEPS .................................. 4
1. INTRODUCTION
This paper describes a demo of Class Based Queuing (CBQ)
that was done as part of the SC'97 conference in San Jose,
California in November 1997.
Basically the goal of the demo was to show two applications
running using different classes of service over a congested
network, where one application has a higher service class
than the other and hence better performance.
The demo was set up between LBNL and the show floor of SC'97
over the ESnet ATM network. Several routers and ATM switches
were traversed along this path. Two hosts at LBNL sent video
at the same sending rate, and one host at the show floor
received both videos. See Figure 1 below.
One of the main challenges was to generate a particular load
of both high and low classes of service traffic, and then
funnel this traffic load onto a network segment that is now
capable of carrying the load in order to simulate network
congestion. The congestion point is where the higher classed
traffic will get better service. There must also be either
only one congestion point along the demo path OR mechanisms
in place to handle other congestion points in a similar
fashion. In this case, we designed the demo to have only one
congestion point. In the real world, however, any subsequent
congestion point along a path from point A to point B must
also be dealt with to ensure higher class traffic gets
better service.
Participants:
Van Jacobson LBNL-Network Research
Rebecca Nitzan LBNL-ESnet
Bob Olsen Cisco-QOS Development
Brian Tierney LBNL-Imaging and Distributed Computing
Others that helped: Sally Floyd, Jim Gagliardi, John Jones,
Jason Lee, Craig Leres, Clint Wadsworth.
2. DEMO SETUP
This section describes a hop-by-hop sequence of events
through the network starting from the hosts sending the bulk
of the application data to the end host receiving the data
application.
Both hosts lbl-high and lbl-low (shown in Figure 1 below)
are sending data, each coming from a different source
address. The first router the traffic encounters sets the
IP precedence bits (router trw1-gw shown in Fig. 1 below).
There are three precedence bits available to set, so a range
of 0-7 can be used. The default is zero. For this demo,
every packet that enters the router trw1-gw, with source
address lbl-high, has at least one of it's precedence bits
set in the IP header. Traffic with source address lbl-low
entering this router is left as is, that is it's default
value of all zeros.
It is important to note that both demo sending hosts were
connected over separate Ethernets to the first hop router
trw1-gw. We wanted to have the video traffic exceed 10Mbps.
At this point the router had a load of traffic no greater
than 20Mbps soming into it.
The next hop router (esnet-demo2) is connected over a
100Mbps Fast Ethernet. It is also important to note that at
this point we needed to allow for >10Mbps to come into this
router. This router will be doing the Weighted Fair Queuing
(WFQ).
Traffic exits this router to the next hop over a 10Mbps Eth-
ernet connection. Before it exits the router, and because
the amount of traffic exceeds the 10Mbps capacity of the
Ethernet connection, the router esnet-demo1 uses it's WFQ
algorithm on the traffic. Each packet's IP precedence bits
are examined and queued accordingly. There are eight queues,
with varying levels of priority that can be set per queue.
That is, it is possible with the particular version of Cisco
code we were using to set the number of time slots of
service each queue entry will receive for each particular
queue.
It is also possible to look at the packet drops for each
priority level queue thereby enabling us to see if the low
priority traffic was being dropped and all the high priority
traffic was getting through with no drops or fewer drops.
The output interface to the next hop router (esnet-demo1) is
a slower 10Mbps Ethernet. This is how we simulated a conges-
tion point. A traffic flow of >10Mbps is coming in from the
fast Ethernet directed onto the 10Mbps Ethernet thereby
causing the desired congested effect.
After several iterations of Cisco code, and adjustments of
buffer sizes, we were able to verify that the high priority
traffic was getting through with zero drops and the lower
priority traffic was dropping on the output queue.
From this point on the path to the demo was not congested.
The portion of the ATM network that was shared with other
production traffic, we set up a 10Mbps CBR pipe to ensure
that all the test traffic had at 10Mbps allocated to it.
The rest of the path to the show floor was dedicated to this
test.
------- ------- ------- -------
| esnet-|-------| ESnet |--| ESnet |--//--| Show |
| demo1 |AIP | atm | | atm | | atm |
| |OC3 | switch| | switch| | switch|
------- | lbl-sw| | oak-sw| |sc97-sw|
| ------- ------- -------
| |<--10 Mbps CBR-->| |
| -> traffic shaped |
| |
| 10Mbps ether |
------- <----------WFQ here |
| esnet-| |
| demo2 | |
------- |
| |
| |
| 100Mbps ether |AIP OC3
------- ---------
| tr1gw |=======> 2 ethers for | sc97cbq |
| rtr | Craig/Van's tests | rtr |
| | | |
| |<-IP precedence | |
------- set here ---------
| | |
| | |
| --------- 10Mbps ether |
| | |
|10Mbps ether | |
| | |
--------- --------- ----------
|lbl-high | |lbl-low | |sc97-host |
|(iss4)| | |(iss7) | |(lblcon) |
--------- --------- ----------
| | |
| | |
-------------------- Fast Ether ---------------- Fast Ether
lblnet SCInet
Figure 1 - Diagram of the CBQ demonstration.
3. GOTCHAS
Several problems arose.
1 - By using a 10Mbbps ethernet shared media as a congestion
TCP acks coming back from the show floor host were also sub-
jected to this congestion. This caused the TCP to back off
to a trickle. We solved this by adding another Ethernet path
between esnet-demo1 and esnet-demo2 routers. One was for
outbound traffic and one for inbound traffic. This better
simulated a congested full-duplex wide area network link
where the path to the show floor was congested and the path
back was not. In the real world, most likely the priority
bits and WFQ would be used for the path back as well. Due to
a limitation on routers this was not possible for this demo,
mainly because it was not anticipated in advance.
2 - The CBR was set to 10Mbps, without accounting for the
ATM overhead. The demo was sending a full 10Mbps of IP
traffic. Once the ATM overhead was added, the rate was
closer to 12Mbps. This caused the switch doing the traffic
shaping to have congested queues which affected all traffic
going the particular ports causing a large number of cell
drops, including the ESnet production traffic.
3 - The TCP backed off or never was able to send fast enough
with the test application. In order to congest the 10Mbps
Ethernet segment, we sent additional UDP traffic. Since UDP
is not as polite as TCP, it would not back off which pro-
duced the desired effects of congestion. However, with the
WFQ scheme no queue is starved completely. It seemed the
slightest indication of congestion caused the TCP to back
off. Indeed, it was not always obvious where the TCP was
actually dropping packets to cause the backoff.
4. NEXT STEPS
First, the CBR ATM problem has been addressed. The problem
was duplicated in a local test environment. Basically, a
switch doing traffic shaping will run into buffer space
problems once it is sent more data than the shaped amount
indicates. For example, sending at a rate of 12Mbps over a
switch set up for shaping at 10Mbps caused all other traffic
to drop at a rate of near 5%, where "other" traffic totaled
less than 500kbps. This was over OC3 switch ports. It not
entirely clear if this behavior is acceptable for a switch.
Second, there are other versions of TCP that may be less
inclined to back off as quickly. This may be helpful.
Energy Sciences Network_
LBNL
nitzan@es.net
Last modified: Mon Feb 2, 15:17:16 PST 1998