Updated
Apr 13th, 2012
First Posted
Apr 13th, 2012

Bandwidth Management Technology Comparison

by Dennis Baasch, ET/BWMGR architect A lot of people have been asking how our ET/BWMGR product compares with other products on the market. Its very easy for even the most knowledgeable of network engineers to be buried in the jargon used when describing bandwidth management technology. Frankly, even I don't understand half of the stuff on rivals Packeteer and Allot's websites. And the reason is simple: its designed to "sound good" in a somewhat deceptive way. Remember that its been said that the definition of marketing is, "tricking people into thinking that you have what they want." Which makes understanding how the technology works extremely important when selecting a product for your needs. This document is designed to illustrate the various bandwidth management technologies in widespread use so as to make the lingo used by various vendors a bit less cryptic.

The Technologies

To begin, lets assume we have some Box that sits between an internet connection and some network, and that the purpose of the Box is to somehow affect the flow of traffic to and from the internet. The Box has two ports, one going to an internet router and another connected to a switch that services any number of networks within the "intranet". All traffic must flow through the Box in order to get from the internet to the intranet, or from the intranet to the internet. Hence, every data frame can potentially be affected by the Box. Normally, without any sort of bandwidth management enabled, data frames are passed through the Box as quickly as possible. Data frames come in from the Internet and are passed as quickly as possible to the port connected to the intranet, and vice versa. This is how your typical router or switch functions. Now lets compare how some bandwidth management technologies running on the Box work.

Queuing Algorithms - CBQ and Fair Queuing

Normally, all data to be sent is put into whats called a "queue". Since the connection on one side of the Box may be faster than the connection on the other side, data frames may arrive on one side of the box faster than the other. You don't want the Box to throw away the data, so its put into a "queue" until it can be processed by the slower interface. The data in the queue is then processed sequentially on a first-come, first-serve basis. Typically, it is sent out as fast as the target medium allows, which facilitates the best possible throughput with the lowest possible latency. CBQ, the most popular of the techniques and the one used in most low-end bandwidth management products (YDI, Microtik, turboPIPE, etc,.) stands for "class based queuing". Its a fairly simple technique where data is categorized into user-defined "classes" and then the queues are maintained for each class. Data can then be sent out according to time schedules and prioritized by processing the queues at specific intervals and/or in order of priority. However, in order to "reorder" data frames according to specified priorities, data frames are forced into "class" queues (which means they must be delayed) and processed at specific intervals, sent in order of priority. The purpose of this is to assure that higher priority data frames will always be sent out before lower priority frames. Therefore, high priority data cannot be bottlenecked by anything with a lower priority. The negatives of CBQ are that it introduces latencies (delays) into virtually all traffic that is being managed, and that it is mono-directional in that only outgoing traffic can be controlled. It is also not practical to have a very large number of classes (queues) due to excessive overhead and loss of precision, so controlling hundreds of streams (or hosts, as an ISP might want to) will not work well. CBQ processes the defined class queues in a round-robin fashion: thus, as the number of defined classes increases, the efficiencies of the management decrease. CBQ works best in a corporate environment where the user has control of both ends of the link, and where there are a few identifiable types of traffic that need to be managed. "Fair Queueing" is a similar technique that attempts to allocate bandwidth based on the usage by individual flows. It attempts to be a bit more intelligent in making choices, and it's based on relative allocations rather than priorities. The problem with fair queuing is that in the vast majority of cases it does not do what you want to do; you don't want "fairness" between flows, you want fairness between your users or customers. "Flows" are simply a property of a larger entity, as each entity can have any number of flows. Also, like CBQ, Fair Queuing works well in environments where there are a small number of users and a small number of definable flows. When you have 1000 defined "pipes" or "flows", for example, its not practical to manage the flows "fairly" between so many queues. HTB, the latest "craze" in the linux camp, is "yet another" queuing technique with the same problems: it doesn't scale well and its still a queue-based model. HTB addresses some of the precision issues with queuing algorithms, but it is simply not a technique that you can count on for the long run to manage a large network.

WRED, Diffserv, Token Buckets and other Silliness

At the turn of the century, Cisco had a problem. Bandwidth management was becoming an important concept, and their hardware was wholly inadequate to do it. They had supplied the world with routers and switches with just enough CPU power to do routing; with little extra cycles available to do cpu-intensive bandwidth management. At the time, Cisco dominated the Internet,and in order to address the inadequacies of their hardware, they came up with a series of bad algorithms that could be used on cpu-starved systems to shape traffic in a primitive way. Much like in the days of Microsoft dominance, many other vendors copied these algorithms and they became de-facto standards, not because they were good algorithms, but because they didn't have any better ideas.

TCP Window Shaping and TCP Window Pacing

Back in 1996, when Emerging Technologies was the only company in the world with a real Traffic Shaper, I ran into Packeteer at a trade show; they were a startup that had a new idea; using TCP pacing, or TCP Window Shaping to manage flows. It was a novel idea that I didn't think was necessary; our queue based techniques worked well and at the time, most people only had 56K of bandwidth, or maybe a T1 at most. So we weren't managing very big networks. In 2000 I wrote an analysis similar to this one, making the claim that TCP Pacing wasn't necessary and that our technology was more precise and more efficient. In 2002, when we incorporated Window Shaping into our technology, I took some heat. "You said it was no good, now you're doing it!", they'd say. Well as you gain experience, you learn. While I didn't like the idea of using TCP Window Manipulation as the primary means of limiting bandwidth, we needed a way to limit flows on congested networks. Many networks had more users that they could accommodate at once, and when too many customers were active at once, router's would have latencies that were too high. To illustrate how backup occurs, it's important to understand how TCP works. When you're downloading a file or a web page element, servers will send 1 "window" of data at a time. Then it waits for an ack, telling it that the data has been received and the receiver is ready for more data. 10 years ago, most PCs only used windows of 32K or so; while now they're using much larger windows. As an example, lets compare a window of 64K, which is lower than most modern systems use, to a "shaped" window of 8K. While the browser advertises a window of 64K to the server, the shaper will change the announcement to 8k, so the server will only forward 8K of data at a time. Take the case when you have 3 TCP sessions active in the following diagram. The router can only forward 1 packet at a time; with 3 connections theres substantially more data to process than when shaping is active. When you have 1000s of active connections, the differences in the amount of data to process is dramatic. With local connections, large windows are required to achieve acceptable throughput. But with Internet connections, larger windows just cause congestion; you may get the same throughput with a window of 12K as you do with 64K. So the extra data just causes congestion and latency in your network. The Traffic Shaper determines the correct "Pace" to maximize the traffic flow. The result of all of this is that you can get more customers onto your network. The Window Shaping creates "fairness", because each connection is only using the bandwidth that it needs rather than using big bursts of data. It's important not to confuse Window Pacing with Window Scaling. Many vendors using lesser technologies (such as those based on Linux) mention scaling, but TCP Window scaling is just a TCP option that allows you to expand the TCP window beyond 64K. At this point in time, everyone supports it.

The ET/BWMGR Technology

First of all, when we asked one or our competitor's salesman what they knew about the ET/BWMGR, they told us that it was "like any other public domain software you can download from the internet." Perhaps this is what they'd like you to think, but it is completely untrue. The ET/BWMGR does not have one line of public code; it is a completely custom product, owned 100% by Emerging Technologies, Inc., and you cannot get it anywhere else. The fact that it runs on FreeBSD unix and LINUX platforms does not make it public domain, nor does it imply that the product is somehow inferior. We chose these operating systems for two reasons: first that they are rather popular, and second because they are both very fast. And you need speed with a product like this. Since its obvious we dont believe that any of the previously mentioned "technologies" is suitable for large-scale deployment, you can probably guess that the ET/BWMGR doesn't use CBQ, Fair Queuing or TCP rate limiting as its core management technology. We also don't like to use the term "shaper", because its one of those purposely vague words that really doesn't mean anything. People dont need "shapers", they need to manage traffic. We do, however, utilize both priority queueing and TCP rate limiting as peripheral technologies to achieve what we believe is an optimal balance which maximizes capacity and efficiency. The first concept of our product is common sense. When you connect to the internet with a 28.8K modem, things are slower than when you are on a T1, and you don't lose packets. You don't need to change the window sizes of TCP sessions, because TCP is fundamentally self-throttling. On a slower line, the ACK packets required to "open" the window (and allow more data to be sent) come back slower, so the session moves along more slowly than on a high speed line. Our core technology simply emulates slower access lines for each defined rule (or stream), so the sending machine and the receiving machine see traffic at the same rates they would on a slower line. By monitoring each defined stream (a stream being a host, network or traffic type with a bandwidth limit), traffic of ANY type can be PRECISELY controlled at virtually any bit rate you choose. The bandwidth manager keeps track of how much data has been sent and only allows as much data as you've specified to pass, just as if it were a separate device on a slower line. This method doesn't normally require data to be discarded, unless there is more data in the pipe than is possible to forward as the specified rate without encountering retransmissions. When TCP traffic is transmitted, it would result in the same data being queued multiple times if the original traffic wasn't dropped. But limiting the speed of traffic is not that only element of bandwidth management that needs consideration. When congestion occurs, it is desirable for the bandwidth management device to be able to make on-the-fly decisions about what traffic streams should be streamlined and which streams should be throttled further. Drops are avoided whenever possible, but when discards are necessary, the ET/BWMGR drops lowest priority traffic rather than critical data. Because the prioritization is a secondary process, the ET/BWMGR has a higher capacity in terms of the number of classes it can manage because there is a separation of "pure limitation" and prioritization. Basically, the ET/BWMGR gives you the best features of queue-based bandwidth management without the negatives, while supplying more capacity. The ET/BWMGR utilizes TCP window manipulation to shorten the window, making on-the-fly decisions (we call it "Adaptive Window Management"), thus reducing the backup in the system and reducing the occurrance of packet loss. But, since it is not using rate limiting to achieve the limits, most of the negatives of rate limiting do not occur. Since window management is used only to reduce the flow and not to enforce the rate, windows can be managed "generally", resulting in the desired flow reduction without creating much overhead. In fact, the ET/BWMGR can manage any number of tcp streams, unlike our competitor's products, allowing efficient and effective management even at full gigabit traffic levels.

"Guaranteeing" Bandwidth

Many vendors tell you that they can "guarantee" bandwidth, but there is no real way to do this with any efficiency on the INTERNET. You can guarantee access to the wire, but you cannot guarantee the throughput of any particular application because you have no way of controlling or predicting what the delays (or packet loss) characteristics of the internet wires between you and your destination. You can guarantee minimum access to the wire, and guarantee that a particular traffic stream gets a specific "relative" chunk in relation to other defined traffic types. You can make guarantees on private networks (e.g., corporate networks that do not use internet connections), as the WAN links are just extensions of the LANs. With proper deployment of bandwidth management devices, you can guarantee access (to the bit in most cases) in such private networks. The ET/BWMGR allows you to "allocate" bandwidth to a particular application or customer, or the bandwidth can be shared when it is not immediately needed. Some products on the market say that they can "guarantee" bandwidth and also "share" it when its not in use. But this is impossible. Bandwidth in networking terms is not a spectrum, in that it only is definable over time; one bit gets sent at a time, you can't chop up the bit time, and you can't chop up a packet. So if bandwidth is "shared", once you relinquish the allocation by beginning to send some other data, you lose the total time period required to transmit that packet (which in ethernet is at least 60*8 bits). In that time, you have lost the ability to "guarantee" the bandwidth. If the bandwidth is needed during that time, it cannot be granted, so you have lost the guarantee. In order to absolutely guarantee bandwidth, you MUST pre-allocate enough time slots for the data and make them exclusively available to that data as long as the allocation exists.

The Similarities

All of the technologies described can limit your traffic. They all can identify specific types of traffic or hosts or networks, and set specific policies to control the traffic. They all can be purchased with HTML interfaces and graphing and reporting capabilites, and you can get them in handsome, 1U boxes that fit nicely in your rack or on a desktop.

ET/BWMGR Auto-Shaping "Magic"

Nothing is really "magic", but the auto-shaping capabilities of products using the ET/BWMGR technology is about as close as it gets. With a single setting you can dramatically reduce the queue depths of your egress router, reduce or eliminate congestion and gradually "slow" each connection until an equilibrium is obtained. It's an advantage that makes our product uniquely suitable for a network of any size.

ET/BWMGR Packets Per Second (PPS) Rules

The ET/BWMGR is the only product we know of that has "packets per second" rules. At first glance, they may not appear to be terribly useful. But they are very powerful. They work on a principal that isn't very scientific, but more of a "seat of the pants" observation of how things work. Most "good" protocols, such as HTTP, ftp, DNS, are fairly well-behaved. Bandwidth hog protocols, such as p2p, behave very badly. PPS rules allow you to slow down a host based on the volume of activity rather than the number of bytes of traffic that a client uses. For example, suppose that you give a client a setting of 50 packets per second incoming. If the client is downloading a file, with the maximum 1500 byte packets, the client could get about (1500 * 8 * 50) 600Kbps throughput, best case. If the client is running a p2p program that is querying clients and pinging servers to see if they are available, the packets are generally very small. Perhaps all of them are 60 bytes. So the bandwidth used in that case might be only (60 * 8 * 50) 24Kbps. So you've achieved a goal of many providers, which is to allow good things to run at higher speeds than bad things, without having to specifically define which things are good and which are bad. What's really good about PPS rules is that they are CPU-friendly. They allow you to control p2p and other unruly protocols without having to detect and decode them. On a large network (or with a lower-end appliance), you severely diminish the capacity of the box by enabling protocol sniffing, because each and every TCP and/or UDP packet has to be parsed and examined. None of this is required with PPS rules. PPS rules allow you to control "bad" protocols at a much higher volume of traffic than by using other methods.

ET/BWMGR vs Packetshaper

The difference between the ET/BWMGR and Blue Coat's packetshaper is one of basic philosophy. Their concept is to micro-manage the content on your network, by analyzing your traffic, identifying bottlenecks and then creating policies based on what you think would work better. Our concept is a bit different. By setting general policies for each client and by generally auto-shaping your TCP flows, your network will flow freely without having to set restrictions which may not be suitable when traffic patterns change or different applications are in use. As an example, suppose you were designing a highway. They would say to have a lane for fire trucks and police, and a lane for busses, and a lane for slow-moving vehicles. We, on the other hand, say that you can have an organized, freely-flowing highway by having enough lanes and setting the speed limit to a reasonable value that is likely to keep everyone happy. Another difference between our products is capacity. In order to decode and classify all of the traffic on your network, you have to parse and examine every packet header for several levels. While this creates some interesting graphing possibilities, it also consumes a tremendous amount of CPU cycles, which diminishes the capacity of the system. For smaller networks this may not matter, but at the high end its the difference between the system being able to handle the traffic with minimal latency and not. Our methods are much more conducive to high-end traffic management, where the mix of protocols and the number of sessions can be difficult to micro-manage.

ET/BMWGR vs Allot NetEnforcer (Fair Queuing)

From Allot's marketing literature you would think that they somehow invented fair queuing, but all they've really done is added a bunch of features to it and spent a lot of investor's money hyping them up. They do have some high-end features for distributed processing, but as for bandwidth management its pretty basic stuff at a very high price. The big advantage that we have over any of Allot's products is that Allot does not do TCP window manipulation, which means that they cannot naturally reduce the congestion on your network. They've taken the perspective of Packeteer, in that they seem focused on controlling specific protocols rather than establishing a free-flowing network environment.

Other Vendors

Just about all other vendors use some variation of the public domain code in Linux with their own front end.

Conclusion

We believe that our packet shaping technology measures up against the best products on the market, at any price. Disclaimer: Competitor's descriptions are general and may not accurately describe newer versions of software. Please check with them for specifics about their products.
Add Comment

Next: Bandwidth Management Tutorial