Why your maximum throughput is less than your bandwidth.

SASE Secure Access Service Edge

Network switchThis posting is more technical than most on this blog.  It was motivated by a recent customer that has a 6Mbps MPLS network connecting Boston to China.  The network was been performing very well.  But then they recently started using FTP to transfer files and they could not attain anywhere close to 6Mbps of throughput.  This entry goes into the details.

TCP provides reliable transport of data between devices. The sender sends data in a stream in which bytes are identified by sequence numbers. The receiver acknowledges that it has received the data by sending the sequence number (acknowledgment number) of the next byte of data that it expects to receive. The receiver also advertises its TCP Window Size to the sender to advertise the amount of data that it can accept.

This “TCP Window” is the maximum number of bytes that can be transmitted by the sender without being acknowledged as having been successfully received by the other end. This means the receiver can safely expect that no more than window-size bytes of data need to be buffered for that connection, and that any additional data beyond that in sequence order can legitimately be discarded (because it is so defined by the TCP protocol).

When you have a high latency connection, such as satellite or a very long distance, the latency has a big effect on thru-put since TCP specifies that acknowledgements must be sent and received as the data is streaming.

How can you determine your maximum transfer rate?  The formula below will do the trick.  Just remember that this formula (the Mathis algorithm) does not account for packet loss, so reduce the number to account for this:

Maximum Possible Transfer Rate = TCP Window Size/RTT
Where RTT is Ping Response in Millseconds/1000.

An example:

  1. Ping time from NY to Los Angeles: 70ms. Therefore, RTT= 70/1000=.07
  2. Maximum Possible Transfer Rate = 32/.07 = 457 KiloBytes/sec or 3.657 Mbps (Remember: 1 byte = 8 bits) This tells you that for a single data stream, such as FTP, a circuit larger than 3.657Mbps will not allow you to transfer a file any faster than 3.657Mbps.
  3. It doesn’t matter how big your circuit is.

Here is another example:

  1. Ping time from NY to Shanghai is 280ms.Therefore, RTT = 280/1000=.28
  2. Maximum Possible Transfer Rate = 32/.28 = 114 KiloBytes/sec or 914Kbps.
  3. If you have an E1/T1, you will have full access to 1540 Kbps, but with a single data flow, such as FTP, you will be limited to 914Kbps.

Now what if you make configuration changes to your TCP Window size to 64K:

For the NY to LA example, 64/.07=914 KiloBytes/sec
For NY to Shanghai: 64/.28=228 KiloBytes/sec

So what do we learn from this?

There are only two things you can do to affect your data thru-put on a wide area network:

  1. Increase your TCP window size, or
  2. Reduce latency.

So a low latency connection makes a big difference on your thru-put. If latency is 1 second round-trip, the peak data rate can never exceed 65KB/second, which is 524Kbps, using a TCP Window of 65,535 bytes.

What size TCP window is needed to fill a 10Mbps circuit that has a 70ms latency between two locations?  You can use this online WAN Throughput Calculator, but here is the calculation so you understand:
.07 seconds x 10Mbps x 1byte/8bits = 87,500 bytes required window size to use entire bandwidth with one data stream.

Standard TCP allows a maximum window size of 64,000 bytes. Cisco’s WINScale TCP option allows you to configure a larger window size.  You should experiment with this, since larger window sizes also mean larger retransmits when packets are lost, so a point of diminshing returns will be approached.

If you need more thru-put for FTP, run multiple sessions so that you can take advantage of the limitation that apply to a single data stream.

Here are some numbers to save you the calculations.  If you configure a 64KB TCP Window Size, your maximum thoughtput, without packet loss, is the following:

RTTMax ThroughputMax ThroughputMax Throughput
Latency64K Window64K WindowWindows XP
 In KbpsIn KBpsIn Kbps
0.1 mS803,755 Kbps100,469.4565,965 Kbps
50 mS10,371 Kbps1,296.42,795 Kbps
60 mS8,658 Kbps1,082.32,330 Kbps
70 mS7,431 Kbps928.91,998 Kbps
80 mS6,509 Kbps813.61,749 Kbps
90 mS5,790 Kbps723.81,555 Kbps
100 mS5,214 Kbps651.81,400 Kbps
110 mS4,742 Kbps592.81,272 Kbps
120 mS4,349 Kbps543.61,167 Kbps
130 mS4,016 Kbps502.01,077 Kbps
140 mS3,730 Kbps466.31,000 Kbps
150 mS3,482 Kbps435.3933 Kbps
200 mS2,614 Kbps326.8700 Kbps
250 mS2,093 Kbps261.6560 Kbps
260 mS2,012 Kbps251.5539 Kbps
270 mS1,938 Kbps247.9519 Kbps
280 mS1,869 Kbps233.6500 Kbps
290 mS1,804 Kbps225.5483 Kbps
300 mS1,744 Kbps218.0467 Kbps
350 mS1,496 Kbps187.0400 Kbps
400 mS1,309 Kbps163.6350 Kbps
450 mS1,164 Kbps145.5311 Kbps
500 mS1,047 Kbps130.9280 Kbps

 

I hope that this post adds some clarity to an often misunderstood concept.  This entire concept changes when you include Network Optimization or Network Acceleration in your wide area network.  This technology can increase your throughput on by several 100% for many types of files or applications, improving performance and reducing the need for expensive WAN bandwidth.  If you want faster WAN performance, you truly need to look at WAN Optimization or WAN Acceleration.  The long term payoff in productivity and lower bandwidth cost over time is clear.  SD-WAN-Experts can help you make the right decision and obtain the best prices.  Contact us today!

Share this post

Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on pinterest
Share on print
Share on email