TCP Throughput Tests
What is my target measuring TCP performance?
This is one very important question that needs to be well addressed before going through advanced TCP performance test.
TCP is the most widely used protocol for applications transiting over the Internet. Given all factors influencing its performance, it is very important to make sure of what one is trying to prove by doing TCP performance testing.
Test typical application behavior
Use the scripts called TCP Throughput. These scripts send large chunks of bytes from Endpoint 1 to Endpoint 2, then waits for an acknowledgment.
It thus simulates the core file transfer transaction performed by many applications.
TCP auto adapts to available bandwidth therefore will be a good indicator of the data available for an application.
Network transmission capability verification -> Use UDP
If one is after defining pure network transmission capability, TCP is likely not the best solution to prove the network. It will be a lot more efficient to use UDP tests which act more as bit blasters and would report transparently any unidirectional data loss. UDP is not connection oriented and therefore a lot more adequate when testing line transmissions.
When looking at network transmission capability, Ixia performance endpoint is able to run UDP tests that are a lot more precise to report on line capacity transmission capability and packet loss.
UDP traffic generation is however dangerous as it will not adapt to available bandwidth and therefore is likely to create router congestions and affect other traffic.
Make sure you use the right Operating System
One key factor is the TCP stack used by the performance endpoint in the test. The most common OS stacks, Linux and Windows, have different behaviors and therefore can behave with different performance on same environment. It is important to understand that Ixia performance endpoint is using the underlying OS stack and therefore the performance of the stack is part of the performance test itself.
Ixia performance endpoints come with most common OS and OS versions, allowing to control what version can be used.
It is also very important to understand the stack parameters on the OS can have a big impact on the results.
Define what the target performance is: what are you looking as a good result?
Depending on the characteristics of the line you are trying to test, it is important to define a target maximum throughput that is proving a good result for a single TCP stream.
This will help defining how the performance endpoint has to be used to use the stack.
For example, it is likely that a FTP server on a 1G local LAN can be expected to reach 500M throughput or more, but if 1G line is used between 2 different remote sites with long delays, is this 500M throughput still be a valid target?
Am I looking to prove a specific application?
All applications are specific in the way they would use the TCP stack: it cannot be expected that small volume transactions to a mail server would behave the same way as long download from FTP servers.
Hawkeye script libraries provide a very wide range of common applications that can be complemented by specific built applications to help understand the behavior over a specific network.
It is also important then to compare how a standard TCP throughput test compares to a specific application test over TCP. Hawkeye allows running both aspects and understanding what could explain the difference in expected performance.
Do I want to use default OS settings or tune them for the TCP stack?
Most real applications that use a TCP stack use default parameters. However, the most common Operating Systems (Windows and Linux) allow controlling the way the TCP stack is used through socket APIs. It does not allow controlling how the stack is going to behave and auto adjust its window based on the environment, but will allow to control the memory buffers for send and receive to optimize the use of the stack.
These parameters are very important if one’s target is to understand the best possible TCP performance a stream can get. As changing these parameters can influence greatly on the stack optimization, there will be some performance impact by tuning these parameters.
Ixia performance endpoint has implemented these parameters. A typical test is to run a “default settings” TCP test to see what the default stack is capable of, and then adjust these parameters based on what is known of the environment (typically the RTT) to find out what is the maximum achievable performance.
WARNING: If you leave OS parameters by default then when you set Hawkeye manually force OS to use some buffers sizes it can happen that the values to are pushing are over the limits configured on the OS and so are not taken into account.
Do I need one or several TCP streams?
If one is testing a link for TCP performance, it is important to know if one single TCP stream is the best approach. Typically, links would transport in real life several streams in parallel. These streams would all correspond to different applications or even to same applications using different channels.
Therefore, if link capacity for TCP is the target, it is often more efficient to test several TCP streams in parallel, which would usually result in better aggregated performance than a single TCP stream.
One important point: speed measurements free tools that are available on the Internet usually use several concurrent TCP streams to assess line capacities.
Each Stream is limited to a maximum rate of 4.19Gbps. To test rates higher than 4.19Gbps, please use one of the N Stream Tests.
TCP Throughput Tests Descriptions
TCP throughput tests performance
TCP throughput tests are typically dependent on the underlying stack, overall capability (cpu, network card) of the active endpoint.
For powerful enough devices (xr2000, servers, high range laptops), it should be possible to generate up to line rate TCP streams (resulting in a goodput calculation of 960 to 970 Mbps depending on the transport layer.)
Less powerful endpoints will typically provide less capability that can be measured on case by case.
As long delays can influence dramatically TCP throughput calculations it is recommended to verify the maximum throughput capable with new devices by running end to end scripts.
Optimizing Ixia performance Endpoint for Performance Testing
Ixia performance endpoint Stack utilization parameters
The first item is the way the endpoint will be using the TCP script itself.
In tcp advanced test setup:
SEND
size = file_size (1000000)
buffer = send_buffer_size (65535)
rate = send_data_rate (to be filled up)
The parameters used are:
-
Filesize: how much data is sent, and then wait for acknowledgement. Increasing it will result into getting more data sent at once to the stack, and increase the length of the timing record (how long between each performance record).
Recommendation for TCP performance testing is to target 1 timing record per second, so make sure based on expected throughput it takes at least 1 second to download filesize. Using up to 2 or even 5 seconds sometimes help getting the best performance.
-
Send and Receive buffer size: determine how much data will be sent to the TCP stack for processing for each SEND and RECEIVE operation. They are unrelated to TCP sliding window service.
Recommendation for endpoint send and received buffer size for performance endpoint is to make sure they are not too small, so that they are not too many cycles. They should also not be in excess to the CONNECTION send and receive buffers (see below) that are used to configure the TCP stack. Typical recommended value is 16kbytes, 32kbytes or 64kbytes. Unless misconfigured, this should not have a great impact on TCP performance.
-
Send data Rate: determine the maximum data rate the Performance endpoint will try to push sending through the stack. It can be set to unlimited (in which case the data will be used at maximum rate) or any value in KB.
Recommendation for endpoint send data rate is to use unlimited when trying to perform as much as possible. However, a side effect using unlimited value is that the stack will be used at maximum but can be overloaded by performance endpoint throwing data in bursts or spikes. When targeting a defined maximum throughput, it is a good idea to use the defined throughput so that the TCP stack is used in a more even or stable way, and therefore it is stabilized. When pushing the stack and/or network to the limit, it is always better to do some binary search with send data rate to define the most stable value given network conditions and endpoint send and receive OS performance.
Operating System TCP Stack settings
A prerequisite to any further tuning in Ixia Performance Endpoint is to have the TCP stack parameters in the OS itself.
Find below optimization elements for the 2 main Operating Systems (Windows and Linux).
Linux Operating systems
The default values are found with TCP allocated memory:
$ cat /proc/sys/net/ipv4/tcp_mem
The default and maximum amount for the receive socket memory:
$ cat /proc/sys/net/core/rmem_default
$ cat /proc/sys/net/core/rmem_max
The default and maximum amount for the send socket memory:
$ cat /proc/sys/net/core/wmem_default
$ cat /proc/sys/net/core/wmem_max
The maximum amount of option memory buffers:
$ cat /proc/sys/net/core/optmem_max
Recommended changes:
Set the max OS send buffer size (wmem) and receive buffer size (rmem) to 16 MB for queues on all protocols. In other words set the amount of memory that is allocated for each TCP socket when it is opened or created while transferring files.
CAUTION: The default value of rmem_max and wmem_max is about 128 KB in most Linux distributions, which may be enough for a low-latency general purpose network environment or for apps such as DNS / Web server. However, if the latency is large, the default size might be too small. Please note that the following settings going to increase memory usage on your server
# echo 'net.core.wmem_max=16777216' >> /etc/sysctl.conf
# echo 'net.core.rmem_max=16777216' >> /etc/sysctl.conf
Set minimum size, initial size, and maximum size in bytes:
# echo 'net.ipv4.tcp_rmem= 10240 87380 16777216 ' >> /etc/sysctl.conf
# echo 'net.ipv4.tcp_wmem= 10240 87380 16777216 ' >> /etc/sysctl.conf
Turn on window scaling which can be an option to enlarge the transfer window:
# echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
Enable TCP timestamps as defined in RFC1323:
# echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
Enable select acknowledgments:
# echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
By default, TCP saves various connection metrics in the route cache when the connection closes, so that connections established in the near future can use these to set initial conditions. Usually, this increases overall performance, but may sometimes cause performance degradation. If set, TCP will not cache metrics on closing connections. The benefits or activating or disabling caching might depend on test expected.
# echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf
Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them.
# echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
The commands above configure the parameters and made them persistent into Linux. To reload the changes:
# sysctl -p
Windows Operating Systems
Please refer at following: http://msdn.microsoft.com/en-us/library/cc558565(v=bts.10).aspx
This article contains all information from Microsoft to optimize TCP stack for performance management.