Bidirectionally Testing Network Connections
Beginning in Spring 2009, I started seeing network problems at some customer sites: data transmissions would hang and time out; connections via OpenVPN links would work at first but time out as soon as more data
Bidirectionally Testing Network Connections
Beginning in Spring 2009, I started seeing network problems at some customer sites: data transmissions would hang and time out; connections via OpenVPN links would work at first but time out as soon as more data was transmitted; VPN connections using UDP would not work at all (but would work magically when changing OpenVPN's port to 80/TCP). All these symptoms are usually tied to traffic shaping, problems with the maximum packet size (maximum transfer unit, MTU) or other issues usually found between the end-points of the transmission. But how do you test and record the performance of "your" Internet?
Using Network Tools for Debugging
First of all, you can use network tools to inspect the path between the two communication partners.
- hping3 is a tool to construct arbitrary TCP/IP packets and send them to hosts. This is very useful for sending TCP SYN packets and seeing if TCP SYN+ACK packets return. You can also send packets with a custom payload, and check for MTU issues.
- ping is a well-known tool for sending ICMP Echo Request packets.
- tracepath is used for discovering the MTU between source and destination. It uses UDP packets, starting with a size of 65536 byte and reducing the size after receiving timeouts or errors.
- traceroute is the classic tool for displaying the intermediate hops between source and destination.
Most tools display error messages, round-trip/response times and other useful information. Although many people rely on the results of tools, always keep in mind that sending test packets is not the same as sending real data over different protocols. There are routers and other network devices out there that do not treat every packet equally.
One site reported very strange effects. Immediately after you used a Bittorrent client or a protocol over "bad ports", the bandwidth for "regular" network traffic dropped notably. After half an hour or more, the performance would return to normal. This behaviour may indicate a violation of net neutrality, the presence of filters, or other defects. The big problem is to find the cause, since it could be the network path, flaky hardware, broken software, prioritisation of network traffic, policy enforcement, or some entirely different reason. If there are intermediate network devices manipulating your packet flow, then there's a chance you will never identify or "see" these devices with the tools available.
Using Data Transmissions for Debugging
Using "real" data and "real" protocols for testing is better, but how do you do that? Clicking on download links in your Web browser and using a stop watch isn't well suited to finding hard facts. What about the other direction? Downloads are treated differently from uploads, if you're on an asymmetric Internet connection. You need to test transfers in both directions, and you need to use different protocols and ports. Why not let a script do all of this, and collect a nice summary?
First, we thought to use a shell script containing a few tools, and send data around. The problem is that you might want to do some statistics. Filtering the output and calculating transmission rates can be quite difficult. That's why a Perl script was created. It uses HTTP, HTTPS, SCP, FTP, SMTP, and IMAP. All protocols except SMTP are automatically used bidirectionally. The script uses a configuration file where login, password, paths, and other parameters can be configured. You can use different servers for every