Take the 2-minute tour ×

I'm having a performance issue that I don't understand. The system I'm working on has two threads that look something like this:

Version A:

  • Thread 1: Data Processing -> Data Selection -> Data Formatting -> FIFO
  • Thread 2: FIFO -> Socket

Where 'Selection' thins down the data and the FIFO at the end of thread 1 is the FIFO at the beginning of thread 2 (the FIFOs are actually TBB Concurrent Queues). For performance reasons, I've altered the threads to look like this:

Version B:

  • Thread 1: Data Processing -> Data Selection -> FIFO
  • Thread 2: FIFO -> Data Formatting -> Socket

Initially, this optimization proved to be successful. Thread 1 is capable of much higher throughput. I didn't look too hard at Thread 2's performance because I expected the CPU usage would be higher and (due to data thinning) it wasn't a major concern. However, one of my colleagues asked for a performance comparison of version A and version B. To test the setup I had thread 2's socket (a boost asio tcp socket) write to an instance of iperf on the same box ( with the goal of showing the maximum throughput.

To compare the two set ups I first tried forcing the system to write data out of the socket at 500 Mbps. As part of the performance testing I monitored top. What I saw surprised me. Version A did not show up on 'top -H' nor did iperf (this was actually as suspected). However, version B (my 'enhanced version') was showing up on 'top -H' with ~10% cpu utilization and (oddly) iperf was showing up with 8%.

Obviously, that implied to me that I was doing something wrong. I can't seem to prove that I am though! Things I've confirmed:

  • Both versions are giving the socket 32k chunks of data
  • Both versions are using the same boost library (1.45)
  • Both have the same optimization setting (-O3)
  • Both receive the exact same data, write out the same data, and write it at the same rate.
  • Both use the same blocking write call.
  • I'm testing from the same box with the exact same setup (Red Hat)
  • The 'formatting' part of thread 2 is not the issue (I removed it and reproduced the problem)
  • Small packets across the network is not the issue (I'm using TCP_CORK and I've confirmed via wireshark that the TCP Segments are all ~16k).
  • Putting a 1 ms sleep right after the socket write makes the CPU usage on both the socket thread and iperf(?!) go back to 0%.
  • Poor man's profiler reveals very little (the socket thread is almost always sleeping).
  • Callgrind reveals very little (the socket write barely even registers)
  • Switching iperf for netcat (writing to /dev/null) doesn't change anything (actually netcat's cpu usage was ~20%).

The only thing I can think of is that I've introduced a tighter loop around the socket write. However, at 500 Mbps I wouldn't expect that the cpu usage on both my process and iperf would be increased?

I'm at a loss to why this is happening. My coworkers and I are basically out of ideas. Any thoughts or suggestions? I'll happily try anything at this point.

share|improve this question
Just a wild guess, but maybe thread 2's data set no longer fits comfortably into the CPU's cache, whereas previously it did? –  Jeremy Friesner Mar 11 '11 at 23:51
Sounds like you need to profile with something like oprofile. –  Sam Miller Mar 12 '11 at 1:00
add comment

3 Answers

This is going to be very hard to analyze without code snippets or actual data quantities.

One thing that comes to mind: if the pre-formatted data stream is significantly larger than post-format, you may be expending more bandwidth/cycles copying a bunch more data through the FIFO (socket) boundary.

Try estimating or measuring the data rate at each stage. If the data rate is higher at the output of 'selection', consider the effects of moving formatting to the other side of the boundary. Is it possible that no copy is required for the select->format transition in configuration A, and configuration B imposes lots of copies?

... just a guesses without more insight into the system.

share|improve this answer
The post formatted data is not much larger than the pre formatted. Maybe 1% larger if even that. The data rate going into the FIFO is ~500 Mbps and the data going out of the socket is ~500 Mbps so you can see that the data is changed only a very small amount. Sorry, there isn't more code involved here. Is there a particular piece that would be helpful? I tried to keep it simple here since I believe I've isolated the issue (bursty writes to a socket) and the code base is quite large. Edit: Both configurations require a copy. –  user656071 Mar 12 '11 at 0:35
add comment

What if the FIFO was the bottleneck in version A. Then both threads would sit and wait for the FIFO most of the time. And in version B, you'd be handing the data off to iperf faster.

share|improve this answer
add comment

What exactly do you store in the FIFO queues? Do you store packets of data i.e buffers?

In version A, you were writing formatted data (probably bytes) to the queue. So, sending it on the socket involved just writing out a fixed size buffer.

However in version B, you are storing high level data in the queues. Formatting it is now creating bigger buffer sizes that are being written directly to the socket. This causes the TCp/ip stack to spend CPU cycles in fragmenting & overhead...

THis is my theory based on what you have said so far.

share|improve this answer
In version A and B we store buffers in the FIFO. Both versions write 32k chunks of data to the socket at a time. And since TCP_CORK is in use I've minimized TCP/IP overhead as best I can. It could be I'm not understanding something fundamental about socket writes? –  user656071 Mar 12 '11 at 10:54
add comment

Your Answer

