title: Finding performance issue that may be due to thread locking (possibly)
body: <p>I've spent a little time running valgrind/callgrind to profile a server that does a lot of TCP/IP communications using many threads.  After some time improving the performance, I realised that in this particular test scenario, the process is not CPU bound so the performance "improvements" I'd looked at were of no use.</p>
  
  <p>In theory, the CPU should be very busy.  I know the TCP/IP device it connects to isn't the limitation as the server runs on two machines. One is a PC the other is an embedded device with an Arm processor.  Even the embedded device only gets to about 2% CPU usage but it does far fewer transactions - about a tenth.  Both systems only get up to about 2% even though we're trying to get data as fast as possible.</p>
  
  <p>My guess is that some mutex is locked and is holding up a thread.  This is a pure guess!  There are a few threads in the system with common data.  Perhaps there are other possibilities but how do I tell?</p>
  
  <p>Is there anyway to use a tool like valgrind/callgrind that might show the time spent in system calls?  I can also run it on Windows with Visual Studio 2012 if that's better.</p>
  
  <p>We might have to try walking through the code or something but not sure that we have time.</p>
  
  <p>Any tips appreciated.</p>
  
  <p>Thanks.</p>