title: COM Runtime Breakdown in Multithreaded Server Application
body: <p>We are experiencing intermittent catastrophic failures of the COM runtime in a large server application.</p>
  
  <p>Here's what we have:</p>
  
  <p>A server process running as a Windows service hosts numerous free-threaded COM components written in C++/ATL. Multiple client processes written in C++/MFC and .NET use these components via cross-procces COM calls (incl .NET interop) on the same machine. The OS is Windows Server 2008 Terminal Server (32-bit).
  The entire software suite was developed in-house, we have the source code for all components. A tracing toolkit writes out errors and exceptions generated during operation.</p>
  
  <p>What is happening:</p>
  
  <p>After some random period of smooth sailing (5 days to 3 weeks) the server's COM runtime appears to fall apart with any combination of these symptoms:</p>
  
  <ul>
  <li>RPC_E_INVALID_HEADER (0x80010111) - "OLE received a packet with an invalid header" returned to the caller on cross-process calls to server component methods</li>
  <li>Calls to CoCreateInstance (CCI) fail for the CLSCTX_LOCAL_SERVER context </li>
  <li><p>CoInitializeEx(COINIT_MULTITHREADED) calls fail with CO_E_INIT_TLS (0x80004006)</p></li>
  <li><p>All in-process COM activity continues to run, CCI works for CLSCTX_INPROC_SERVER.</p></li>
  <li>The overall system remains responsive, SQL Server works, no signs of problems outside of our service process.</li>
  <li>System resources are OK, no memory leaks, no abnormal CPU usage, no thrashing</li>
  </ul>
  
  <p>The only remedy is to restart the broken service.</p>
  
  <p>Other (related) observations:</p>
  
  <ul>
  <li>The number of cores on the CPU has an adverse effect - a six core Xeon box fails after roughly 5 days, smaller boxes take 3 weeks or longer.</li>
  <li>.NET Interop might be involved, as running a lot of calls accross interop from .NET clients to unmanaged COM server components also adversely affects the system.</li>
  <li>Switching on the tracing code inside the server process prolongs the working time to the next failure.</li>
  </ul>
  
  <p>Tracing does introduce some partial synchronization and thus can hide multithreaded race condition effects. On the other hand, running on more cores with hyperthreading runs more threads in parallel and increases the failure rate. </p>
  
  <p>Has anybody experienced similar behaviour or even actually come accross the RPC_E_INVALID_HEADER HRESULT? There is virtually no useful information to be found on that specific error and its potential causes.
  Are there ways to peek inside the COM Runtime to obtain more useful information about COM's private resource pool usage like memory, handles, synchronization primitives? Can a process' TLS slot status be monitored (CO_E_INIT_TLS)?</p>