title: Communication between workers in uwsgi
body: <p>I'm relatively new to Python, coming from a .Net background.</p>
  
  <p>The short version: How can I create an application-wide singleton or some other mechanism to allow multiple threads/processes to communicate with each other?</p>
  
  <p>Maybe I'm spoilt but in .Net, I'd just create something in <code>App_Start</code> or something else at the "application level". How can I do the same in python/uwsgi?</p>
  
  <p>The long version:</p>
  
  <p>We've got a restful API written using Django.</p>
  
  <p>Some of the calls require some pre-processing and then passing to a back-end system which performs long-running operations.</p>
  
  <p>The flow at the moment looks something like...</p>
  
  <ul>
  <li>Receive request to process all documents matching given criteria</li>
  <li>Api determines which documents match those criteria (on the order of 100,000 - takes 15-20s)</li>
  <li>Api generates a uuid for this batch request</li>
  <li>Api publishes a message to a back-end queue for each of those documents, referencing the batch id.</li>
  <li>Api listens on a different queue for "completed" messages and counts success/failures for each batch id (~1-15 minutes)</li>
  <li>While processing is happening, UI can request an update for a specific batch id</li>
  </ul>
  
  <p>We need to listen to the response queue using a different thread to that which is used to serve pages since it's in a <a href="http://www.laurentluce.com/posts/distributed-messaging-using-rabbitmq-and-python/" rel="nofollow">wait spin-loop</a>...</p>
  
  <pre><code>while True:
      self.channel.wait()
  </code></pre>
  
  <p>I was handling this by getting a reference to a <code>QueueManager</code> which is a singleton. The manager fires off the initial request, records the batch id and then, in a second thread, monitors the queue and updates local state.</p>
  
  <p>We don't actually care about preserving the state long-term - If the messages are on the queue, the processing will be done by the back-end and the state monitoring is only a cue to the user that things are underway. If they browse away, they also lose access to the status (batch id is stored in-memory in  JS)</p>
  
  <p>This had a couple of benefits - we avoided using a database for syncing information (and the associated cleanup). We were able to use a single thread as a message consumer and didn't have to worry about concurrency issues as only one thread will ever collect messages/update the local state.</p>
  
  <p>So... Now it's time to run it using uwsgi I've found a major issue. If I set the number of processes to 1, the the singleton works as expected, but all requests are blocked during the 15-20s that the api is processing data. Clearly that's unacceptable. Conversely, if I spin up multiple workers, each has its own singleton and its own message listener - so it's pretty much random if the publisher and consumer are the same process. And even if they are, the request for a status update probably won't end up at that same process.</p>
  
  <p>How can I swap state information between multiple workers? Is there a way to use multiple threads instead of multiple workers?</p>
  
  <p>It seems like I really need:</p>
  
  <ul>
  <li>n threads, each serving requests</li>
  <li>1 thread listening on the queue</li>
  <li>some in-memory way of communicating between them</li>
  </ul>
  
  <p>Note I've already got <code>--enable-threads</code> but that only seems to apply to new threads I spawn (no idea why that wouldn't be enabled by default)</p>