title: Gearman with multiple servers and php workers
body: <p>I'm having a problem with gearman workers running on multiple servers which i can't seem to solve.</p>
  
  <p>The problem occurs when a worker server is taken offline, rather than the worker process being cancelled, and causes all other worker processes to error and fail.</p>
  
  <p>Example with just 1 client and 2 workers -</p>
  
  <p>Client:</p>
  
  <pre><code>$client = new GearmanClient ();
  
  $client-&gt;addServer ('192.168.1.200');
  $client-&gt;addServer ('192.168.1.201');
  
  $job = $client-&gt;do ('generate_tile', serialize ($arrData));
  </code></pre>
  
  <p>Worker:</p>
  
  <pre><code>$worker = new GearmanWorker ();
  
  $worker-&gt;addServer ('192.168.1.200');
  $worker-&gt;addServer ('192.168.1.201');
  
  $worker-&gt;addFunction ('generate_tile', 'generate_tile');
  
  while (1)
  {
      if (!$worker-&gt;work ())
      {
  
          switch ($worker-&gt;returnCode ())
          {
  
              default:
                  echo "Error: " . $worker-&gt;returnCode () . ': ' . $worker-&gt;error () . "\n";
                  break;
  
          }
  
      }
  }
  
  function generate_tile ($job) { ... }
  </code></pre>
  
  <p>The worker code is being run on 2 separate servers. When every server is up and running both workers execute jobs as expected. When one of the worker processes is cancelled, the other worker executes all jobs as expected.</p>
  
  <p>However, when the server with the cancelled worker process is shutdown and taken completely offline, requests to the client script hang and the remaining worker process does not pick up any jobs.</p>
  
  <p>I get the following set of errors from the remaining worker process:</p>
  
  <pre><code>Error: 46: gearman_con_wait:timeout reached
  Error: 46: gearman_con_wait:timeout reached
  Error: 4: gearman_con_flush:write:110
  Error: 46: gearman_con_wait:timeout reached
  Error: 4: gearman_con_flush:write:113
  Error: 4: gearman_con_flush:write:113
  Error: 4: gearman_con_flush:write:113
  ....
  </code></pre>
  
  <p>When i start-up the other server, not starting the worker process on it, the remaining worker process immediately jumps into life and executes any remaining jobs.</p>
  
  <p>It seems clear to me that i need some code in the worker process to cope with any servers that may be offline, however i cannot see how to do this.</p>
  
  <p>Many thanks,</p>
  
  <p>Andy</p>