title: Best practice: Monitor processes
body: <p>I was wondering what the best practice solution would be to constantly monitor and resart processes, because there are multiple ways in doing it.</p>
  
  <p>Additional info:
  I have a unix program which uses multiple processes to work. There's a main process, it always starts first and is not likely to die or terminate without stopping the program.</p>
  
  <p>Then I spawn multiple "module" processes, which take care of some work and communicate through the main process. Those modules sometimes die because of exceptions, and because it's an external program I can't resolve the issues, so I have to restart them if they die.</p>
  
  <p>I've made a program to check if any of the modules died and restart them, but I need to run it manually. My program checks if the pid files of the modules exist and if they listen on a specific tcp port. If the pid file doesn't exist or the socket can't establish connection, it restarts the module.</p>
  
  <p>My thoughts so far:</p>
  
  <ul>
  <li>Cron job to run the checks every minute and restart any dead modules. (kind of an overkill, because they don't die that frequently)</li>
  <li>Daemon running in the background, which starts the modules and receives notifications if they die, so it doesn't have to check them constantly. (SIGCHLD signal, os.wait)</li>
  </ul>
  
  <p>If I use the daemon method, how should I communicate with the daemon through my interface? (socket, or maybe a file which gets read if the daemon receives a specific signal)</p>
  
  <p>Usually I would just go with the daemon because it seems to be the best practice method to restart the modules asap(cron only runs once a minute), but I've wanted to get some opinions from more experienced users. (I've never done something like this before, and asking doesn't hurt anyone :D)</p>
  
  <p>I apologize if these questions are answered somewhere else, but I couldn't find any related question.</p>
  
  <p>P.S. If I forgot something or you need more infos, please feel free to ask. :)</p>