title: PM2: Stagger process restart for many processes
body: <p>I'm using PM2 to manage my processes.  Currently, I have a series of bots on it, each of which has the following basic configuration:</p>
  <pre><code>{
    &quot;apps&quot; : [
      {
        &quot;name&quot; : &quot;rasa_bot-rasa&quot;,
        &quot;script&quot; : &quot;python&quot;,
        &quot;pid_file&quot; : &quot;~/rasa_bot/rasa_bot.pid&quot;,
        &quot;out_file&quot; : &quot;~/logs/rasa_bot/rasa_bot.log&quot;
        &quot;cron_restart&quot;: &quot;0 23 * * Sat&quot;
      },
      #other supporting processes go here, with the same cron_restart
    ]
  }
  </code></pre>
  <p>Each bot has about three supporting processes, and all have the same cron restart. So, 10 programs means 30 processes, all restarting at the same time.  As such, when the restart rolls around, alot of resources are used, which seems to have lead to this intermittent error:</p>
  <pre><code>2020-11-28T23:00:16: PM2 log: Process with pid &lt;PID here&gt; could not be killed
  2020-11-28T23:00:16: PM2 error: app=rasa_bot-rasa id=45 pid=&lt;PID here&gt; could not be stopped
  2020-11-28T23:00:16: PM2 error: Process with pid &lt;PID here&gt; already exists
  2020-11-28T23:00:16: PM2 error: Error: Process with pid &lt;PID here&gt; already exists
  2020-11-28T23:00:16: PM2 log: App [rasa_bot-rasa:45] exited with code [0] via signal [SIGKILL]
  </code></pre>
  <p>This causes the bot to exit, and not come back up; it will simply have an &quot;errored&quot; status.  To resolve this, I'd like to setup two things:</p>
  <ol>
  <li>A rolling restart in pm2, where it will a restart a process, wait for it to finish, and then move on to the next one.</li>
  <li>A way to get pm2 to restart a process with the status &quot;errored&quot;.</li>
  </ol>
  <p>So far, I'm still looking into the rolling restart, and the closest I've found is clustered mode, which does not seem to be quite what I'm looking for.  Does anyone have any advise on this?</p>
  <p>EDIT:  Can anyone help with this?  Right now, PM2 is not restarting it automatically, so the only way I have to resolve this is by manually restarting each process after it has an error. If I can get it to restart on an 'error; status, or to stagger the restarts so each process does not restart all at once, that would be great.</p>
  <p>I'm getting errors like this in the pm2 log:</p>
  <pre><code>2021-01-17T02:00:11: PM2 log: Process with pid 2627 could not be killed
  2021-01-17T02:00:11: PM2 error: app=testbotserver id=3 pid=2627 could not be stopped
  2021-01-17T02:00:11: PM2 error: Process with pid 2627 already exists
  2021-01-17T02:00:11: PM2 error: Error: Process with pid 2627 already exists
      at Object.God.logAndGenerateError (/home/user/pm2/node_modules/pm2/lib/God/Methods.js:39:12)
      at Object.God.startProcessId (/home/user/pm2/node_modules/pm2/lib/God/ActionMethods.js:295:21)
      at /home/user/pm2/node_modules/pm2/lib/God/ActionMethods.js:439:20
      at /home/user/pm2/node_modules/pm2/lib/God/ActionMethods.js:350:16
      at Timeout._onTimeout (/home/user/pm2/node_modules/pm2/lib/God/Methods.js:176:16)
      at ontimeout (timers.js:482:11)
      at tryOnTimeout (timers.js:317:5)
      at Timer.listOnTimeout (timers.js:277:5)
  </code></pre>