title: Python overlapping processes?
body: <p>I've run into some weird behavior and and I am not exactly sure how to address this issue.</p>
  
  <p><strong>Overview:</strong></p>
  
  <p>I have a Linux/Ubuntu server with a API that receives thousands of records an hour. These records are stored in a table in my database, we'll call it a staging table, where the records are processed at a later time.</p>
  
  <p>I run a cron every minute to grab x records from this staging table and process them with a simple python script. </p>
  
  <p><strong>Problem:</strong></p>
  
  <p>Lets say this script can process x records per minute. Then each process finishes before the next one begins. However, if I increase the number of records I'm trying to process to 2x then the cron that fired at time 0 has not finished processing it's batch of records when the next cron that fires at time 1 starts. So now we have two running processes, the cron that fired at time 0 which is almost done and the cron which fired at time 1 which just started. </p>
  
  <p>This is where something unexpected happens. I have a simple function in the script to process and save processed results to another table. When there are two scripts that are overlapping (as described above), these records each get saved twice to the database. If only one process is running it only gets saved once. </p>
  
  <p>I'm scratching my head and not sure how this is possible since each script should be independent. I didn't think it would be useful to include code since it's literally just a cron firing a script to 1. pull records from staging table. 2. make changes to those records 3. save those changed records to new table.</p>
  
  <p>If you think it's helpful, I would be glad to upload code to the question. Can anyone point me in the right direction? </p>
  
  <p>Thanks</p>