title: Most efficient way to do batch INSERT IGNORE using doctrine 2
body: <p>I have a script that needs go get a list of entries in the database then iterate over those creating new entries in another table if they dont already exists. </p>
  
  <p>Currently im doing:</p>
  
  <pre><code>foreach($entries as $entry){
      $newItem = new Item();
      $newItem-&gt;setAttribute($entry-&gt;getAttribute());
      $entityManager-&gt;persist($newItem);
      try{
          $entityManager-&gt;flush();
      } catch(\Exception $e){
          if(!strpos($e-&gt;getMessage(),'Duplicate')){
              throw $e;
          }
          $entityManager = $this-&gt;getDoctrine()-&gt;getManager(); 
          //refreshes the entity manager
      }
  
  }
  </code></pre>
  
  <p>However doing it this way is very time intensive, there are 1000's of entries and the script some times takes upwards of 10 minutes to complete. I have seen other posts suggest when doing batch processing like this to flush every 20 or so records the problem with that is that if one of those 20 are a duplicate then the whole transaction dies, im not sure how i would go back and try and find the offending entry to exclude it before resubmitting them again.</p>
  
  <p>Any help with this will be greatly appreciated.</p>