title: Improve memory usage when importing data
body: <p><em>I'm actually searching for an advice, not a solution.</em></p>
  
  <p>I made a service that compare data from a distant database and insert, update, or remove local data if necessary.
  There are more than 50k entries in the source database, and local memory usage slow the script make it veeeery long.</p>
  
  <p>Here is part of the service :</p>
  
  <pre><code>function compare( $uosLocal, $uosDistant ) {
  
      $idsLocal = array_keys($uosLocal);
      $idsDistant = array_keys($uosDistant);
  
      $toInsert   = array_diff( $idsDistant, $idsLocal );
      $toDisable  = array_diff( $idsLocal, $idsDistant );
  
  
      $this-&gt;insert( array_map( function ($uoid) use ($uosDistant) {
          return ($uosDistant[$uoid]);
      }, $toInsert )  );
  
      $this-&gt;delete( array_map( function ($uoid) use ($uosLocal) {
          return ($uosLocal[$uoid]);
      }, $toDisable )  );
  
      $uosLocal = $this-&gt;getLocal();
      $uosDistant = $this-&gt;getDistant();
      $idsLocal = array_keys($uosLocal);
      $idsDistant = array_keys($uosDistant);
  
      $toUpdate   = array_intersect( $idsDistant, $idsLocal );
  
      $this-&gt;update( $toUpdate, $uosDistant, $uosLocal  );
  
  
  
  }
  
  function insert( $array_uos ) {
  
      $em = $this-&gt;doctrine-&gt;getManager();
  
      $count = 0;
      $total = count($array_uos);
      $precision = 2;
      $suffixes = array('', 'k', 'M', 'G', 'T');
  
      if ($total &gt; 0)
          $this-&gt;output-&gt;writeln(" $total to insert : ");
  
      foreach ($array_uos as $simple_uo) {
  
          $uo = new UO();
          $uo
              -&gt;setName($simple_uo['NAME'])
              -&gt;setId($simple_uo['ID'])
          ;
  
          $em-&gt;persist($uo);
          unset($uo);
  
          $count++;
  
          if ($count % 100 === 0) {
  
              $base = log(memory_get_usage(), 1024);
  
              $em-&gt;flush();
              $this-&gt;output-&gt;writeln('['. (new \DateTime())-&gt;format('H:i:s') .']     ' . floor( $count / $total * 100 *100) /100 . "% ($count / $total)  (memory : " . round(pow(1024, $base - floor($base)), $precision) . $suffixes[floor($base)] . ") ");
  
          }
  
      }
  
      $em-&gt;flush();
  
      $this-&gt;output-&gt;writeln( " $count was inserted " );
  
  }
  </code></pre>
  
  <p>It's the <code>insert</code> method that causes huge memory usage.</p>
  
  <p>Take a look to logs : </p>
  
  <pre><code>Starting data extraction from referentiel :  56215 to insert :
  [09:49:55]     0.17% (100 / 56215)  (memory : 55.37M)
  [09:49:55]     0.35% (200 / 56215)  (memory : 56.43M)
  [09:49:55]     0.53% (300 / 56215)  (memory : 56.98M)
  [09:49:56]     0.71% (400 / 56215)  (memory : 57.47M)
  [09:49:56]     0.88% (500 / 56215)  (memory : 57.95M)
  [09:49:57]     1.06% (600 / 56215)  (memory : 58.45M)
  [09:49:57]     1.24% (700 / 56215)  (memory : 59.11M)
  [09:49:57]     1.42% (800 / 56215)  (memory : 59.59M)
  [09:49:58]     1.6% (900 / 56215)  (memory : 60.07M)
  [09:49:58]     1.77% (1000 / 56215)  (memory : 60.56M)
  [09:49:59]     1.95% (1100 / 56215)  (memory : 61.09M)
  [09:50:00]     2.13% (1200 / 56215)  (memory : 61.59M)
  [09:50:00]     2.31% (1300 / 56215)  (memory : 62.09M)
  [09:50:01]     2.49% (1400 / 56215)  (memory : 62.83M)
  [09:50:01]     2.66% (1500 / 56215)  (memory : 63.32M)
  [09:50:02]     2.84% (1600 / 56215)  (memory : 63.82M)
  [09:50:03]     3.02% (1700 / 56215)  (memory : 64.32M)
  [09:50:03]     3.2% (1800 / 56215)  (memory : 64.81M)
  [09:50:04]     3.37% (1900 / 56215)  (memory : 65.3M)
  [09:50:05]     3.55% (2000 / 56215)  (memory : 65.79M)
  [09:50:05]     3.73% (2100 / 56215)  (memory : 66.31M)
  [09:50:06]     3.91% (2200 / 56215)  (memory : 66.84M)
  [09:50:07]     4.09% (2300 / 56215)  (memory : 67.33M)
  [09:50:08]     4.26% (2400 / 56215)  (memory : 67.82M)
  [09:50:09]     4.44% (2500 / 56215)  (memory : 68.31M)
  [09:50:10]     4.62% (2600 / 56215)  (memory : 68.81M)
  [09:50:10]     4.8% (2700 / 56215)  (memory : 69.8M)
  [09:50:11]     4.98% (2800 / 56215)  (memory : 70.28M)
  [09:50:12]     5.15% (2900 / 56215)  (memory : 70.78M)
  [09:50:13]     5.33% (3000 / 56215)  (memory : 71.27M)
  [09:50:14]     5.51% (3100 / 56215)  (memory : 71.77M)
  [09:50:15]     5.69% (3200 / 56215)  (memory : 72.26M)
  [09:50:16]     5.87% (3300 / 56215)  (memory : 72.75M)
  [09:50:17]     6.04% (3400 / 56215)  (memory : 73.24M)
  [09:50:18]     6.22% (3500 / 56215)  (memory : 73.74M)
  [09:50:20]     6.4% (3600 / 56215)  (memory : 74.23M)
  [09:50:21]     6.58% (3700 / 56215)  (memory : 74.72M)
  [09:50:22]     6.75% (3800 / 56215)  (memory : 75.21M)
  [09:50:23]     6.93% (3900 / 56215)  (memory : 75.7M)
  [09:50:24]     7.11% (4000 / 56215)  (memory : 76.2M)
  [09:50:26]     7.29% (4100 / 56215)  (memory : 76.73M)
  [09:50:27]     7.47% (4200 / 56215)  (memory : 77.3M)
  [09:50:28]     7.64% (4300 / 56215)  (memory : 77.79M)
  </code></pre>
  
  <p>Is it normal, an can it be improve ?</p>
  
  <p>Thanks alot!</p>