title: Where do I have memory leaks and how to fix it? Why memory consumption increases?
body: <p>I am struggling a few days already with a problem of growing memory consumption by console application in .Net Core 2.2, and just now I ran out of ideas what else I could improve.</p>
  <p>Im my application I have a method that triggers <code>StartUpdatingAsync</code> method:</p>
  <pre><code>public MenuViewModel()
          {
              if (File.Exists(_logFile))
                  File.Delete(_logFile);
  
              try
          {
              StartUpdatingAsync(&quot;basic&quot;).GetAwaiter().GetResult();
          }
          catch (ArgumentException aex)
          {
              Console.WriteLine($&quot;Caught ArgumentException: {aex.Message}&quot;);
          }
  
              Console.ReadKey();
          }
  </code></pre>
  <p><code>StartUpdatingAsync</code> creates 'repo' and instance is getting from DB a list of objects to be updated (around 200k):</p>
  <pre><code>private async Task StartUpdatingAsync(string dataType)
          {
              _repo = new DataRepository();
              List&lt;SomeModel&gt; some_list = new List&lt;SomeModel&gt;();
              some_list = _repo.GetAllToBeUpdated();
  
              await IterateStepsAsync(some_list, _step, dataType);
          }
  </code></pre>
  <p>And now, within <code>IterateStepsAsync</code> we are getting updates, parsing them with existing data and updating DB. Inside of each <code>while</code> I was creating new instances of all new classes and lists, to be sure that old ones are releasing memory, but it didnt help. Also I was <code>GC.Collect()</code> at the end of the method, what also is not helping. Please note, that method below triggers lots of parralel <strong>Tasks</strong>, but they supposed to be disposed within it, am I right?:</p>
  <pre><code>private async Task IterateStepsAsync(List&lt;SomeModel&gt; some_list, int step, string dataType)
          {
              List&lt;Area&gt; areas = _repo.GetAreas();
              int counter = 0;
  
              while (counter &lt; some_list.Count)
              {
                  _repo = new DataRepository();
                  _updates = new HttpUpdates();
                  List&lt;Task&gt; tasks = new List&lt;Task&gt;();
                  List&lt;VesselModel&gt; vessels = new List&lt;VesselModel&gt;();
                  SemaphoreSlim throttler = new SemaphoreSlim(_degreeOfParallelism);
  
                  for (int i = counter; i &lt; step; i++)
                  {
                      int iteration = i;
                      bool skip = false;
  
                      if (dataType == &quot;basic&quot; &amp;&amp; (some_list[iteration].Mmsi == 0 || !some_list[iteration].Speed.HasValue)) //if could not be parsed with &quot;full&quot;
                          skip = true;
  
                      tasks.Add(Task.Run(async () =&gt;
                      {
                          string updated= &quot;&quot;;
                          await throttler.WaitAsync();
                          try
                          {
                              if (!skip)
                              {
                                  Model model= await _updates.ScrapeSingleModelAsync(some_list[iteration].Mmsi);
                                  while (Updating)
                                  {
                                      await Task.Delay(1000);
                                  }
                                  if (model != null)
                                  {
                                      lock (((ICollection)vessels).SyncRoot)
                                      {
                                          vessels.Add(model);
  
                                          scraped = BuildData(model);
                                      }
                                  }
                              }
                              else
                              {
                                  //do nothing
                              }
                          }
                          catch (Exception ex)
                          {
                              Log(&quot;Scrape error: &quot; + ex.Message);
                          }
                          finally
                          {
                              while (Updating)
                              {
                                  await Task.Delay(1000);
                              }
                              Console.WriteLine(&quot;Updates for &quot; + counter++ + &quot; of &quot; + some_list.Count + scraped);
  
                              throttler.Release();
                          }
  
                      }));
                  }
  
                  try
                  {
                      await Task.WhenAll(tasks);
                  }
                  catch (Exception ex)
                  {
                      Log(&quot;Critical error: &quot; + ex.Message);
                  }
                  finally
                  {
                      _repo.UpdateModels(vessels, dataType, counter, some_list.Count, _step);
  
                      step = step + _step;
  
                      GC.Collect();
                  }
              }
          }
  </code></pre>
  <p>Inside of the method above, we are calling <code>_repo.UpdateModels</code>, where is updated DB. I tryed two approaches, with using <strong>EC Core</strong> and <strong>SqlConnection</strong>. Both with similar results. Below you can find both of them.</p>
  <p><strong>EF Core</strong></p>
  <pre><code>internal List&lt;VesselModel&gt; UpdateModels(List&lt;Model&gt; vessels, string dataType, int counter, int total, int _step)
          {
  
              for (int i = 0; i &lt; vessels.Count; i++)
              {
                  Console.WriteLine(&quot;Parsing &quot; + i + &quot; of &quot; + vessels.Count);
  
                  Model existing = _context.Vessels.Where(v =&gt; v.id == vessels[i].Id).FirstOrDefault();
                  if (vessels[i].LatestActivity.HasValue)
                  {
                      existing.LatestActivity = vessels[i].LatestActivity;
                  }
                  //and similar parsing several times, as above
              }
  
              Console.WriteLine(&quot;Saving ...&quot;);
              _context.SaveChanges();
              return new List&lt;Model&gt;(_step);
          }
  </code></pre>
  <p><strong>SqlConnection</strong></p>
  <pre><code>internal List&lt;VesselModel&gt; UpdateModels(List&lt;Model&gt; vessels, string dataType, int counter, int total, int _step)
          {
              if (vessels.Count &gt; 0)
              {
                  using (SqlConnection connection = GetConnection(_connectionString))
                  using (SqlCommand command = connection.CreateCommand())
                  {
                      connection.Open();
                      StringBuilder querySb = new StringBuilder();
  
                      for (int i = 0; i &lt; vessels.Count; i++)
                      {
                          Console.WriteLine(&quot;Updating &quot; + i + &quot; of &quot; + vessels.Count);
                          //PARSE
  
                          VesselAisUpdateModel existing = new VesselAisUpdateModel();
  
                          if (vessels[i].Id &gt; 0)
                          {
                              //find existing
                          }
  
                          if (existing != null)
                          {
                              //update for basic data
                              querySb.Append(&quot;UPDATE dbo.&quot; + _vesselsTableName + &quot; SET Id = '&quot; + vessels[i].Id+ &quot;'&quot;);
                              if (existing.Mmsi == 0)
                              {
                                  if (vessels[i].MMSI.HasValue)
                                  {
                                      querySb.Append(&quot; , MMSI = '&quot; + vessels[i].MMSI + &quot;'&quot;);
                                  }
                              }
                              //and similar parsing several times, as above
  
                              querySb.Append(&quot; WHERE Id= &quot; + existing.Id+ &quot;; &quot;);
  
                              querySb.AppendLine();
                          }
                      }
  
                      try
                      {
                          Console.WriteLine(&quot;Sending SQL query to &quot; + counter);
                          command.CommandTimeout = 3000;
                          command.CommandType = CommandType.Text;
                          command.CommandText = querySb.ToString();
                          command.ExecuteNonQuery();
                      }
                      catch (Exception ex)
                      {
                          Console.WriteLine(ex.Message);
                      }
                      finally
                      {
                          connection.Close();
                      }
                  }
              }
              return new List&lt;Model&gt;(_step);
          }
  </code></pre>
  <p>Main problem is, that after tenths/hundreds of thousands of updated models my console application memory consumption increases continuously. And I have no idea why.</p>
  <p><strong>SOLUTION</strong> my problem was inside of <code>ScrapeSingleModelAsync</code> method, where I was using incorrectly <code>HtmlAgilityPack</code>, what I could debug thanks to <em>cassandrad</em>.</p>