title: C# Massive memory leak in GeckoFX with HtmlAgilityPack
body: <p>I'm writing a program to scrape lots of websites of companies (up to 100,000) for up-to-date contact information as well as some information about their field of operations in C#. Because most of the websites can't be displayed in the regular .NET webbrowser I'm using geckofx to navigate to these websites and to find the content relevant to me I select nodes with HtmlAgilityPack.</p>
  
  <p>The process is always the same: If I have a URL for a company I visit the website right away, otherwise I use bing to look for a web address (Google seems to dislike being used automatically). On the website I look for a link to an imprint and links to pages that could indicate some area of activity, I navigate to these links and look for catchphrases that I specified beforehand. Everything is running synchronously, I wait for the browser to trigger its <code>DocumentCompleted</code> event everytime.</p>
  
  <p>An example:</p>
  
  <pre><code>//I navigate to bing looking for my company's name and postal code
  Variables.browser.Navigate("https://www.bing.com/search?q=" + c.Name.Replace(" ", "+") + "+" + c.Zip.Replace(" ", "+"));
  
  //I wait for the browser to finish loading. The Navigating event sets BrowserIsReady to false and the DocumentCompleted event sets it to true
  do
  {
      f.Application.DoEvents();
  } while (!Variables.BrowserIsReady);
  
  HtmlDocument browserDoc = new HtmlDocument();
  browserDoc.LoadHtml(Variables.browser.Document.Body.OuterHtml);
  
  //I select the relevant node in the document
  HtmlNode sidebarNode = browserDoc.DocumentNode.SelectSingleNode("//div[contains(concat(\" \", normalize-space(@class), \" \"), \" b_entityTP \")]");
  
  if (sidebarNode != null)
  {
      Variables.logger.Log("Found readable sidebar. Loading data...");
      string lookedUpName, lookedUpStreet, lookedUpCity, lookedUpZip, lookedUpPhone, lookedUpWebsite;
  
      HtmlNode infoNode = sidebarNode.SelectSingleNode("//div[contains(concat(\" \", normalize-space(@class), \" \"), \" b_subModule \")]");
  
      HtmlNode nameNode = infoNode.SelectSingleNode("//div[contains(concat(\" \", normalize-space(@class), \" \"), \" b_feedbackComponent \")]");
      if (nameNode != null)
      {
          string[] dataFacts = nameNode.GetAttributeValue("data-facts", "").Replace("{\"", "").Replace("\"}", "").Split(new string[] { "\",\"" }, StringSplitOptions.None);
          foreach (string dataFact in dataFacts)
          {
              //... abbreviated
          }
      }
      //And at the end of every call to a node object I set it back to null
      nameNode = null;
  }
  </code></pre>
  
  <p>My geckofx is not allowed to write cache to memory or to load images from websites, which I set by using</p>
  
  <pre><code>GeckoPreferences.Default["browser.cache.memory.enabled"] = false;
  GeckoPreferences.Default["permissions.default.image"] = 2;
  </code></pre>
  
  <p>before creating my GeckoWebBrowser instance.</p>
  
  <p>After every scraped website I call </p>
  
  <pre><code>//CookieMan is used as a global variable so I don't have to recreate it every time.
  private static nsICookieManager CookieMan;
  //...
  CookieMan = Xpcom.GetService&lt;nsICookieManager&gt;("@mozilla.org/cookiemanager;1");
  CookieMan = Xpcom.QueryInterface&lt;nsICookieManager&gt;(CookieMan);
  CookieMan.RemoveAll();
  Gecko.Cache.ImageCache.ClearCache(true);
  Gecko.Cache.ImageCache.ClearCache(false);
  Xpcom.GetService&lt;nsIMemory&gt;("@mozilla.org/xpcom/memory-service;1").HeapMinimize(true);
  </code></pre>
  
  <p>to delete cookies, image cache (which I'm not sure is even created) and to minimize Xulrunners memory usage.</p>
  
  <p>Nevertheless, after starting quite nicely with an approximate runtime of 2-3 seconds per record and comfortable 200-300mb memory usage, both quickly blow up to 16-17 seconds per record and over 2gb of used memory for my crawler alone after 1 hour.</p>
  
  <p>I tried forcing garbage collection with <code>GC.Collect();</code> (which I know, you're not supposed to do) and even recycling the entire browser object by stopping, disposing and recreating it to try and get rid of unused junk in the memory, but to no avail. I was also trying to shut down the Xulrunner and starting it again, but <code>Xpcom.Shutdown()</code> seems to stop the entire app, so I wasn't able to do that.</p>
  
  <p>I'm pretty much out of ideas at this point and would very much appreciate new hints to approaches I haven't yet taken.</p>