title: How to manage a crawler URL frontier?
body: <p>Guys</p>
  
  <p>I have the following code to add visited links on my crawler.
  After extracting links i have a <strong>for loop</strong> which loop thorough each individual <strong>href tags</strong>.</p>
  
  <p>And after i have visited a link , opened it , i will add the URL to  a visited link collection variable defined above.</p>
  
  <pre><code>private final Collection&lt;String&gt; urlForntier = Collections.synchronizedSet(new HashSet&lt;String&gt;()); 
  </code></pre>
  
  <p>The crawler implementation is mulithread and assume if i have visited 100,000 urls, if i didn't terminate the crawler it will grow day by day . and It will create memory issues ? Please , what option do i have to refresh the variable without creating inconsistency across threads ?</p>
  
  <p>Thanks in advance!</p>