title: Perl mechanize Find all links array loop issue
body: <p>I am currently attempting to create a <strong>Perl webspider using WWW::Mechanize.</strong></p>
  
  <p>What I am trying to do is <strong>create a webspider that will crawl the whole site</strong> of the URL (entered by the user) and extract all of the links from every page on the site.</p>
  
  <p><strong>But I have a problem with how to spider the whole site to get every link, without duplicates</strong>
  What I have done so far (the part im having trouble with anyway):</p>
  
  <pre><code>foreach (@nonduplicates) {   #array contain urls like www.tree.com/contact-us, www.tree.com/varieties....
  $mech-&gt;get($_);
  my @list = $mech-&gt;find_all_links(url_abs_regex =&gt; qr/^\Q$urlToSpider\E/);  #find all links on this page that starts with http://www.tree.com
  
  #NOW THIS IS WHAT I WANT IT TO DO AFTER THE ABOVE (IN PSEUDOCODE), BUT CANT GET WORKING
  #foreach (@list) {
  #if $_ is already in @nonduplicates
  #then do nothing because that link has already been found
  #} else {
  #append the link to the end of @nonduplicates so that if it has not been crawled for links already, it will be 
  </code></pre>
  
  <p><strong>How would I be able to do the above?</strong></p>
  
  <p><strong>I am doing this to try and spider the whole site to get a comprehensive list of every URL on the site, without duplicates.</strong></p>
  
  <p>If you think this is not the best/easiest method of achieving the same result I'm open to ideas.</p>
  
  <p><strong>Your help is much appreciated, thanks.</strong> </p>