title: Java Webcrawler to extract emails
body: <p>I want to write a web crawler that starts at one page and goes to each link on that page looking for an email address. This is what I have so far, but it's not doing anything other than going from webpage to webpage. </p>
  
  <pre><code>`package com.netinstructions.crawler;
  
  import java.util.HashSet;
  import java.util.LinkedList;
  import java.util.List;
  import java.util.Set;
  
  public class WebCrawler {
  
      private static final int MAX_PAGES_TO_SEARCH = 26;
      private Set&lt;String&gt; pagesVisited = new HashSet&lt;String&gt;();
      private List&lt;String&gt; pagesToVisit = new LinkedList&lt;String&gt;();
      private List&lt;String&gt; emails = new LinkedList&lt;&gt;();
  
  private String nextUrl()
  {
      String nextUrl;
      do
      {
          nextUrl = this.pagesToVisit.remove(0);
      } while(this.pagesVisited.contains(nextUrl));
      this.pagesVisited.add(nextUrl);
      return nextUrl;
  }
  
  public void search(String url, String searchWord)
  {
      while(this.pagesVisited.size() &lt; MAX_PAGES_TO_SEARCH)
      {
          String currentUrl;
          SpiderLeg leg = new SpiderLeg();
          if(this.pagesToVisit.isEmpty())
          {
              currentUrl = url;
              this.pagesVisited.add(url);
          }
          else
          {
              currentUrl = this.nextUrl();
          }
          leg.crawl(currentUrl); // Lots of stuff happening here. Look at the crawl method in
          // SpiderLeg
          leg.searchForWord(currentUrl, emails);
          this.pagesToVisit.addAll(leg.getLinks());
          this.pagesToVisit.addAll(leg.getLinks());
      }
      System.out.println(emails.toString());
      //System.out.println(String.format("**Done** Visited %s web page(s)", this.pagesVisited.size()));
  }
  }
  </code></pre>
  
  <p>And this is my Spider Leg Class</p>
  
  <pre><code>package com.netinstructions.crawler;
  
  import java.io.IOException;
  import java.util.LinkedList;
  import java.util.List;
  import java.util.regex.Matcher;
  import java.util.regex.Pattern;
  
  import org.jsoup.Connection;
  import org.jsoup.Jsoup;
  import org.jsoup.nodes.Document;
  import org.jsoup.nodes.Element;
  import org.jsoup.select.Elements;
  
  public class SpiderLeg
  {
  // We'll use a fake USER_AGENT so the web server thinks the robot is a normal web browser.
  private static final String USER_AGENT =
          "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1";
  private List&lt;String&gt; links = new LinkedList&lt;String&gt;();
  private Document htmlDocument;
  
  
  /**
   * This performs all the work. It makes an HTTP request, checks the response, and then gathers
   * up all the links on the page. Perform a searchForWord after the successful crawl
   *
   * @param url
   *            - The URL to visit
   * @return whether or not the crawl was successful
   */
  public boolean crawl(String url)
  {
      try
      {
          Connection connection = Jsoup.connect(url).userAgent(USER_AGENT);
          Document htmlDocument = connection.get();
          this.htmlDocument = htmlDocument;
          if(connection.response().statusCode() == 200) // 200 is the HTTP OK status code
          // indicating that everything is great.
          {
              System.out.println("\n**Visiting** Received web page at " + url);
          }
          if(!connection.response().contentType().contains("text/html"))
          {
              System.out.println("**Failure** Retrieved something other than HTML");
              return false;
          }
          Elements linksOnPage = htmlDocument.select("a[href]");
          //System.out.println("Found (" + linksOnPage.size() + ") links");
          for(Element link : linksOnPage)
          {
              this.links.add(link.absUrl("href"));
          }
          return true;
      }
      catch(IOException ioe)
      {
          // We were not successful in our HTTP request
          return false;
      }
  }
  
  
  /**
   * Performs a search on the body of on the HTML document that is retrieved. This method should
   * only be called after a successful crawl.
   *
   * @param searchWord
   *            - The word or string to look for
   * @return whether or not the word was found
   */
  public void searchForWord(String searchWord, List&lt;String&gt; emails)
  {
  
      if(this.htmlDocument == null)
      {
          System.out.println("ERROR! Call crawl() before performing analysis on the document");
          //return false;
      }
      Pattern pattern =
              Pattern.compile("\"^[A-Z0-9._%+-]+@[A-Z0-9.-]+\\\\.[A-Z]{2,6}$\", Pattern.CASE_INSENSITIVE");
  
      Matcher matchs = pattern.matcher(searchWord);
  
      while (matchs.find()) {
          System.out.println(matchs.group());
      }
  }
  
  
  public List&lt;String&gt; getLinks()
  {
      return this.links;
  }
  
  }
  </code></pre>
  
  <p>My web crawler was taken from another source and I changed a few things. I added a List to hold the emails and return them all in a list to me. I think I am going wrong in my way that I take the email and put it in the list, but I am not sure how to fix it.</p>