title: Issue with Java web crawler using jsoup
body: <p>I am having issues getting my web crawler class to work for my java class project. Here is the prompt:</p>
  <p>In this project, you are to create a web crawler class.</p>
  <p>Two notes before we discuss what to do with the class:</p>
  <p>Make sure that you sleep for at least 0.5 seconds between hitting each link. This is to make sure that you do not ultimately DDOS any site that you wish to crawl. To get your program to sleep, check out this documentation from oracle: <a href="https://docs.oracle.com/javase/tutorial/essential/concurrency/sleep.html" rel="nofollow noreferrer">https://docs.oracle.com/javase/tutorial/essential/concurrency/sleep.html</a> (Links to an external site.)
  For the sake of not traversing advertisement sites, we will be using wikipedia links.</p>
  <p>This class needs to do two things:</p>
  <p>Have a function that traverses 1000 links.
  Have a function that counts words, that is, every time you see a specific word, increment a number associated with that word. (Hint: Sets may be a good option for this). Note: Your function should count words, not html elements / attributes.</p>
  <p>I do not necessarily care how you implement this project so long as you have a class that, at the end of your 1000 link traversal your class:</p>
  <ul>
  <li><p>prints out the title of each page you've traversed</p>
  </li>
  <li><p>prints out a list of words encountered. Again, html elements / attributes should not be in this list (i.e. no '<p>' or ''s should be in your list).</p>
  </li>
  </ul>
  <hr />
  <p>This is the code I have so far, not exactly sure what am i doing wrong? It gets all kinds of errors (examples below) right off the bat that have to do with datatypes, but I thought I had it all correct. Haven't had luck researching as I am very new to programming.</p>
  <p>//crawler.java</p>
  <pre><code>package edu.umsl
  
  import java.io.IOException;
  import java.util.HashMap;
  import java.util.HashSet;
  import org.jsoup.Jsoup;
  import org.jsoup.nodes.Document;
  import org.jsoup.nodes.Element;
  import org.jsoup.select.Elements;
  
  public class crawler {
      // define max number of pages to visit
      private static final int MAX_PAGES = 1000;
      // collect all titles
      private HashSet&lt;String&gt; titles = new HashSet&lt;&gt;();
      // keep track of url visited
      private HashSet&lt;String&gt; urlVisited = new HashSet&lt;&gt;();
      // keep track of words and count
      private HashMap&lt;String, Integer&gt; map = new HashMap&lt;&gt;();
  
      // recursive function to crawl web
      public void getLinks(String startURL) {
          // condition to end the recursion
          if ((titles.size() &lt; MAX_PAGES) &amp;&amp; !urlVisited.contains(startURL)) {
              // add new url to set
              urlVisited.add(startURL);
              try {
                  Document doc = Jsoup.connect(startURL).get();
                  Elements linksFromPage = doc.select(&quot;a[href]&quot;);
                  // take all text to count words
                  String title = doc.select(&quot;title&quot;).first().text();
                  titles.add(title);
                  String text = doc.body().text();
                  CountWords(text);
                  for (Element link : linksFromPage) {
                      if (titles.size() &lt;= MAX_PAGES) {
                          Thread.sleep(1000);
                          getLinks(link.attr(&quot;abs:href&quot;));
                      } else {
                          System.out.println(&quot;URL couldnt visit&quot;);
                          System.out.println(startURL + &quot;, &quot; + urlVisited.size());
                      }
                  }
              } catch (IOException e) {
                  e.printStackTrace();
              }
  //           catch (InterruptedException e) {
  //               e.printStackTrace();
  //           }
              catch (Exception e) {
                  e.printStackTrace();
              }
          }
      }
  
      // method to print all titles
      public void PrintAllTitles() {
          for (String t : titles) {
              System.out.println(t);
          }
      }
  
      // method to print word and count
      public void PrintAllWordsAndCount() {
          for (String key : map.keySet()) {
              System.out.println(key + &quot; : &quot; + map.get(key));
          }
      }
  
      private void CountWords(String text) {
          String[] lines = text.split(&quot; &quot;);
          for (String word : lines) {
              if (map.containsKey(word)) {
                  int val = map.get(word);
                  val += 1;
                  map.remove(word);
                  map.put(word, val);
              } else {
                  map.put(word, 1);
              }
          }
      }
  }
  </code></pre>
  <p>//driver.java</p>
  <pre><code>package edu.umsl
  
  
  public class driver {
  
     public static void main(String[] args) {
    
         crawler c = new crawler();
         c.getLinks(&quot;https://en.wikipedia.org/wiki/Science&quot;);
         System.out.println(&quot;*******************************Printing all titles*******************************&quot;);
         c.PrintAllTitles();
         System.out.println(&quot;*******************************Printing all Words*******************************&quot;);
         c.PrintAllWordsAndCount();
     }
  
  }
  </code></pre>
  <p>Here is the first error that i get. all of the other errors are the same, with additional repetitions of the line:
  at edu.umsl.Crawler.getLinks(Crawler.java:50)</p>
  <pre><code>org.jsoup.UnsupportedMimeTypeException: Unhandled content type. Must be text/*, application/xml, or application/*+xml. Mimetype=image/svg+xml, URL=https://upload.wikimedia.org/wikipedia/commons/3/37/People_icon.svg
  at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:772)
  at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:707)
  at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:297)
  at org.jsoup.helper.HttpConnection.get(HttpConnection.java:286)
  at edu.umsl.Crawler.getLinks(Crawler.java:33)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Crawler.getLinks(Crawler.java:50)
  at edu.umsl.Driver.main(Driver.java:8)
  </code></pre>
  <p>UPDATE: I had the attribute key in link.atr as &quot;abs:href&quot; instead of &quot;a[href]&quot;</p>
  <p>This solved the error issue! Thanks!</p>