title: Web Crawler specifically for downloading images and files
body: <p>I am doing an assignment for one of my classes. </p>
  
  <p>I am supposed to write a  webcrawler that download files and images from a website given a specified crawl depth. </p>
  
  <p>I am allowed to use third party parsing api so I am using <strong>Jsoup</strong>. I've also tried <strong>htmlparser</strong>. Both nice softwares but they are not perfect. </p>
  
  <p>I used the <strong>default java URLConnection</strong> to check content type before processing the url but it becomes really slow as the number of links grows. </p>
  
  <p><strong>Question : Anyone know any specialized parser api for images and links ?</strong> </p>
  
  <p>I could start writing mine using Jsoup but am being lazy. Besides why reinvent the wheel if there could be a working solution out there? Any help would be appreciated.</p>
  
  <p>i need to check contentType while looping through the links to check if the link is to a file, in an effective way but Jsoup does not have what i need. Heres what i have: 
  **              </p>
  
  <pre><code>    HttpConnection mimeConn =null;
      Response mimeResponse = null;
      for(Element link: links){
  
          String linkurl =link.absUrl("href");
          if(!linkurl.contains("#")){
  
              if(DownloadRepository.curlExists(link.absUrl("href"))){
                  continue;
              }
  
              mimeConn = (HttpConnection) Jsoup.connect(linkurl);
              mimeConn.ignoreContentType(true);
              mimeConn.ignoreHttpErrors(true);
              mimeResponse =(Response) mimeConn.execute();
  
              WebUrl webUrl = new WebUrl(linkurl,currentDepth+1);
              String contentType = mimeResponse.contentType();
  
              if(contentType.contains("html")){
                  page.addToCrawledPages(new WebPage(webUrl));
              }else if(contentType.contains("image")){                    
                  page.addToImages(new WebImage(webUrl));
              }else{
                  page.addToFiles(new WebFile(webUrl));
              }
  
              DownloadRepository.addCrawledURL(linkurl);
  
          }**
  </code></pre>
  
  <p><strong>UPDATE</strong>
  Based on Yoshi's answer, I was able to get my code to work right. Here's the link:</p>
  
  <p><a href="https://github.com/unekwu/cs_nemesis/blob/master/crawler/crawler/src/cu/cs/cpsc215/project1/parser/Parser.java" rel="nofollow">https://github.com/unekwu/cs_nemesis/blob/master/crawler/crawler/src/cu/cs/cpsc215/project1/parser/Parser.java</a></p>