title: How do we build a website Crawler using Java
body: <p>posting this question again.I have started with the crawler , but i am stucked with the indexing part.I want an efficient and fast way to index the links.Currently what i am doing is i am inserting the links into the database , but checking for unique links is overhead so can anyone suggest me any better way to do this.</p>
  
  <hr>
  
  <p>Hi I am trying to build a website crawler , which will crawl the whole website and get all of the links within it.Something very similar to "XENU". But I am not able to figure out how to go for it. I have one algorithm in my mind but that would be very slow, it is mention below.</p>
  
  <ol>
  <li>Get the source of home page.     </li>
  <li>Get all the anchor tag from the
      source. </li>
  <li>Get the URLs from the
      anchor tag.</li>
  <li>Check if the url
      belongs to the same site or external
      site.</li>
  <li>Get the source for the urls
      we found in the above process and
      mark those urls as checked. </li>
  <li>Repeat the process until there are
          no un marked urls.</li>
  </ol>
  
  <p>This is some what rough idea of what I came up with. But it will be very slow. So can any one suggest me some other approach or enhance this algorithm.</p>
  
  <p>Regards,
  Sagar.</p>