title: Making web crawler download multiple web pages locally
body: <p>I would like my web crawler to download all the browsed URL's locally. At the minute it will download every site it comes to but then overwrite the local file in each website visited. The crawler start at www.bbc.co.uk, downloads that file and then when it hits another it overwrites that file with the next URL. How can I make it download them in to single files so I have a collection at the end? I have this code below but I dont know where to go from here. Any advice would be great. The URL inside the brackets (URL) is a string which is used to manipulate all the browsed webpages. </p>
  
  <pre><code>         URL url = new URL(URL);
               BufferedWriter writer;
                   try (BufferedReader reader = new BufferedReader
                      (new InputStreamReader(url.openStream()))) {
                              writer = new BufferedWriter
                            (new FileWriter("c:/temp/data.html", true));
                              String line;
                              while ((line = reader.readLine()) != null) {
                                  //System.out.println(line);
                              writer.write(line);
                              writer.newLine();
                              }        
                          }
                               writer.close();
  </code></pre>