title: How can I Retrieve the HTML of a Search Engine Query Result?
body: <p>I am trying to retrieve the html of a Google search query result using Java. That is, if I do a search in Google.com for a particular phrase, I would like to retrieve the html of the resulting web page (the page containing the links to possible matches along with their descriptions, URLs, ect…).</p>
  
  <p>I tried doing this using the following code that I found in a related post:</p>
  
  <pre><code>import java.io.*;
  import java.net.*;
  import java.util.*;
  
  public class Main {
  
      public static void main (String args[]) {
  
          URL url;
          InputStream is = null;
          DataInputStream dis;
          String line;
  
          try {
              url = new URL("https://www.google.com/#hl=en&amp;output=search&amp;sclient=psy-ab&amp;q=UCF&amp;oq=UCF&amp;aq=f&amp;aqi=g4&amp;aql=&amp;gs_l=hp.3..0l4.1066.1471.0.1862.3.3.0.0.0.0.382.1028.2-1j2.3.0...0.0.OxbV2LOXcaY&amp;pbx=1&amp;bav=on.2,or.r_gc.r_pw.r_cp.r_qf.,cf.osb&amp;fp=579625c09319dd01&amp;biw=944&amp;bih=951");
              is = url.openStream();  // throws an IOException
              dis = new DataInputStream(new BufferedInputStream(is));
  
              while ((line = dis.readLine()) != null) {
                  System.out.println(line);
              }
          } catch (MalformedURLException mue) {
               mue.printStackTrace();
          } catch (IOException ioe) {
               ioe.printStackTrace();
          } finally {
              try {
                  is.close();
              } catch (IOException ioe ) {
                  // nothing to see here
              }
          }
      }
  } 
  </code></pre>
  
  <p>From: <a href="https://stackoverflow.com/questions/238547/how-do-you-programmatically-download-a-webpage-in-java">How do you Programmatically Download a Webpage in Java</a></p>
  
  <p>The URL used in this code was obtained by doing a Google search query from the Google homepage. For some reason I do not understand, if I write the phrase that I want to search for in the URL bar of my web browser and then use the URL of the resulting search result page in the code I get a 403 error.</p>
  
  <p>This code, however, did not return the html of the search query result page. Instead, it returned the source code of the Google homepage.</p>
  
  <p>After doing further research I noticed that if you view the source code of a Google search query result (by right clicking on the background of the search result page and selecting “View page source”) and compare it with the source code of the Google homepage, they are both identical.</p>
  
  <p>If instead of viewing the source code of the search result page I save the html of the search result page (by pressing ctrl+s), I can get the html that I am looking for.</p>
  
  <p>Is there a way to retrieve the html of the search result page using Java?</p>
  
  <p>Thank you!</p>