title: App Engine's URLFetch: http GET works locally but not when deployed to GAE on a specific URL
body: <p><strong>The Problem:</strong> </p>
  
  <p>Locally my application works fine. My HTTP GET returns code 200 and a final URL of: <a href="http://vow.mlspin.com/clients/index.aspx" rel="nofollow noreferrer">http://vow.mlspin.com/clients/index.aspx</a>?</p>
  
  <p>The rest of my application works great too.</p>
  
  <p>When I deploy my code to GAE servers (I use the Eclipse plugin to do my deployment) my application ceases to work because the wrong html page is returned!
  the return code is still 200, but the URL is now (final URL): <a href="http://vow.mlspin.com/clients/signin.aspx?id=" rel="nofollow noreferrer">http://vow.mlspin.com/clients/signin.aspx?id=</a></p>
  
  <p><strong>My Question is:</strong> Is there a problem with redirects? Are the google app engine servers somehow blacklisted? What am I doing wrong here? Has anyone encountered this before?</p>
  
  <p>The closest question I've found was this one: <a href="https://stackoverflow.com/questions/8412685/http-get-from-gae-java">Http GET from GAE Java</a> I've implemented this suggestion but it hasn't worked for me so far.</p>
  
  <p>Thank you all in advance!</p>
  
  <p>Additional Info -> below are HTTPResponse headers from the same exact GET request, one from the local deployment and the other from the GAE deployment.</p>
  
  <p><strong>Local HTTP Response Headers</strong></p>
  
  <pre><code>Date :: Tue, 24 Apr 2012 04:12:32 GMT
  Server :: Microsoft-IIS/6.0
  X-Powered-By :: ASP.NET
  X-AspNet-Version :: 2.0.50727
  P3P :: CP="NOI DSP COR NID ADMa OPTa OUR NOR"
  Cache-Control :: no-cache
  Pragma :: no-cache
  Expires :: -1
  Content-Type :: text/html; charset=utf-8
  Content-Length :: 133704
  </code></pre>
  
  <p><strong>Deployed HTTP Response Headers</strong></p>
  
  <pre><code>date :: Tue, 24 Apr 2012 04:11:19 GMT
  server :: Microsoft-IIS/6.0
  x-powered-by :: ASP.NET
  x-aspnet-version :: 2.0.50727
  p3p :: CP="NOI DSP COR NID ADMa OPTa OUR NOR"
  cache-control :: private
  content-type :: text/html; charset=utf-8
  content-length :: 4991
  x-google-cache-control :: remote-fetch
  via :: HTTP/1.1 GWA
  </code></pre>
  
  <p><strong>How I'm crafting my requests:</strong></p>
  
  <p>First I tried the easy way</p>
  
  <pre><code>Document doc = Jsoup.connect(baseMLSURL).get();
  </code></pre>
  
  <p>then I tried to go low level and just use java.net</p>
  
  <pre><code>private String getHttpFromServer(String url) throws IOException
      log.severe("getting http from: "+ url);
      StringBuilder sb = new StringBuilder();
      URL yahoo = new URL(url);
      URLConnection yc = yahoo.openConnection();
      yc.setRequestProperty("Host", "vow.mlspin.com");
      yc.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1");
      BufferedReader in = new BufferedReader(
                              new InputStreamReader(
                              yc.getInputStream()));
      String inputLine;
  
      while ((inputLine = in.readLine()) != null) {
          sb.append(inputLine.replaceAll("&amp;nbsp", " ")+"\r\n");
      }
      in.close();
  
      return sb.toString();
  }
  </code></pre>
  
  <p>Finally I've also tried to use Google's URLFetcher </p>
  
  <pre><code>private String getHttpUsingFetchService(String url) throws MalformedURLException, IOException {
      URLFetchService fetchService = URLFetchServiceFactory.getURLFetchService();
      HTTPResponse targetResponse = fetchService.fetch(new URL(url)); // Error
      log.severe("Code returned from request: "+targetResponse.getResponseCode());
      log.severe("final URL: "+targetResponse.getFinalUrl());
      String result = new String(targetResponse.getContent());
      return result.replaceAll("&amp;nbsp;", " ");
  }
  </code></pre>