title: How to read html content from this page with Java?
body: <p>My Java app is trying to read content from the following url : <a href="https://www.iplocation.net/?query=62.92.63.48" rel="nofollow noreferrer">https://www.iplocation.net/?query=62.92.63.48</a></p>
  <p>I used the following method :</p>
  <pre><code>  StringBuffer readFromUrl(String Url)
    {
      StringBuffer sb=new StringBuffer();
      BufferedReader in=null;
      
      try
      {
        in=new BufferedReader(new InputStreamReader(new URL(Url).openStream()));
        String inputLine;
      
        while ((inputLine=in.readLine()) != null) sb.append(inputLine+&quot;\n&quot;);
        in.close();
      }
      catch (Exception e) { e.printStackTrace(); }
      finally 
      {
        try 
        {
          if (in!=null)
          {
            in.close();
            in=null;
          }
        }
        catch (Exception ex) { ex.printStackTrace(); }
      }
      return sb;
    }
  </code></pre>
  <p>Usually it works fine for other urls, but for this one, the result is different from what's showing in a browser, it looks like this :</p>
  <pre><code>&lt;html&gt;
  &lt;head&gt;
  &lt;META NAME=&quot;robots&quot; CONTENT=&quot;noindex,nofollow&quot;&gt;
  &lt;script&gt;
  (function(){function getSessionCookies(){var cookieArray=new Array();var cName=/^\s?incap_ses_/;var c=document.cookie.split(&quot;;&quot;);for(var i=0;i&lt;c.length;i++){var key=c[i].substr(0,c[i].indexOf(&quot;=&quot;));var value=c[i].substr(c[i].indexOf(&quot;=&quot;)+1,c[i].length);if(cName.test(key)){cookieArray[cookieArray.length]=value}}return cookieArray}function setIncapCookie(vArray){var res;try{var cookies=getSessionCookies();var digests=new Array(cookies.length);for(var i=0;i&lt;cookies.length;i++){digests[i]=simpleDigest((vArray)+cookies[i])}res=vArray+&quot;,digest=&quot;+(digests.join())}catch(e){res=vArray+&quot;,digest=&quot;+(encodeURIComponent(e.toString()))}createCookie(&quot;___utmvc&quot;,res,20)}function simpleDigest(mystr){var res=0;for(var i=0;i&lt;mystr.length;i++){res+=mystr.charCodeAt(i)}return res}function createCookie(name,value,seconds){var expires=&quot;&quot;;if(seconds){var date=new Date();date.setTime(date.getTime()+(seconds*1000));var expires=&quot;; expires=&quot;+date.toGMTString()}document.cookie=name+&quot;=&quot;+value+expires+&quot;; path=/&quot;}function test(o){var res=&quot;&quot;;var vArray=new Array();for(var j=0;j&lt;o.length;j++){var test=o[j][0];switch(o[j][1]){case&quot;exists&quot;:try{if(typeof(eval(test))!=&quot;undefined&quot;){vArray[vArray.length]=encodeURIComponent(test+&quot;=true&quot;)}else{vArray[vArray.length]=encodeURIComponent(test+&quot;=false&quot;)}}catch(e){vArray[vArray.length]=encodeURIComponent(test+&quot;=false&quot;)}break;case&quot;value&quot;:try{try{res=eval(test);if(typeof(res)===&quot;undefined&quot;){vArray[vArray.length]=encodeURIComponent(test+&quot;=undefined&quot;)}else if(res===null){vArray[vArray.length]=encodeURIComponent(test+&quot;=null&quot;)}else{vArray[vArray.length]=encodeURIComponent(test+&quot;=&quot;+res.toString())}}catch(e){vArray[vArray.length]=encodeURIComponent(test+&quot;=cannot evaluate&quot;);break}break}catch(e){vArray[vArray.length]=encodeURIComponent(test+&quot;=&quot;+e)}case&quot;plugin_extentions&quot;:try{var extentions=[];try{i=extentions.indexOf(&quot;i&quot;)}catch(e){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=indexOf is not a function&quot;);break}try{var num=navigator.plugins.length if(num==0||num==null){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=no plugins&quot;);break}}catch(e){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=cannot evaluate&quot;);break}for(var i=0;i&lt;navigator.plugins.length;i++){if(typeof(navigator.plugins[i])==&quot;undefined&quot;){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=plugins[i] is undefined&quot;);break}var filename=navigator.plugins[i].filename var ext=&quot;no extention&quot;;if(typeof(filename)==&quot;undefined&quot;){ext=&quot;filename is undefined&quot;}else if(filename.split(&quot;.&quot;).length&gt;1){ext=filename.split('.').pop()}if(extentions.indexOf(ext)&lt;0){extentions.push(ext)}}for(i=0;i&lt;extentions.length;i++){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=&quot;+extentions[i])}}catch(e){vArray[vArray.length]=encodeURIComponent(&quot;plugin_ext=&quot;+e)}break}}vArray=vArray.join();return vArray}var o=[[&quot;navigator&quot;,&quot;exists&quot;],[&quot;navigator.vendor&quot;,&quot;value&quot;],[&quot;navigator.appName&quot;,&quot;value&quot;],[&quot;navigator.plugins.length==0&quot;,&quot;value&quot;],[&quot;navigator.platform&quot;,&quot;value&quot;],[&quot;navigator.webdriver&quot;,&quot;value&quot;],[&quot;platform&quot;,&quot;plugin_extentions&quot;],[&quot;ActiveXObject&quot;,&quot;exists&quot;],[&quot;webkitURL&quot;,&quot;exists&quot;],[&quot;_phantom&quot;,&quot;exists&quot;],[&quot;callPhantom&quot;,&quot;exists&quot;],[&quot;chrome&quot;,&quot;exists&quot;],[&quot;yandex&quot;,&quot;exists&quot;],[&quot;opera&quot;,&quot;exists&quot;],[&quot;opr&quot;,&quot;exists&quot;],[&quot;safari&quot;,&quot;exists&quot;],[&quot;awesomium&quot;,&quot;exists&quot;],[&quot;puffinDevice&quot;,&quot;exists&quot;],[&quot;navigator.cpuClass&quot;,&quot;exists&quot;],[&quot;navigator.oscpu&quot;,&quot;exists&quot;],[&quot;navigator.connection&quot;,&quot;exists&quot;],[&quot;window.outerWidth==0&quot;,&quot;value&quot;],[&quot;window.outerHeight==0&quot;,&quot;value&quot;],[&quot;window.WebGLRenderingContext&quot;,&quot;exists&quot;],[&quot;document.documentMode&quot;,&quot;value&quot;],[&quot;eval.toString().length&quot;,&quot;value&quot;]];try{setIncapCookie(test(o));document.createElement(&quot;img&quot;).src=&quot;/_Incapsula_Resource?SWKMTFSR=1&amp;e=&quot;+Math.random()}catch(e){img=document.createElement(&quot;img&quot;);img.src=&quot;/_Incapsula_Resource?SWKMTFSR=1&amp;e=&quot;+e}})();
  &lt;/script&gt;
  &lt;script&gt;
  (function() { 
  var z=&quot;&quot;;var b=&quot;7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D2273746128......6F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B&quot;;for (var i=0;i&lt;b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+&quot;,&quot;;}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();
  &lt;/script&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;iframe style=&quot;display:none;visibility:hidden;&quot; src=&quot;//content.incapsula.com/jsTest.html&quot; id=&quot;gaIframe&quot;&gt;&lt;/iframe&gt;
  &lt;/body&gt;&lt;/html&gt;
  </code></pre>
  <p>So what's the proper way to read the html content that shows up in the browser, in this case ?</p>
  <p>Edit : After reading the suggestions, I've updated my program to look like the following :</p>
  <pre><code>StringBuilder response=new StringBuilder();
  String USER_AGENT=&quot;Mozilla/5.0&quot;,inputLine;
  BufferedReader in=null;    
  
  try
  {
    HttpURLConnection con=(HttpURLConnection)new URL(Url).openConnection();
    con.setRequestMethod(&quot;GET&quot;);
    con.setRequestProperty(&quot;Accept-Charset&quot;,&quot;UTF-8&quot;);
    con.setRequestProperty(&quot;User-Agent&quot;,USER_AGENT);                         // Add request header
  
    int responseCode=con.getResponseCode();
    in=new BufferedReader(new InputStreamReader(con.getInputStream()));
    while ((inputLine=in.readLine())!=null) { response.append(inputLine); }
    in.close();
  }
  catch (Exception e) { e.printStackTrace(); }
  finally 
  {
    try { if (in!=null) in.close(); }
    catch (Exception ex) { ex.printStackTrace(); }
  }
  return response.toString();
  </code></pre>
  <p>Yet still didn't work, the response I got look like this :</p>
  <pre><code>&lt;html style=&quot;height:100%&quot;&gt;&lt;head&gt;&lt;META NAME=&quot;ROBOTS&quot; CONTENT=&quot;NOINDEX, NOFOLLOW&quot;&gt;&lt;meta name=&quot;format-detection&quot; content=&quot;telephone=no&quot;&gt;&lt;meta name=&quot;viewport&quot; content=&quot;initial-scale=1.0&quot;&gt;&lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=edge,chrome=1&quot;&gt;&lt;/head&gt;&lt;body style=&quot;margin:0px;height:100%&quot;&gt;&lt;iframe src=&quot;/_Incapsula_Resource?CWUDNSAI=24&amp;xinfo=8-75933493-0 0NNN RT(1479758027223 127) q(0 -1 -1 -1) r(0 -1) B12(4,315,0) U10000&amp;incident_id=516000100118713619-514529209419563176&amp;edet=12&amp;cinfo=04000000&quot; frameborder=0 width=&quot;100%&quot; height=&quot;100%&quot; marginheight=&quot;0px&quot; marginwidth=&quot;0px&quot;&gt;Request unsuccessful. Incapsula incident ID: 516000100118713619-514529209419563176&lt;/iframe&gt;&lt;/body&gt;&lt;/html&gt;
  </code></pre>
  <p>Could someone show some sample code that works ?</p>
  <p>Thanks to @thatguy I've modified my program to look like the following :</p>
  <pre><code>import java.util.*;
  import java.util.concurrent.*;
  import java.io.*;
  import java.net.*;
  import java.util.Map.Entry;
  
  public class Read_From_Url_Runner implements Callable&lt;String[]&gt;
  {
    int Id;
    String Read_From_Url_Result[]=null,IP_Location_Url=&quot;https://www.iplocation.net/?query=[IP]&quot;,IP=&quot;62.92.63.48&quot;,Cookie,Result[],A_Url;
    
    public Read_From_Url_Runner(int Id)
    {
      this.Id=Id;
      
      A_Url=IP_Location_Url.replace(&quot;[IP]&quot;,IP);
      Cookie=getIncapsulaCookie(A_Url);
      Out(&quot;Cookie = [ &quot;+Cookie+&quot; ]&quot;);
      
      try
      {
        Result=call();
  //      for (int i=0;i&lt;Result.length;i++) Out(Result[i]);
      }
      catch (Exception e) { e.printStackTrace(); }
    }
    
    public String[] call() throws InterruptedException
    {
      String Text;
      
      try
      {
        Text=readUrl(A_Url,Cookie);
        Out(Text);
      }
      catch (Exception e)
      {
        Out(&quot; --&gt; Error in data : IP = &quot;+IP);
  //    e.printStackTrace();
      }
      return Read_From_Url_Result;
    }
    
    public static String readUrl(String url,String incapsulaCookie)
    {
      StringBuilder response=new StringBuilder();
      String USER_AGENT=&quot;Mozilla/5.0&quot;,inputLine;
      BufferedReader in=null;
  
      try
      {
        HttpURLConnection connection=(HttpURLConnection)new URL(url).openConnection();
        connection.setRequestMethod(&quot;GET&quot;);
        connection.setRequestProperty(&quot;Accept&quot;,&quot;text/html; charset=UTF-8&quot;);
        connection.setRequestProperty(&quot;User-Agent&quot;,USER_AGENT);
        connection.setDoInput(true);
        connection.setDoOutput(true);
        connection.setRequestProperty(&quot;Cookie&quot;,incapsulaCookie);                           // Set the Incapsula cookie
        Out(connection.getRequestProperty(&quot;Cookie&quot;));
  
        in=new BufferedReader(new InputStreamReader(connection.getInputStream()));
        while ((inputLine=in.readLine())!=null) { response.append(inputLine+&quot;\n&quot;); }
        in.close();
      }
      catch (Exception e) { e.printStackTrace(); }
      finally
      {
        try { if (in!=null) in.close(); }
        catch (Exception ex) { ex.printStackTrace(); }
      }
      return response.toString();
    }
    
    public static String getIncapsulaCookie(String url)
    {
      String USER_AGENT=&quot;Mozilla/5.0&quot;,incapsulaCookie=null,visid=null,incap=null;          // Cookies for Incapsula, preserve order
      BufferedReader in=null;
  
      try
      {
        HttpURLConnection cookieConnection=(HttpURLConnection)new URL(url).openConnection();
        cookieConnection.setRequestMethod(&quot;GET&quot;);
        cookieConnection.setRequestProperty(&quot;Accept&quot;,&quot;text/html; charset=UTF-8&quot;);
        cookieConnection.setRequestProperty(&quot;User-Agent&quot;,USER_AGENT);
        cookieConnection.connect();
        
        for (Entry&lt;String,List&lt;String&gt;&gt; header : cookieConnection.getHeaderFields().entrySet())
        {
          if (header.getKey()!=null &amp;&amp; header.getKey().equals(&quot;Set-Cookie&quot;))               // Incapsula gives you the required cookies
          {
            for (String cookieValue : header.getValue())                                   // Search for the desired cookies
            {
              if (cookieValue.contains(&quot;visid&quot;)) visid=cookieValue.substring(0,cookieValue.indexOf(&quot;;&quot;)+1);
              if (cookieValue.contains(&quot;incap_ses&quot;)) incap=cookieValue.substring(0,cookieValue.indexOf(&quot;;&quot;));
            }
          }
        }
        incapsulaCookie=visid+&quot; &quot;+incap;
        cookieConnection.disconnect();
      }
      catch (Exception e) { e.printStackTrace(); }
      finally
      {
        try { if (in!=null) in.close(); }
        catch (Exception ex) { ex.printStackTrace(); }
      }
      return incapsulaCookie;
    }
    
    private static void out(String message) { System.out.print(message); }
    private static void Out(String message) { System.out.println(message); }
    
    public static void main(String[] args)
    {
      final Read_From_Url_Runner demo=new Read_From_Url_Runner(0);
    }
  }
  </code></pre>
  <p>But this only got the first portion of the response as shown below :</p>
  <p><a href="https://i.stack.imgur.com/KxTT3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KxTT3.png" alt="enter image description here" /></a></p>
  <p>What I really wanted to get is something like the following :</p>
  <p><a href="https://i.stack.imgur.com/gZAnX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gZAnX.png" alt="enter image description here" /></a></p>
  <p>This result was got by running my program at : <a href="https://stackoverflow.com/questions/40733223/how-to-shut-down-javafx">How to shut down Javafx?</a></p>