title: Sending POST data with Java
body: <p>Ok, so the place I work at posts the weekly work schedule online and basically, I would like to write a program (which I will eventually be turning into an Android app, so I am writing the program in Java) to send data to the website (my username and password), then once it has logged in, to grab the schedule from the website. Once I have grabbed the schedule, I will parse it for the events (which I plan to automatically add to my phones calendar as events).</p>
  
  <p>Anyway, I am having a bit of trouble doing this. So, basically, I made a little Java function to send POST data to the website, it looks like this:</p>
  
  <pre><code>public void test1 (){
      try {
  
          // First, set the URL to connect to
          String url = "https://mywalmart.com/cleartrust/ct_logon_en.html";
  
          // Next set the character encoding
          String charset = "UTF-8";
  
          // Format the query string
          String query = (new String()).format ("auth_mode=%s&amp;user=%s&amp;password=%s&amp;x=%s&amp;y=%s", 
                  URLEncoder.encode("basic", charset), 
                  URLEncoder.encode("...", charset), 
                  URLEncoder.encode("...", charset),
                  URLEncoder.encode("111", charset),
                  URLEncoder.encode("36", charset));
  
          // Open a connection to the website, set a 10 second timeout, and set it to POST
          URLConnection connection = new URL(url).openConnection();
          connection.setReadTimeout(10000);
          connection.setDoOutput(true);
  
          // Mimic Mozilla web browser
          connection.setRequestProperty("Host", "mywalmart.com");
          connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0");
          connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
          connection.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
          connection.setRequestProperty("Accept-Encoding", "gzip, deflate");
          connection.setRequestProperty("Connection", "keep-alive");
          connection.setRequestProperty("Referer", "https://.../cleartrust/ct_logon_en.html");
  
          // Send the POST data to the host
          OutputStream output = null;
          try {
              output = connection.getOutputStream();
              output.write(query.getBytes(charset));
          } finally {
              if (output != null) try { output.close(); } catch (IOException logOrIgnore) {}
          }
  
          // Get the headers sent to us, and display them all.
          Map&lt;String, List&lt;String&gt;&gt; headers = connection.getHeaderFields ();
          for (Map.Entry&lt;String, List&lt;String&gt;&gt; entry : headers.entrySet ())
          {
              String key = entry.getKey ();
              for (String value : entry.getValue ())
                  System.out.println (key + ": " + value);
          }
  
          // Get the input stream for the HTML portion
          InputStream response = connection.getInputStream();
          Scanner in = new Scanner (response);
  
          // Display all of the HTML
          while (in.hasNextLine()) {
              System.out.println (in.nextLine ());
          }
      } catch (IOException ex) {
          Logger.getLogger(Controller.class.getName()).log(Level.SEVERE, null, ex);
      }
  }
  </code></pre>
  
  <p>The form on the website that I am trying to mimic a connection to is as such (excluding some useless &lt;div&gt; tags and such):</p>
  
  <pre><code>&lt;form name="ctlogonform" action="ct_logon_en.html" method="post" accept-charset="UTF-8"&gt;
      &lt;input type="hidden" name="auth_mode" value="basic" /&gt;
      &lt;input type="text" name="user" /&gt;
      &lt;input type="password" name="password" /&gt;
      &lt;input type="image" src="images/btnLogin.jpg" /&gt;
  &lt;/form&gt;
  </code></pre>
  
  <p>Now, all that it returns when I run the program is this:</p>
  
  <pre><code>null: HTTP/1.1 200 OK
  Content-Length: 8069
  Content-Type: text/html
  </code></pre>
  
  <p>followed by the standard HTML code of the webpage.</p>
  
  <p>So I then created a test PHP webpage, with the following code:</p>
  
  <pre><code>&lt;html&gt;
    &lt;head&gt;
      &lt;title&gt;POST Test&lt;/title&gt;
    &lt;/head&gt;
    &lt;body&gt;
      All header data:&lt;br&gt;
  &lt;?php
  foreach (getallheaders() as $name =&gt; $value) {
      echo "$name: $value&lt;br&gt;\n";
  }
  ?&gt;&lt;br&gt;
      All variables set via POST are here:&lt;br&gt;
  &lt;?php 
  foreach($_POST as $vblname =&gt; $value) echo $vblname . ' = ' . $value . "&lt;br&gt;\n"; 
  ?&gt;
    &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>And, if I run the exact same script as above, except with the created PHP page, I get the following:</p>
  
  <pre><code>null: HTTP/1.1 200 OK
  Date: Mon, 05 Dec 2011 02:36:48 GMT
  Content-Length: 1268
  Connection: close
  Content-Type: text/html
  Server: Apache
  X-Powered-By: PHP/5.2.17
  &lt;html&gt;
    &lt;head&gt;
      &lt;title&gt;POST Test&lt;/title&gt;
    &lt;/head&gt;
    &lt;body&gt;
      All header data:&lt;br&gt;
  User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0&lt;br&gt;
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&lt;br&gt;
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7&lt;br&gt;
  Accept-Encoding: gzip, deflate&lt;br&gt;
  Referer: https://mywalmart.com/cleartrust/ct_logon_en.html&lt;br&gt;
  Host: bf-test.horizon-host.com&lt;br&gt;
  Connection: keep-alive&lt;br&gt;
  Content-type: application/x-www-form-urlencoded&lt;br&gt;
  Content-Length: 60&lt;br&gt;
  &lt;br&gt;
      All variables set via POST are here:&lt;br&gt;
  auth_mode = basic&lt;br&gt;
  user = ...&lt;br&gt;
  password = ...&lt;br&gt;
  x = 111&lt;br&gt;
  y = 36&lt;br&gt;
    &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>So, that tells me that I am successfully sending the POST data, and I am setting my headers correctly, the only problem is that the website my job uses isn't picking it up, or I am not sending it to the correct website. I have tried both <a href="https://mywalmart.com/cleartrust/ct_logon_en.html" rel="nofollow">https://mywalmart.com/cleartrust/ct_logon_en.html</a> and <a href="https://mywalmart.com/ct_logon_en.html" rel="nofollow">https://mywalmart.com/ct_logon_en.html</a>, yet they both do the same thing, and it doesn't send me a redirect either.</p>
  
  <p>So, now that that's all covered! My question is, how could I successfully send the POST data to the website to mimic a web browser, so that I could access the page that would normally be accessed after a successful login?</p>
  
  <p>(also, I have put '...' in places of various places, such as the URL of my work website, my username/password, etc.)</p>
  
  <p>(Update: I had masked 'mywalmart.com' with '...', however, I realized that with a simple Google search of 'ct_logon_en.html' you would be able to find the original URL, so I have no real reason to attempt to hide it.)</p>