title: Read a JSP servlet page using PHP (libcurl), Python (liburl), or AJAX
body: <p>So, I'm attempting to curl a page: <code>http://rutgers.bncollege.com/webapp/wcs/stores/servlet/TextBookProcessDropdownsCmd?campusId=35577418</code> in order to extract some data. The issue is that I keep getting either a 404 error or 302 status on the header. I suspect it has something to do with Barnes and Noble's Tomcat not properly redirecting to the servlet when requested remotely. That's just speculation though. I have tried multiple implementations using both libcurl in PHP5, liburl in Python, AJAX (framework and non-framework), and using the curl binary from my terminal.</p>
  
  <p>Here's an example of the output I receive when I echo the response text:</p>
  
  <blockquote>
    <p>An error has occurred:</p>
    
    <p>Error Code: 404</p>
    
    <p>Message Target: /BNCB_GenericError.jsp</p>
    
    <p>Servlet Name: JSP 1.2 Processor</p>
    
    <p>Stack Trace: [Ljava.lang.StackTraceElement;@14b6c4d</p>
    
    <p>Root Cause: N/A </p>
  </blockquote>
  
  <p>Here's are the headers I'm sending and receiving:</p>
  
  <p><strong>Response Headers</strong></p>
  
  <blockquote>
    <p>Expires   Thu, 01 Dec 1994 16:00:00 GMT</p>
    
    <p>Cache-Control no-cache="set-cookie,set-cookie2"</p>
    
    <p>Location  <a href="http://uncc.bncollege.com/webapp/wcs/stores/servlet/TBDropDownView?campusId=1748054&amp;dojo.transport=xmlhttp&amp;dojo.preventCache=1300287790307&amp;ddkey=TextBookProcessDropdownsCmd" rel="nofollow">http://uncc.bncollege.com/webapp/wcs/stores/servlet/TBDropDownView?campusId=1748054&amp;dojo.transport=xmlhttp&amp;dojo.preventCache=1300287790307&amp;ddkey=TextBookProcessDropdownsCmd</a></p>
    
    <p>Content-Length    0</p>
    
    <p>PerfHeader    duration=D=56606,</p>
    
    <p>time=t=1300287776952692</p>
    
    <p>Content-Type  text/html;</p>
    
    <p>charset=ISO-8859-1</p>
    
    <p>Content-Language  en-US Date  Wed, 16</p>
    
    <p>Mar 2011 15:02:57 GMT</p>
    
    <p>Connection    keep-alive</p>
    
    <p>Vary  Accept-Encoding</p>
    
    <p>Set-Cookie    WC_SESSION_ESTABLISHED=true;Domain=.bncollege.com;Path=/</p>
    
    <p>WC_ACTIVESTOREDATA=%2d1%2c0;Domain=.bncollege.com;Path=/WC_USERSESSION_46349649=46349649%2cnull%2cnull%2c%2d2000%2cnull%2cnull%2cnull%2cnull%2cnull%2cnull%2cnull%2cnull%2c%5b0%7cnull%7cnull%7cnull%7c%2d2000%5d%2c8XwO3l7WhszbuSO41vmZUDtbpoQ%3d;Domain=.bncollege.com;Path=/</p>
    
    <p>JSESSIONID=0000AuZi2Uo6F6Ft5xihFdUsBQn:app06z02;Domain=.bncollege.com;Path=/</p>
    
    <p>TS884e96=b7fb55c6fcd8aff3987bcdb831a8255a16b4cbcb208252614d80d120;</p>
  </blockquote>
  
  <p><strong>Request Headers</strong></p>
  
  <blockquote>
    <p>Host  uncc.bncollege.com</p>
    
    <p>User-Agent    Mozilla/5.0 (Macintosh; U;</p>
    
    <p>Intel Mac OS X 10.6; en-US;</p>
    
    <p>rv:1.9.2.15pre) Gecko/20110227</p>
    
    <p>Firefox/3.6.15pre (Mac Community</p>
    
    <p>Build, ElFurbe)</p>
    
    <p>Accept    text/html,application/xhtml+xml,application/xml;q=0.9,<em>/</em>;q=0.8</p>
    
    <p>Accept-Language   en-us,en;q=0.5</p>
    
    <p>Accept-Encoding   gzip,deflate</p>
    
    <p>Accept-Charset    ISO-8859-1,utf-8;q=0.7,*;q=0.7</p>
    
    <p>Keep-Alive    115 Connection  keep-alive</p>
    
    <p>Referer   <a href="http://localhost/bn.php" rel="nofollow">http://localhost/bn.php</a></p>
    
    <p>Origin    <a href="http://localhost" rel="nofollow">http://localhost</a></p>
  </blockquote>
  
  <p>And here's the code for that:</p>
  
  <pre><code>function bufferURL($url,$bindArgs) {
      $ch = curl_init();
      curl_setopt($ch, CURLOPT_URL, 'http://rutgers.bncollege.com/webapp/wcs/stores/servlet/TBWizardView?catalogId=10001&amp;storeId=58552&amp;langId=-1');
      curl_setopt($ch, CURLOPT_POST, 1);
      curl_setopt($ch, CURLOPT_HEADER, 0);
      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
      curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
      curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
      curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
      curl_exec($ch);
  
      $url .= '?';
      foreach ($bindArgs as $a =&gt; $b) $url .= $a . '=' . $b . '&amp;';
      $url = substr($url,0,strlen($url)-1);
  
      curl_setopt($ch, CURLOPT_HTTPGET, true);
      curl_setopt($ch, CURLOPT_URL, $url);
      echo curl_exec($ch);
  }
  </code></pre>
  
  <p>BN appears to be using Dojo to perform their AJAX queries to the servlet; however, even when using the same request format, I am unable to replicate.</p>