title: Curl/Fetch question - Referer, Post data, Cookies
body: <p>Got a Curl question (and yeah, i've searched Google, this site, as well as the docs!!)</p>
  
  <p>Somewhat new to curl. Trying to use Curl from the cmdline (linux) to fetch pages from a college website (sjsu). I can somewhat get the 1st page, but the 2nd/2rd pages use referers/cookies.. I think this is where things are screwed up..</p>
  
  <p>I'm also dealing with fetching data from within frames on the given page. If I analyze the page, in combination with the livehttpheader data, this shouldn't be an issue.</p>
  
  <p>I've tried using cookie-jar, as well as cookie, in various combinations with the "-e" for the referer, and the "-d" for the posting data...</p>
  
  <p>I'm posting the data from the livehttpheader process for the pages, as well as the test shell script i'm using.. Any thoughts/pointers would be greatly appreciated...</p>
  
  <p>Thanks</p>
  
  <p>-bruce</p>
  
  <p>** this is the data that is displayed, when i select the "submit" btn on the location/term page to get to the "class select" page" as i understand the process, the Curl function should more or less replicate this process, with the referer, the postdata, and the target http/url</p>
  
  <h1>    livehttpheaders data:</h1>
  
  <pre><code>https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
  
  
  
  POST /psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL HTTP/1.1
  
  Host: cmshr.sjsu.edu
  
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118     Fedora/3.0.11-1.fc9 Firefox/3.0.11
  
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  
  Accept-Language: en-us,en;q=0.5
  
  Accept-Encoding: gzip,deflate
  
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  
  Keep-Alive: 300
  
  Connection: keep-alive
  
  Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&amp;PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&amp;PortalRegistryName=EMPLOYEE&amp;PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&amp;PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&amp;PortalHostNode=HRMS&amp;NoCrumbs=yes
  
  Cookie: I4Web_uuid=ece1faf2b59ace0ba40a4849c80e1d1b@e18f35cc4*0339838ad3db6ba9a32396e947e5d0d6@1418f55cc4*8bc80f3fe499835a27c78d5593811a3a@3818fc5cc4*166fa2936446e617426d945230e89e49@1e19045cc4*88210f324b634276bce6758d896eca38@8195a5cc4*d1da10f4207bca1bd889fea53873c471@3c1baf5cc4*d56e90391bc1ab51115f5c448ac2a124@71be65cc4*87ccd281d6940c53a49366c366796911@3e1bee5cc4*bb23f7cd74e303c96a11b4fdec5b7646@812a35cc6*84f727d9a8f7a11871756dad0c00d545@1a3b445cb4*; cssln164HSJ1-84915=B1h9LG2L1yydzpj3TvqQrHBHKdbWpnl8!695030755; ExpirePage=https://cmshr.sjsu.edu/psp/HSJPRDF/; PS_LOGINLIST=https://cmshr.sjsu.edu/HSJPRDF; PS_TOKENEXPIRE=9_Jan_2010_18:06:54_GMT; SignOnDefault=CMSPUBLIC; cssln169HSJ1-84915=JpgTLL9RJVQPWjphl3Vybx1xnvW8zbMl!-1963651055; cssln162HSJ1-84915=1LrQLH7b20GmB0kx50cf54GPGjj97WWM!813942549; cssln118HSJ1-84915=xbDyLLFZZCTbpLFhJQpCNwNF2ppgjH4y!-923260117; PS_TOKEN=AAAApgECAwQAAQAAAAACvAAAAAAAAAAsAARTaGRyAgBOcQgAOAAuADEAMBQLu6wR/++AI1n7RObpon/kagLJUAAAAGYABVNkYXRhWnicHclNDkBADIbhd4ZYWbkHYYKMrSF+gghxEPdzOJ3plz5tWuBVOopRSOnPm+HYuTl56NlYcAkjB1PKLPdVPhdDjaGkks7D9FqxoqEQjdiEvQtpJZYfnXoL+Q==
  
  Content-Type: application/x-www-form-urlencoded
  
  Content-Length: 339
  
  ICType=Panel&amp;ICElementNum=0&amp;ICStateNum=2&amp;ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&amp;ICXPos=0&amp;ICYPos=0&amp;ICFocus=&amp;ICSaveWarningFilter=0&amp;ICChanged=-1&amp;ICResubmit=0&amp;ICSID=HpLTZLhQFp4p&amp;CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&amp;CLASS_SRCH_STRM1=2102&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06
  
  HTTP/1.x 200 OK
  
  Cache-Control: no-cache
  
  Date: Sat, 09 Jan 2010 18:07:47 GMT
  
  Content-Length: 21137
  
  Content-Type: text/html; CHARSET=UTF-8
  
  IgnorePortalRegisteredURL: 1
  
  PortalRegisteredURL: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
  
  UsesPortalRelativeURL: true
  
  X-Powered-By: Servlet/2.4 JSP/2.0
  
  Set-Cookie: PS_TOKENEXPIRE=9_Jan_2010_18:07:47_GMT; domain=.sjsu.edu; path=/; secure
  </code></pre>
  
  <hr>
  
  <pre><code>test Curl shell script:
  =============================================
  #!/bin/sh -v
  #
  # test shell for curl..
  #
  #curl --cookie lcookie.lwp --cookie-jar lcookie.lwp --output "ctest.dat"  -L "http://my.sjsu.edu/"
  
  #foo="http://my.sjsu.edu/"
  
  #curl --cookie lcookie.lwp --cookie-jar lcookie.lwp --output "ctest.dat"  -L "$foo"
  
  #exi
  curl -v --cookie-jar lcookie.lwp --output "ctest2.dat"  https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH
  #exit
  curl -v  -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" --cookie-jar lcookie.lwp --output "ctest3.dat"  -e "https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HRMS/s/WEBLIB_PT_NAV.ISCRIPT1.FieldFormula.IScript_UniHeader_Frame?c=uA%2buCaKuiBh5DTZEFHMBvNKbD7XLjINl&amp;FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&amp;PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&amp;PortalRegistryName=EMPLOYEE&amp;PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&amp;PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&amp;PortalHostNode=HRMS&amp;PortalIsPagelet=true&amp;NoCrumbs=yes"
  
  #get the page with the term/location --- uses the get
  curl -v  -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie-jar lcookie.lwp --output "ctest4.dat"  -e "https://cmshr.sjsu.edu/psp/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&amp;PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&amp;PortalRegistryName=EMPLOYEE&amp;PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&amp;PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&amp;PortalHostNode=HRMS&amp;NoCrumbs=yes"
  
  #
  # the following two lines are attempts to get the page with the class display... neither one works. 
  # -instead, the output files are simply the same as the above page, with the location/term menu..
  #
  # the page should be a pae that lists a class schedule select menu..
  #
  
  #get the page with the search class menu... it's a post
  curl -v  -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie-jar lcookie.lwp --output "ctest5.dat"  -e "Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL?FolderPath=PORTAL_ROOT_OBJECT.PA_HC_CLASS_SEARCH&amp;PortalActualURL=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2fEMPLOYEE%2fHSJPRD%2fc%2fCOMMUNITY_ACCESS.CLASS_SEARCH.GBL&amp;PortalRegistryName=EMPLOYEE&amp;PortalServletURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsp%2fHSJPRDF%2f&amp;PortalURI=https%3a%2f%2fcmshr.sjsu.edu%2fpsc%2fHSJPRDF%2f&amp;PortalHostNode=HRMS&amp;NoCrumbs=yes" -d "ICType=Panel&amp;ICElementNum=0&amp;ICStateNum=2&amp;ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&amp;ICXPos=0&amp;ICYPos=0&amp;ICFocus=&amp;ICSaveWarningFilter=0&amp;ICChanged=-1&amp;ICResubmit=0&amp;ICSID=HpLTZLhQFp4p&amp;CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&amp;CLASS_SRCH_STRM1=2102&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL"
  
  #get the page with the search class menu... it's a post
  curl -v -A "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.11) Gecko/2009061118 Fedora/3.0.11-1.fc9 Firefox/3.0.11" --cookie lcookie.lwp --output "ctest6.dat"  -e "Referer: https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL" -d "ICType=Panel&amp;ICElementNum=0&amp;ICStateNum=2&amp;ICAction=CLASS_SRCH_WRK2_SSR_PB_SRCH%2457%24&amp;ICXPos=0&amp;ICYPos=0&amp;ICFocus=&amp;ICSaveWarningFilter=0&amp;ICChanged=-1&amp;ICResubmit=0&amp;ICSID=HpLTZLhQFp4p&amp;CLASS_SRCH_WRK2_INSTITUTION%2445%24=SJ000&amp;CLASS_SRCH_STRM1=2102&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24=06&amp;CLASS_SRCH_WRK2_SSR_CLS_SRCH_TYPE%2458%24%24rad=06" -L "https://cmshr.sjsu.edu/psc/HSJPRDF/EMPLOYEE/HSJPRD/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL"
  </code></pre>