 <p>You can inspect the page to find out all the fields need to be posted. There is <a href="http://discover-devtools.codeschool.com/" rel="nofollow">a nice tutorial</a> for <code>Chrome DevTools</code>. Other tools like <code>FireBug</code> on FireFox or <code>DragonFly</code> on Opera also do the work while I recommend <code>DevTools</code>.</p> <p>After you post a query. In the <code>Network</code> panel, you can see the form data which actually been sent. In this case:</p> <pre><code>__EVENTTARGET: __EVENTARGUMENT: __LASTFOCUS: __VIEWSTATE:5UILUho/L3O0HOt9WrIfldHD4Ym6KBWkQYI1GgarbgHeAdzM9zyNbcH0PdP6xtKurlJKneju0/aAJxqKYjiIzo/7h7UhLrfsGul1Wq4T0+BroiT+Y4QVML66jsyaUNaM6KNOAK2CSzaphvSojEe1BV9JVGPYWIhvx0ddgfi7FXKIwdh682cgo4GHmilS7TWcbKxMoQvm9FgKY0NFp7HsggGvG/acqfGUJuw0KaYeWZy0pWKEy+Dntb4Y0TGwLqoJxFNQyOqvKVxnV1MJ0OZ4Nuxo5JHmkeknh4dpjJEwui01zK1WDuBHHsyOmE98t2YMQXXTcE7pnbbZaer2LSFNzCtrjzBmZT8xzCkKHYXI31BxPBEhALcSrbJ/QXeqA7Xrqn9UyCuTcN0Czy0ZRPd2wabNR3DgE+cCYF4KMGUjMUIP+No2nqCvsIAKmg8w6Il8OAEGJMAKA01MTMONKK4BH/OAzLMgH75AdGat2pvp1zHVG6wyA4SqumIH//TqJWFh5+MwNyZxN2zZQ5dBfs3b0hVhq0cL3tvumTfb4lr/xpL3rOvaRiatU+sQqgLUn0/RzeKNefjS3pCwUo8CTbTKaSW1IpWPgP/qmCsuIovXz82EkczLiwhEZsBp3SVdQMqtAVcYJzrcHs0x4jcTAWYZUejvtMXxolAnGLdl/0NJeMgz4WB9tTMeETMJAjKHp2YNhHtFS9/C1o+Hxyex32QxIRKHSBlJ37aisZLxYmxs69squmUlcsHheyI5YMfm0SnS0FwES5JqWGm2f5Bh+1G9fFWmGf2QeA6cX/hdiRTZ7VnuFGrdrJVdbteWwaYQuPdekms2YVapwuoNzkS/A+un14rix4bBULMdzij25BkXpDhm3atovNHzETdvz5FsXjKnPlno0gH7la/tkM8iOdQwqbeh7sG+/wKPqPmUk0Cl0kCHNvMCZhrcgQgpIOOgvI2Fp+PoB7mPdb80T2sTJLlV7Oe2ZqMWsYxphsHMXVlXXeju3kWfpY+Ed/D8VGWniE/eoBhhqyOC2+gaWA2tcOyiDPDCoovazwKGWz5B+FN1OTep5VgoHDqoAm2wk1C3o0zJ9a9IuYoATWI1yd2ffQvx6uvZQXcMvTIbhbVJL+ki4yNRLfVjVnPrpUMjafsnjIw2KLYnR0rio8DWIJhpSm13iDj/KSfAjfk4TMSA6HjhhEBXIDN/ShQAHyrKeFVsXhtH5TXSecY6dxU+Xwk7iNn2dhTILa6S/Gmm06bB4nx5Zw8XhYIEI/eucPOAN3HagCp7KaSdzZvrnjbshmP8hJPhnFhlXdJ+OSYDWuThFUypthTxb5NXH3yQk1+50SN872TtQsKwzhJvSIJExMbpucnVmd+V2c680TD4gIcqWVHLIP3+arrePtg0YQiVTa1TNzNXemDyZzTUBecPynkRnIs0dFLSrz8c6HbIGCrLleWyoB7xicUg39pW7KTsIqWh7P0yOiHgGeHqrN95cRAYcQTOhA== __SCROLLPOSITIONX:0 __SCROLLPOSITIONY:106 __VIEWSTATEENCRYPTED: __EVENTVALIDATION:g2V3UVCVCwSFKN2X8P+O2SsBNGyKX00cyeXvPVmP5dZSjIwZephKx8278dZoeJsa1CkMIloC0D51U0i4Ai0xD6TrYCpKluZSRSphPZQtAq17ivJrqP1QDoxPfOhFvrMiMQZZKOea7Gi/pLDHx42wy20UdyzLHJOAmV02MZ2fzami616O0NpOY8GQz1S5IhEKizo+NZPb87FgC5XSZdXCiqqoChoflvt1nfhtXFGmbOQgIP8ud9lQ94w3w2qwKJ3bqN5nRXVf5S53G7Lt+Du78nefwJfKK92BSgtJSCMJ/m39ykr7EuMDjauo2KHIp2N5IVzGPdSsiOZH86EBzmYbEw== ctl00$MainContent$hdnApplyMasterPageWitoutSidebar:0 ctl00$MainContent$hdn1:0 ctl00$MainContent$CorpSearch:rdoByEntityName ctl00$MainContent$txtEntityName:GO ctl00$MainContent$ddBeginsWithEntityName:M ctl00$MainContent$ddBeginsWithIndividual:B ctl00$MainContent$txtFirstName: ctl00$MainContent$txtMiddleName: ctl00$MainContent$txtLastName: ctl00$MainContent$txtIdentificationNumber: ctl00$MainContent$txtFilingNumber: ctl00$MainContent$ddRecordsPerPage:25 ctl00$MainContent$btnSearch:Search Corporations ctl00$MainContent$hdnW:1920 ctl00$MainContent$hdnH:1053 ctl00$MainContent$SearchControl$hdnRecordsPerPage: </code></pre> <p>What I post is <code>Begin with 'GO'</code>. This site is build with <code>WebForms</code>, so there are these long <code>__VIEWSTATE</code> and <code>__EVENTVALIDATION</code> fields. We need send them as well.</p> <p>Now we are ready to make the query. First we need to get a blank form. The following code are written in Python 3.3, through I think they should still work on 2.x.</p> <pre><code>import requests from lxml import etree URL = 'http://corp.sec.state.ma.us/CorpWeb/CorpSearch/CorpSearch.aspx' def get_fields(): res = requests.get(URL) if res.ok: page = etree.HTML(res.text) fields = page.xpath('//form[@id="Form1"]//input') return { e.attrib['name']: e.attrib.get('value', '') for e in fields } </code></pre> <p>With <code>get_fields()</code>, we fetched all <code>&lt;input&gt;</code>s from the form. Note there are also <code>&lt;select&gt;</code>s, I will just hardcode them.</p> <pre><code>def query(data): formdata = get_fields() formdata.update({ 'ctl00$MainContent$ddRecordsPerPage':'25', }) # Hardcode some &lt;select&gt; value formdata.update(data) res = requests.post(URL, formdata) if res.ok: page = etree.HTML(res.text) return page.xpath('//table[@id="MainContent_SearchControl_grdSearchResultsEntity"]//tr') </code></pre> <p>Now we have a generic <code>query</code> function, lets make a wrapper for specific ones.</p> <pre><code>def search_by_entity_name(entity_name, entity_search_type='B'): return query({ 'ctl00$MainContent$CorpSearch':'rdoByEntityName', 'ctl00$MainContent$txtEntityName': entity_name, 'ctl00$MainContent$ddBeginsWithEntityName': entity_search_type, }) </code></pre> <p>This specific example site use a group of <code>&lt;radio&gt;</code> to determine which fields to be used, so <code>'ctl00$MainContent$CorpSearch':'rdoByEntityName'</code> here is necessary. And you can make others like <code>search_by_individual_name</code> etc. by yourself.</p> <p>Sometimes, website need more information to verify the query. By then you could add some <a href="http://docs.python-requests.org/en/latest/user/quickstart/#custom-headers" rel="nofollow">custom headers</a> like <code>Origin</code>, <code>Referer</code>, <code>User-Agent</code> to mimic a browser.</p> <p>And if the website is using JavaScript to generate forms, you need more than <code>requests</code>. <a href="http://phantomjs.org" rel="nofollow"><code>PhantomJS</code></a> is a good tool to make browser scripts. If you want do this in Python, you can use <a href="http://www.riverbankcomputing.co.uk/software/pyqt/" rel="nofollow"><code>PyQt</code></a> with <code>qtwebkit</code>.</p> <p><strong>Update</strong>: It seems the website blocked our Python script to access it after yesterday. So we have to feign as a browser. As I mentioned above, we can add a custom header. Let's first add a <code>User-Agent</code> field to header see what happend.</p> <pre><code>res = requests.get(URL, headers={ 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36', }) </code></pre> <p>And now... <code>res.ok</code> returns <code>True</code>!</p> <p>So we just need to add this header in both call <code>res = requests.get(URL)</code> in <code>get_fields()</code> and <code>res = requests.post(URL, formdata)</code> in <code>query()</code>. Just in case, add <code>'Referer':URL</code> to the headers of the latter:</p> <pre><code>res = requests.post(URL, formdata, headers={ 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36', 'Referer':URL, }) </code></pre>