Take the 2-minute tour ×

I am currently using watir to do a web scraping of a website hiding all data from the usual HTML source. If I am not wrong, they are using XML and those AJAX technology to hide it. Firefox can see it but it is displayed via "DOM Source of selection".

Everything works fine but now I am looking for an equivalent tool as watir but everything need to be done without a browser. Everything need to be done in txt file.

In fact right now, watir is using my browser to emulate the page and return me the whole html code I am looking. I would like to the same but without the browser.

Is it possible ?

Thanks Regards Tak

share|improve this question

2 Answers 2

Your best guess would be to use something like webscarab and capture the URLS of the AJAX requests your browser is doing.
That way, you can just grab the "important" data yourself by simulating those calls with any HTTP library

share|improve this answer

It is possible with a little Python coding.

I wrote a simple script to fetch locations of cargo offices.

First steps

  1. Open the ajax page with Google Chrome for example, in Turkish but you can understand it. http://www.yurticikargo.com/bilgi-servisleri/Sayfalar/en-yakin-sube.aspx
  2. Press F12 to show bottom developer tools and navigate to Network tab.
  3. Navigate XHR tab on the bottom.
  4. Make an AJAX request by selecting an item in the first combobox. And go to Headers Tab
  5. You will GetTownByCity on left pane, click it and inspect it.

    Request URL: (...)/_layouts/ArikanliHolding.YurticiKargo.WebSite/ajaxproxy- sswservices.aspx/GetTownByCity

    Request Method:POST

    Status Code:200 OK

  6. In the Request Payload tree item you will see

    Request Payload :{cityId:34} header.

  7. This will guide us to implement a python code.

Lets do it.

#!/usr/bin/env python
# -*- coding: utf-8 -*-    
import requests
import json
# import simplejson as json
baseUrl = 'http://www.yurticikargo.com/'
ajaxRoot = '_layouts/ArikanliHolding.YurticiKargo.WebSite/'
getTown = 'ajaxproxy-sswservices.aspx/GetTownByCity'
urlGetTown = baseUrl + ajaxRoot + getTown
headers = {'content-type': 'application/json','encoding':'utf-8'}  # We are sending JSON headers, equivalent to Python dictionary
for plaka in range(1,82): # Because Turkiye has number plates from 1 to 81
    payload = {'cityId':plaka}
    data = r.json() # Returning data is in JSON format, if you need HTML use r.content()
    # ... Process the fetched data with JSON parser,
    # If HTML format, Beautiful Soup, Lxml, or etc...

Note that this code is a part of my working code and it is written on the fly, the most important is I did not test it. It may require small modifications to run it.

share|improve this answer

Your Answer


