Take the 2-minute tour ×

Can someone please give me a good tutorial on screen scraping. I have a webpage that my university uses to upload all the data for each class. To get on to the home page for their site there is an entry screen which has a log in button. When pressed it brings up a floating dialog asking for user name and password. Then it goes straight to the homepage. I do not know where it is requesting for authorisation and i would like to be able to get data from the site programmatically. The data i require is through many more screens with logins but if i can get passed this first screen with my id and password I will be happy enough. preferably i would like this in java but any language will do

share|improve this question
By programatically you mean not using a browser at all? If this is the case, why do you need the "screen-scraping"? Would it do just in java? –  aaronps Nov 13 '12 at 11:35
Find in the page a form that is used to submit the login request, get the url, and request that url programmatically and get the cookie, with which you can request subsequent pages with a logged in state. –  neevek Nov 13 '12 at 11:38
Same question as aaronps, why you need scraping ? Do you need to automate the user and verify the data you are getting as result of user actions ? –  amjad Nov 13 '12 at 11:38
Yes if possible i would like to just use java, any method really. I dont need it but i would like to programmatically look up any updates to my class downloads(projects and lectures) and be able to download them straight away as opposed to trawling the webpage and entering my id and pass over and over. Also to answer Neevek it does not seem to have a form request, maybe im just being dumb. The link to it is: qub.ac.uk/qol –  bobjamin Nov 13 '12 at 12:29
add comment

1 Answer

up vote 1 down vote accepted

This sounds like the login dialog is not part of the original page, but constructed on the fly by some JavaScript, possibly through Ajax calls.

What you will need is some sort of headless browser, that supports javaScript and Ajax.

Have a look at HtmlUnit (http://htmlunit.sourceforge.net/), from the introduction:



Edit: here is an example:

I noticed that the page you want to scan (http://qub.ac.uk/qol/) uses basic authentication, so it is not some kind of HTML input form that pops up, but a browser dialog. When you press the 'Login' button on the start page, a page https://qub.ac.uk/qol/ will be loaded, which is secured in that way.

For a test, I only show you how to get the heading from the unsecured http://qub.ac.uk/qol/ page using HtmlUnit, because I have no access to the secret parts, of course.

I think, it should be clear how it works in general. Consult the excellent documentation and other resources on the web for more details on how to use the HtmlUnit API.

package test;

import java.io.IOException;
import java.net.MalformedURLException;

import javax.xml.bind.DatatypeConverter;

import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.DomElement;
import com.gargoylesoftware.htmlunit.html.DomNodeList;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Scraper {

    public static void main(String[] args)
            throws FailingHttpStatusCodeException, MalformedURLException,
            IOException {
        WebClient webClient = new WebClient();

        String username = "user";
        String password = "pw";
        String authString = username + ":" + password;
        String authEncoded = DatatypeConverter.printBase64Binary(authString

        webClient.addRequestHeader("Authorization", "Basic " + authEncoded);

        HtmlPage page = webClient.getPage("http://qub.ac.uk/qol/");
        // System.out.println(page.asXml());
        DomNodeList<DomElement> headings = page.getElementsByTagName("h3");
        for (DomElement e : headings) {
            System.out.println("Got heading: " + e.getTextContent());


share|improve this answer
Can you explain how I can go about logging in using this. I have added the external jars and can use the htmlunit but i'm not sure what to do to achieve what i want. the website is qub.ac.uk/qol –  bobjamin Nov 13 '12 at 16:51
I have added an example and some tips to my answer. I hope you will accept this answer. For more questions about HtmlUnit, see the documentation and ask a new question here, if you can't find an answer elsewhere. –  Udo Klimaschewski Nov 13 '12 at 18:55
add comment

Your Answer


