Take the 2-minute tour ×

I want to get book information such as author name / pages / publish year / etc ... from amazon using HtmlAgilityPack but seems amazon webpages have some problems and I can't access the appropriate fields.

here is what I've done :

I use Firefox and Firebug + FirePath to retrieve desired XPath and then inside my code I summon HtmlAgilityPack and instruct it to get information using acquired XPath that I've got it from Firebug but no luck and till now I couldn't access the "Product Details" part of the amazon.com

and this is my XPath (which is working only with HtmlAgilityPack)

HtmlAgilityPack.HtmlNodeCollection cnt = doc.DocumentNode.SelectNodes("//*[@class='content']");
int i=1;
foreach (HtmlAgilityPack.HtmlNode content in cnt)
    if (i != 3)
    if (i == 3) // i==3 means I've reached the product details but I can't go any further :(

        s = content.SelectSingleNode("").OuterHtml;

      //  break;


How can I access Product Details using appropriate understandable XPath for HtmlAgilityPack?

And why does the syntax of Firebug + FirePath XPath is different from HtmlAgilityPack?

share|improve this question

2 Answers 2

up vote 2 down vote accepted

As @Mystere said, I suggest using the API. But if you are doing this for test purpose, or just because you want to use web scraping to obtain the info (I'm not sure if Amazon allows it or not. You should check it before doing this), here is the thing:

Why are you doing this?

s = content.SelectSingleNode("").OuterHtml;

The following is what you are looking for in case you want to get the HTML source of that part of the page.

s = content.OuterHtml;

When you are scraping, I suggest you trying to identify the part you need to scrape, and see the particularities of that block of content.

If you use:

var node = doc.DocumentNode.SelectNodes("//td[@class='bucket']/div[@class='content']");

that will give you the Product Details block you are looking for. If you want to get some fields like Paperback, Publisher, ... you can do:

string paperback = node.SelectSingleNode("./ul/li[1]/text()").InnerText;
string publisher = node.SelectSingleNode("./ul/li[2]/text()").InnerText;
string language = node.SelectSingleNode("./ul/li[3]/text()").InnerText;

If you want to be sure that the XPath you are using will be correct for HtmlAgilityPack, open the page on Internet Explorer 8 (or 9) and use the Developer Tools (F12) to get the XPath. The thing is that each browser renders the HTML in a particular way. For example, you will always see <tbody> tags in Firefox right after a <table>, so maybe HtmlAgilityPack doesn't, and that simple detail of adding /tbody/ to your XPath can make your program fail.

share|improve this answer
thanks for your solution but seems IE's developer tools doesn't have such tool for getting xpath (or at least I couldn't find it ) –  austin powers Apr 26 '11 at 9:36
@austin You are right. The thing is that I never use the entire XPath, because it is prone to errors when the layout changes even a little. What I do is use Firebug, and if it does not work with HtmlAgilityPack, then I check on Internet Explorer to see what happens. Also, remember, in the HtmlAgilityPack site, you have the HAP Explorer. That will let you paste a HTML code and kind of inspect it. You can also take a look at Anyway, you should check stackoverflow.com/questions/1990225/… –  Oscar Mederos Apr 27 '11 at 0:53

Why don't you just use amazon's web service api that is designed to do this?

share|improve this answer
I know amazon has web service but I want to use web scraping –  austin powers Apr 26 '11 at 7:33
@austin powers - I hope you don't plan on this being useful. Amazon can change their pages at any time, breaking your scraping. Hell, there is no guarantee that any two pages are structured the same way. There's no guarantee that any two requests to the same page will be structured the same way. This is a recipe for failure. –  Erik Funkenbusch Apr 26 '11 at 7:38
yeah I know and specially amazon's html is the hell of spaghetti code I just want to retrieve 50 book info in this way for personal use. –  austin powers Apr 26 '11 at 9:37
Sometimes there is no other way: you can do only parsing of html. I have one of solutions in production: with minor corrections solution works almost 5 years. –  VikciaR Apr 27 '11 at 17:14

Your Answer


