 <p>Yes, this is absolutely possible with Scrapy. If you're just opening a list of URLs that you know rather than scraping the site it I'd say Scrapy is overkill.</p> <p>I would recommend <a href="http://lxml.de/" rel="nofollow">lxml</a> for HTML parsing, it's simple and considerably faster than BeautifulSoup (can be as much as two orders of magnitude). And <a href="http://docs.python-requests.org/en/latest/" rel="nofollow">requests</a> for HTTP because it's super simple.</p> <p>In the snippet below I'm using an XPath query to find the correct definition description element. <code>//dl[dt/text()='term']//dd/text()</code> is essentially saying "find the definition list (dl) element that has a definition term with text content of 'term' (<code>//dl[dt/text()='term']</code>) and then find all definition description (dd) elements and get their text content (<code>//dd/text()</code>)".</p> <pre><code>from StringIO import StringIO import requests from lxml import etree response = requests.get("http://www.tripadvisor.in/members/SomersetKeithers") parser = etree.HTMLParser() tree = etree.parse(StringIO(response.text), parser) def get_definition_description(tree, term): description = tree.xpath("//dl[dt/text()='%s']//dd/text()" % term) if len(description): return description[0].strip() print get_definition_description(tree, "Age:") print get_definition_description(tree, "Gender:") print get_definition_description(tree, "Location:") </code></pre>