Take the 2-minute tour ×

I'm trying to work on a project about page ranking.

I want to make an index (dictionary) which looks like this:
file1.html -> [[cat, ate, food, drank, milk], [file2.html, file3.html]]
file2.html -> [[dog, barked, ran, away], [file1.html, file4.html]]

Fetching links is easy - look for anchor tags.

My question is - how do I fetch text? The text in the html files is not enclosed within any tags like <p>

Thanks in advance for all the help

share|improve this question

2 Answers 2

up vote 0 down vote accepted

If the text isn't enclosed in tags is it really HTML?
As Amber says, you'll have an easier job of this using some HTML parser like BeautifulSoup.

The example below demonstrates a simple method for returning text within tags.
This method works for any tag AFAIK.

>>> from BeautifulSoup import BeautifulSoup as bs
>>> html = '''
... <div><a href="/link1">link1 contents</a></div>
... <div><a href="/link2">link2 contents</a></div>
... '''
>>> soup = bs(html)
>>> for anchor_tag in soup.findAll('a'):
...   print anchor_tag.contents[0]
link1 contents
link2 contents

Apart from that I can imagine that you'd want a dictionary with a count of how many times a certain term appeared in some HTML document. defaultdict is good for that kind of thing:

>>> from collections import defaultdict
>>> d = defaultdict(int)
...   d[anchor_tag.contents[0]] += 1
>>> d
defaultdict(<type 'int'>, {u'link1 contents': 1, u'link2 contents': 1})

Hopefully that gives you some ideas to run with. Come back and open another question if you run into other issues.

share|improve this answer

Use an HTML parser - something like BeautifulSoup.

share|improve this answer
Yes, I'm using beautifulsoup, unfortunately, I'm unable to parse text thats not enclosed within any tags –  csguy11 Oct 16 '10 at 21:21

Your Answer


