Take the 2-minute tour ×

I am using the Java-based Nutch web-search software. In order to prevent duplicate (url) results from being returned in my search query results, I am trying to remove (a.k.a. normalize) the expressions of 'jsessionid' from the urls being indexed when running the Nutch crawler to index my intranet. However my modifications to $NUTCH_HOME/conf/regex-normalize.xml (prior to running my crawl) do not seem to be having any effect.

  1. How can I ensure that my regex-normalize.xml configuration is being engaged for my crawl? and,

  2. What regular expression will successfully remove/normalize expressions of 'jsessionid' from the url during the crawl/indexing?

The following is the contents of my current regex-normalize.xml:

<?xml version="1.0"?>

Here is the command that I am issuing to run my (test) 'crawl':

bin/nutch crawl urls -dir /tmp/test/crawl_test -depth 3 -topN 500
share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

What version of Nutch are you using? I'm not familiar with Nutch but the default download of Nutch 1.0 already contains a rule in regex-normalize.xml which seems to handle this problem.

<!-- removes session ids from urls (such as jsessionid and PHPSESSID) -->

Btw. regex-urlfilter.txt seems to contain something of relevance too

# skip URLs containing certain characters as probable queries, etc.

Then there are some settings in nutch-default.xml which you might want to check out


If that all doesn't help maybe this does: How can I force fetcher to use custom nutch-config?

share|improve this answer
I am using Nutch version 0.8.1. This version has the following setting in nutch-default.xml: urlnormalizer.class ...instead of urlnormalizer.order I changed the value from org.apache.nutch.net.BasicUrlNormalizer to org.apache.nutch.net.RegexUrlNormalizer. This is what causes the regex-normalize.xml file to actually be engaged when crawling. Also, I added the following plugin to the 'plugin-includes' value: urlnormalizer-(pass|regex|basic) This is not included by default in version 0.8.1. Thanks soo much for pointing me in the right direction! –  Anand Krishnan Nov 20 '09 at 20:42
No problem. Now just consider up-voting and accepting my answer –  jitter Nov 20 '09 at 22:02
add comment

Your Answer


