 <p>I wrote a simple method below to check if there is a trailing hashtag/pound sign/number sign that returns a boolean.</p> <pre><code>import java.util.regex.Matcher; import java.util.regex.Pattern; public boolean hasHashTag(String url) { int index = url.lastIndexOf("#"); if(index == -1) { return false; } else { Pattern p = Pattern.compile("[^a-z0-9 ]", Pattern.CASE_INSENSITIVE); Matcher m = p.matcher(url.substring(index+1)); System.out.println(url.substring(index+1) + " "+ (index + 1)); return !m.find(); } } </code></pre> <p>You can now use this method to filter out the duplicates.</p> <pre><code>if(hasHashTag(URLHERE)) { //don't add to urls to search } else { //add url to search } </code></pre>