Take the 2-minute tour ×

I have a list of million urls. I need to extract the TLD for each url and create multiple files for each TLD. For example collect all urls with .com as tld and dump that in 1 file, another file for .edu tld and so on. Further within each file, I have to sort it alphabetically by domains and then by subdomains etc.

Can anyone give me a head start for implementing this in perl?

share|improve this question

1 Answer 1

up vote 6 down vote accepted
  1. Use URI to parse the URL,
  2. Use its host method to get the host,
  3. Use Domain::PublicSuffix's get_root_domain to parse the host name.
  4. Use the tld or suffix method to get the real TLD or the pseudo TLD.

use feature qw( say );

use Domain::PublicSuffix qw( );
use URI                  qw( );

my $dps = Domain::PublicSuffix->new();

for (qw(
)) {
   my $url = $_;

   # Treat relative URLs as absolute URLs with missing http://.
   $url = "http://$url" if $url !~ /^\w+:/;

   my $host = URI->new($url)->host();
   $host =~ s/\.\z//;  # D::PS doesn't handle "domain.com.".

      or die $dps->error();

   say $dps->tld();     # com  uk
   say $dps->suffix();  # com  co.uk
share|improve this answer
It may be a noobish question but I cant stop to ask: Is there any significance when you use qw() in second and third lines? Use URI qw(); qw () as I understand is used to convert it into comma separated strings of its arguments. –  Maxwell Nov 7 '11 at 2:32
@Sahil Shah, I always explicitly list imports. qw( ... ) is convenient for that. I don't switch from qw() to () when I want nothing to be imported. use Domain::PublicSuffix qw(); use URI qw(); means the same as use Domain::PublicSuffix (); use URI ();, and in this particular case, it's probably the same as use Domain::PublicSuffix; use URI;, but it's not a chance I take. –  ikegami Nov 7 '11 at 3:30
Thanks. If the input is without http(like just google.com) , your code is giving me error:Can't locate object method "host" via package "URI::_generic" at line :(my $host = URI->new($url)->host(); ) –  Maxwell Nov 7 '11 at 3:44
@Sahil Shah, The URL "google.com" is a relative URL with no host. Of course it will fail. If you have no host, there is no TLD. If you want to pretend that relative URLs are badly formed URLs that are missing "http://", then go ahead. $url = "http://" . $url if $url !~ /^\w+:/; –  ikegami Nov 7 '11 at 4:05
thanks a lot!. I have a question. I am confused between the terms TLD, domain and hostnames. Is it that a tld is always the last rightmost part of a url? Like if url is google.com I understand com is the TLD. But if the url is like www.sub1.sub2.co.cc ? Is co.cc my TLD now or is only .cc my TLD? In your above code, the dps->suffix is giving domain name right? (co.uk) Can you please clarify this ? –  Maxwell Nov 7 '11 at 13:49

Your Answer


