Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: NLP - natural language regex-collections?

by perlcapt (Pilgrim)
on Oct 17, 2004 at 17:34 UTC ( #399938=note: print w/ replies, xml ) Need Help??

in reply to NLP - natural language regex-collections?

I couldn't find the stuff I had done before, probably was pretty sad Perl since I wrote it back '96. Here, however is some stuff to start with that I hacked just now. (I'll continue with this, but probably in my own direction.) The thesaurus file is merely a list of keywords followed by synonyms. The newWords should probably be appended to the thesaurus file with an appropriate # comment for each word.

A thought, it might be possible to get some first take synonym solutions by parsing the returned text from a dictionary site.

#!/usr/bin/perl -w use strict; use warnings; my $thesaurusPath = ""; my $thesaurusFile = "thesaurus.nav"; my $ignore = '//'; # key to ignore in the thesaurus file my $thesaurus = {}; my $newWords = {}; # read in the thesaurus word list open CMDS, "<${thesaurusPath}${thesaurusFile}" or die "cannot open \"$thesaurusFile\" for reading"; my $line; while($line = <CMDS>){ chomp($line); # remove comments and leading and trailing space $line =~ s/\s*\#.*$//; $line =~ s/^\s+//; $line =~ s/\s+$//; next if not length($line); # uppercase only $line =~ tr/a-z/A-Z/; # break the list apart and stash it my @words = split(/\s/,$line); # key word to which the others resolve is $words[0], the first one for(@words) { $thesaurus->{$_} = $words[0]; } } close CMDS; # now lets see what results with rewriting print "> "; while(<>) { my @result = (); my @words = (); chomp; tr/a-z/A-Z/; # uppercase only last if /^\s*$/; # end if no input @words = split; for(@words){ if(not defined $thesaurus->{$_}){ # increment or create entry for new word if(defined $newWords->{$_}) { ++ $newWords->{$_}; } else { $newWords->{$_} = 1; } push(@result,"?$_?"); # flag it in output }else{ next if $thesaurus->{$_} eq $ignore; push(@result,$thesaurus->{$_}); } } print join(" ",@result),"\n"; print "> "; } print "These words were not recognized:\n"; for (keys %$newWords) { print "$_\t\t$newWords->{$_}\n"; } exit;

Comment on Re: NLP - natural language regex-collections?
Download Code

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://399938]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2014-12-27 03:20 GMT
Find Nodes?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?

    Results (176 votes), past polls