<p>Mapreduce will work just fine, and probably you could do most of the input-output shuffling by pig.</p>

<p>See</p>

<p><a href="http://arxiv.org/abs/1207.4371" rel="nofollow">http://arxiv.org/abs/1207.4371</a></p>

<p>for some algorithms.</p>

<p>Of course, to make sure you get a running start, you don't actually need to be using mapreduce for this task; just split the input yourself, make the simplest fast program to calculate ngrams of a single input file and aggregate the ngram frequencies later.</p>
Producing ngram frequencies for a large dataset
<p>I'd like to generate ngram frequencies for a large dataset. Wikipedia, or more specifically, Freebase's WEX is suitable for my purposes.</p>

<p>What's the best and most cost efficient way to do it in the next day or so?</p>

<p>My thoughts are:</p>

<ul>
<li>PostgreSQL using regex to split sentences and words. I already have the WEX dump in PostgreSQL, and I already have regex to do the splitting (major accuracy isn't required here)</li>
<li>MapReduce with Hadoop</li>
<li>MapReduce with Amazon's Elastic MapReduce, which I know next to nothing about</li>
</ul>

<p>My experience with Hadoop consists of calculating Pi on three EC2 instances very very inefficiently. I'm good with Java, and I understand the concept of Map + Reduce.
PostgreSQL I fear will take a long, long time, as it's not easily parallelisable.</p>

<p>Any other ways to do it? What's my best bet for getting it done in the next couple days?</p>