



For several reasons, though, researchers are wary about using the web in more formal research. One, as Mark Liberman, a Language Log contributor, warns colleagues, is that “there are some mean texts out there”. The web is filled with words intended to attract internet searches to gambling and pornography sites, and these can muck up linguists' results. Originally, such sites would contain these words as lists, so the makers of Google, the biggest search engine, fitted their product with a list filter that would exclude hits without a correct syntactical context. In response, as Dr Liberman notes, many offending websites have hired computational linguists to churn out syntactically correct but meaningless verbiage including common search terms. “When some sandbank over a superslots hibernates, a directness toward a progressive jackpot earns frequent flier miles” is a typical example. Such pages are not filtered by Google, and thus create noise in research data.

There are other problems as well. Search engines, unlike the tools linguists use to analyse standard corpora, do not allow searching for a particular linguistic structure, such as “[Noun phrase] far from [verb phrase]”. This requires indirect searching via samples like “He far from succeeded”. But Philip Resnik, of the University of Maryland, has created a “Linguist's Search Engine” (LSE) to overcome this. When trying to answer, for example, whether a certain kind of verb is generally used with a direct object, the LSE grabs a chunk of web pages (say a thousand, with perhaps a million words) that each include an example of the verb. The LSE then parses the sample, allowing the linguist to find examples of a given structure, such as the verb without an object. In short, the LSE allows a user to create and analyse a custom-made corpus within minutes.

The web still has its drawbacks. Most of it is in English, limiting its use for other languages (although Dr Resnik is working on a Chinese version of the LSE). And it is mostly written, not spoken, making it tougher to gauge people's spontaneous use. But since much web content is written by non-professional writers, it more clearly represents informal and spoken English than a corpus such as the North American News Text Corpus does.

Despite the problems, linguists are gradually warming to the web as a corpus for formal research. An early paper on the subject, written in 2003 by Frank Keller and Mirella Lapata, of Edinburgh and Sheffield Universities, showed that web searches for rare two-word phrases correlated well with the frequency found in traditional corpora, as well as with human judgments of whether those phrases were natural. What problems the web throws up are seemingly outweighed by the advantages of its huge size. Such evidence, along with tools such as Dr Resnik's, should convince more and more linguists to turn to the corpus on their desktop. Young scholars seem particularly keen.
