3.1. APPLICATIONS OF SET SIMILARITY
75
















T
S
Figure 3.1: Two sets with Jaccard similarity 3/8
Section 1.3.1. However, textual similarity also has important uses. Many of
these involve ﬁnding duplicates or near duplicates. First, let us observe that
testing whether two documents are exact duplicates is easy; just compare the
two documents character-by-character, and if they ever diﬀer then they are not
the same. However, in many applications, the documents are not identical, yet
they share large portions of their text. Here are some examples:
Plagiarism
Finding plagiarized documents tests our ability to ﬁnd textual similarity. The
plagiarizer may extract only some parts of a document for his own. He may
alter a few words and may alter the order in which sentences of the original
appear. Yet the resulting document may still contain much of the original. No
simple process of comparing documents character by character will detect a
sophisticated plagiarism.
Mirror Pages
It is common for important or popular Web sites to be duplicated at a number
of hosts, in order to share the load. The pages of these mirror sites will be
quite similar, but are rarely identical. For instance, they might each contain
information associated with their particular host, and they might each have
links to the other mirror sites but not to themselves. A related phenomenon is
the reuse of Web pages from one academic class to another. These pages might
include class notes, assignments, and lecture slides. Similar pages might change
the name of the course, year, and make small changes from year to year. It
is important to be able to detect similar pages of these kinds, because search
engines produce better results if they avoid showing two pages that are nearly
identical within the ﬁrst page of results.
