Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

Comment on

My actual set of phrases will conform to a corpus of roughly 15,000 existing items, so there are no typos, misspellings or synonyms involved.

Then, I would approach the problem this way.

  1. Store the corpus of phrases in its own table each with a unique numeric value.
  2. Each set of phrases then becomes a bitfield with 1-bit set in the appropriate position for each phrase that set contains.
  3. Your similarity can then be some hueristic based on that population counts (bit count) of ANDing and XORing the two bitstrings that represent each set.

    The population count of the result of ANDing two set's bitstrings will tell you how many phrases they have in common;

    The population count of the result of XORing two set's bitstrings will tell you how many phrases that appear in one but not the other.

    You can then combine those two numbers mathematically to reflect whether the sharing of phrases is more important than having phrases not in common -- or vice versa -- and come up with a single number for each pairing that you can then apply a threshold value to.

You'd need a DB that supports bitstrings -- postgresql and mysql seem to -- and AND/XOR & popcount of bitstrings. I couldn't (from a quick look) see a popcount function, but (at least in the case of PgSQL), it should be a simple thing to add a PL/Perl function to do this using Perl's

$popcount = unpack '%b*', $bitstring;

Food for thought perhaps.

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

In reply to Re^3: Comparing sets of phrases stored in a database? by BrowserUk

and:  <code> code here </code>

  • Please read these before you post! —
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    As of 2014-07-10 15:35 GMT
    Find Nodes?
      Voting Booth?

      When choosing user names for websites, I prefer to use:

      Results (213 votes), past polls