title: Difference of sets of multiple values for single column in pandas
body: <p>I've got some grouped tabular data, and in this data there's a column for which each data point can actually have a set of different values. I'm trying to calculate the difference of that set from that of its preceding data point in the group it's a member of. For example, given the data below, I'm trying to calculate the difference of the values of <code>Tokens</code><sup>&dagger;</sup> for <code>Timestep</code> value <em>n</em> from the values of <code>Tokens</code> for the row with <code>Timestamp</code> value <em>n</em> - 1 for each <code>Dyad,Participant</code> combination:</p>
  
  <pre><code>| Dyad | Participant | Timestep | Tokens            |
  |------|-------------|----------|-------------------|
  | 1    | A           | 1        | apple,banana      |
  | 1    | B           | 1        | apple,orange      |
  | 1    | A           | 2        | banana            |
  | 1    | B           | 2        | orange,kumquat    |
  | 1    | A           | 3        | orange            |
  | 1    | B           | 3        | orange,pear       |
  | 2    | A           | 1        | orange,pear       |
  | 2    | B           | 1        | apple,banana,pear |
  | 2    | A           | 2        | banana,persimmon  |
  | 2    | B           | 2        | apple             |
  | 2    | A           | 3        | banana            |
  | 2    | B           | 3        | apple             |
  </code></pre>
  
  <p>How can I best accomplish this using pandas?</p>
  
  <h2>Expected result</h2>
  
  <p>I ultimately want to create a new column with the output of a function <code>token_overlap(data)</code> which computes the ratio of <code>Token</code> values that overlap with the values of the datapoint preceding it:</p>
  
  <pre><code>| Dyad | Participant | Timestep | Tokens            | TokenOverlap |
  |------|-------------|----------|-------------------| -------------|
  | 1    | A           | 1        | apple,banana      | (no value)   |
  | 1    | B           | 1        | apple,orange      | (no value)   |
  | 1    | A           | 2        | banana            | 0.5          |
  | 1    | B           | 2        | orange,kumquat    | 0.333        |
  | 1    | A           | 3        | orange            | 0            |
  | 1    | B           | 3        | orange,pear       | 0.333        |            
  | 2    | A           | 1        | orange,pear       | (no value)   |
  | 2    | B           | 1        | apple,banana,pear | (no value)   |
  | 2    | A           | 2        | banana,persimmon  | 0            |
  | 2    | B           | 2        | apple             | 0.333        |
  | 2    | A           | 3        | banana            | 0.5          |
  | 2    | B           | 3        | apple             | 1            |
  </code></pre>
  
  <h2>Current approach</h2>
  
  <p>I convert the multi-values into a <code>frozenset</code> by using the <code>converters</code> keyword of <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html" rel="nofollow noreferrer"><code>pandas.read_csv(...)</code></a>:</p>
  
  <pre><code>def parse_set(cell_value: str) -&gt; FrozenSet[str]:
      return frozenset(cell_value.split(','))
  
  round_tokens = pandas.read_csv(inpath, converters={"Tokens": parse_set})
  </code></pre>
  
  <p>I then create groups of <code>Dyad,Participant</code> datapoints using <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html" rel="nofollow noreferrer"><code>pandas.DataFrame.groupby(..)</code></a>:</p>
  
  <pre><code>round_tokens.sort_values(["Dyad", "Timestep"])
  dyad_participants = round_tokens.groupby(["Dyad", "Participant"])
  </code></pre>
  
  <p>However, I am unsure how to get each row and it's precedessor's <code>Tokens</code> value (which should be a <code>frozenset</code>): I have some function which tries to do so but I'm unsure if the function itself is wrong or if I'm extracting the row data incorrectly.</p>
  
  <pre><code>def token_overlap(data):
      own_relevant_tokens = data["Tokens"]
      prev_tokens = data.shift(-1)["Tokens"]
      overlap = own_relevant_tokens.intersection(prev_tokens)
      union = own_relevant_tokens.union(prev_tokens)
      return len(overlap) / len(union)
  
  round_tokens["TokenOverlap"] = dyad_participants.apply(token_overlap)
  </code></pre>
  
  <p>However, this doesn't actually work: The actual error is</p>
  
  <blockquote>
    <p>AttributeError: 'Series' object has no attribute 'union'</p>
  </blockquote>
  
  <p>but I know I'm not using/understanding/grokking the pandas API correctly, hence the extremely long-winded question. <strong>How can I group my data and then, inside each group, calculate a metric using set-like values from one row and the same column's values for the row preceding it?</strong></p>
  
  <hr>
  
  <p><sup>&dagger;</sup>In the real data, there are over 1,000 possible values for <code>Tokens</code>, so, at least to me, this task would be even harder if I enumerated the presence of each token as Boolean values, e.g. <code>Token_Apple</code>, <code>Token_Banana</code>, etc.</p>