<jupyter_start><jupyter_text>How do ratings behave after users have seen many captions? This notebook looks at the "vote decay" of users. The New Yorker caption contest organizer, <NAME>, has received many emails like the one below (name/personal details left out for anonymity)> Here's my issue. >> First time I encounter something, I might say it's funny. >> Then it comes back in many forms over and over and it's no longer funny and I wish I could go back to the first one and say it's not funny. >> But it's funny, and then I can't decide whether to credit everyone with funny or keep hitting unfunny. What I really like to find out is who submitted it first, but often it's slightly different and there may be a best version. Auggh!>> How should we do this???We can investigate this: we have all the data at hand. We record the timestamp, participant ID and their rating for a given caption. So let's see how votes go after a user has seen $n$ captions!<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
import caption_contest_data as ccd<jupyter_output><empty_output><jupyter_text>Reading in dataLet's read in the data. As the last column can contain a non-escaped comma, we have to fix that before doing any analysis.Note that two versions of this notebook exist (the previous notebook can be found in [43bc5d]). This highlights some of the differences required to read in the earlier datasets.[43bc5d]:https://github.com/nextml/caption-contest-data/commit/43bc5d23ee287b8b34cc4eb0181484bd21bbd341<jupyter_code>contest = 540

responses = ccd.responses(contest)

print(len(responses))
responses.head()<jupyter_output>452547<jupyter_text>Seeing how many captions a user has seenThis is the workhorse of the notebook: it sees how many captions one participant has seen. I sorted by timestamp (and with an actual timestamp, not a str) to collect the ratings in the order a user has seen. I do not assume that only one user answers at a time.<jupyter_code>last_id = None
i = 0
num_responses = []
captions_seen = []
responses = responses.sort_values(by='timestamp_query_generated')
# responses = responses[0:1000]  # debug
captions_seen_by = {}
captions_seen = []
for _, response in responses.iterrows():
    id_, rating = response['participant_uid'], response['target_reward']
    if id_ not in captions_seen_by:
        captions_seen_by[id_] = 0
    captions_seen_by[id_] += 1
    captions_seen += [captions_seen_by[id_]]
    num_responses += [i]
responses['number of captions seen'] = captions_seen
responses.head()<jupyter_output><empty_output><jupyter_text>Viewing the dataNow let's format the data to view it. We can view the data in two ways: as we only have three rating values, we can view the probability of a person rating 1, 2 or 3, and can also view the mean.In this, we rely on `pd.pivot_table`. This can take DataFrame that looks like a list of dictionaries and compute `aggfunc` (by default `np.mean`) for all items that contain common keys (indicated by `index` and `columns`). It's similar to Excel's pivot table functionality.  Probability of rating {1, 2, 3}<jupyter_code>def prob(x):
    n = len(x)
    ret = {'n': n}
    ret.update({name: np.sum(x == i) for name, i in [('unfunny', 1),
                                                  ('somewhat funny', 2),
                                                  ('funny', 3)]})
    return ret

probs = responses.pivot_table(index='number of captions seen',
                              columns='alg_label', values='target_reward',
                              aggfunc=prob)
probs.head()
d = {label: dict(probs[label]) for label in ['RandomSampling']}
for label in d.keys():
    for n in d[label].keys():
        if d[label][n] is None:
            continue
        for rating in ['unfunny', 'somewhat funny', 'funny']:
            d[label][n][rating] = d[label][n][rating] / d[label][n]['n']
df = pd.DataFrame(d['RandomSampling']).T
df = pd.concat({'RandomSampling': df}, axis=1)
df.head()
plt.style.use("default")

fig, axs = plt.subplots(figsize=(8, 4), ncols=2)

alg = "RandomSampling"
show = df[alg].copy()
show["captions seen"] = show.index

for y in ["funny", "somewhat funny", "unfunny"]:
    show.plot(x="captions seen", y=y, ax=axs[0])
show.plot(x="captions seen", y="n", ax=axs[1])

for ax in axs:
    ax.set_xlim(0, 100)
    ax.grid(linestyle='--', alpha=0.5)

plt.style.use("default")

def plot(alg):
    fig = plt.figure(figsize=(10, 5))
    ax = plt.subplot(1, 2, 1)
    df[alg][['unfunny', 'somewhat funny', 'funny']].plot(ax=ax)
    plt.xlim(0, 100)
    plt.title('{} ratings\nfor contest {}'.format(alg, contest))
    plt.ylabel('Probability of rating')
    plt.xlabel('Number of captions seen')
    plt.grid(linestyle="--", alpha=0.6)

    ax = plt.subplot(1, 2, 2)
    df[alg]['n'].plot(ax=ax, logy=False)
    plt.ylabel('Number of users')
    plt.xlabel('Number of captions seen, $n$')
    plt.title('Number of users that have\nseen $n$ captions')
    plt.xlim(0, 100)
    plt.grid(linestyle="--", alpha=0.6)
    
for alg in ['RandomSampling']:
    fig = plot(alg)
    plt.show()<jupyter_output><empty_output>