# Exploring data for the attention index

The idea of the attention index is to provide a score that indicates the impact of an article, and can easily be aggregated by subject, publisher or other axis.

The index comprises of two parts:

- **promotion** how important the article was to the publisher, based on the extent to which they chose to editorially promote it
- **response** how readers reacted to the article, based on social engagements

The index will be a number between 0 and 100. 50% is driven by the promotion, and 50% by response:

### Promotion Score

The promotion score should take into account:

- whether the publisher chose make the article a lead article on their primary front (30%)
- how long the publisher chose to retain the article on their front (40%)
- whether they chose to push the article on their facebook brand page (30%)

It should be scaled based on the value of that promotion, so a popular, well-visited site should score higher than one on the fringes. And similarly a powerful, well-followed brand page should score higher than one less followed.

### Response Score

The response score takes into account the number of engagements on Facebook.

The rest of this notebook explores how those numbers could work, starting with the response score because that is easier, I think.

# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
data = pd.read_csv("articles_2017-06-01_2017-06-30.csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      7dff0d3d3e8ae6a4f43d582b294bd03c0a4746f3
      http://www.economist.com/news/leaders/21722855...
      The British election: The middle has fallen ou...
      2017-06-01 09:30:29.938
      2017-06-01
      25767.00
      21.00
      2017-06-01T10:58:01.747Z
      5827.00
      16202.00
      3738.00
      The Economist
      economist_com
      6023
      10444
      44.00
      True
      8097249.00
      2017-06-04T08:19:05.000Z
      1825

      663d3c01e00503ac647c38e5d3d775cafe7e2ca0
      http://www.economist.com/news/indicators/21722...
      Retail sales, producer prices, wages and excha...
      2017-06-01 10:40:16.109
      2017-06-01
      0.00
      0.00
      2017-06-01T10:54:02.805Z
      0.00
      0.00
      0.00
      The Economist
      economist_com
      0
      459
      45.00
      False
      nan
      NaN
      1825

      8bb899dbd63148a4a173882bbbc940492737d0b7
      http://www.economist.com/news/indicators/21722...
      Foreign reserves
      2017-06-01 10:40:16.691
      2017-06-01
      1.00
      0.02
      2017-06-01T11:56:02.710Z
      0.00
      0.00
      1.00
      The Economist
      economist_com
      0
      349
      45.00
      False
      nan
      NaN
      1825

      e8184c4ea8498103945968fcacd5c3c00f883cba
      https://www.nytimes.com/2017/05/31/nyregion/me...
      February Thaw at the Flea Market
      2017-06-01 00:03:03.133
      2017-06-01
      217.00
      0.90
      2017-06-01T03:08:00.900Z
      22.00
      192.00
      3.00
      New York Times
      nytimes_com
      0
      0
      nan
      False
      nan
      NaN
      120

      0d8ec825a1f491a0c526859f608ba9b15c7a8ba2
      http://www.foxnews.com/politics/2017/05/31/pri...
      Priebus, Conway among White House staffers who...
      2017-06-01 00:00:07.065
      2017-06-01
      2375.00
      20.86
      2017-06-01T00:14:01.139Z
      941.00
      1259.00
      175.00
      Fox News
      foxnews_com
      0
      839
      78.00
      True
      15382550.00
      2017-06-01T00:04:36.000Z
      285

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    138236.00
    mean       1611.24
    std       10560.04
    min           0.00
    50%          33.00
    75%         337.00
    90%        2342.00
    95%        6598.25
    99%       30067.65
    99.5%     46674.58
    99.9%    119804.98
    max     1486144.00
    Name: fb_engagements, dtype: float64

There's one article there with 1 million plus engagements, let's just double check that.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      28acc178e3f07c46fcd16c0324e8b0c5ecdccf35
      http://www.huffingtonpost.com/entry/i-dont-kno...
      I Don't Know How To Explain To You That You Sh...
      2017-06-28 03:05:06.097
      2017-06-27 01:57:54
      1486144.00
      1102.03
      2017-06-28T14:06:02.064Z
      217510.00
      1060499.00
      208135.00
      HuffPost
      huffingtonpost_com
      0
      0
      nan
      True
      9576627.00
      2017-06-28T03:00:27.000Z
      215

Yup, that's a biggie.

```python
data.fb_engagements.mode()
```

    0   0.00
    dtype: float64

Going back to the enagement counts, we see the mean is 1,611, mode is zero, median is 33, 90th percentile is 2,342, 99th percentile is 30,067, 99.5th percentile is 46,674. The standard deviation is 10,560, significantly higher than the mean, so this is not a normal distribution.

We want to provide a sensible way of allocating this to the 50 buckets we have available. Let's just bucket geometrically first:

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

plt.figure(figsize=(12,4.5))
plt.hist(data.fb_engagements, bins=1)
#plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
#plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()

```

Well that's not very useful. Almost everything will score less than 0 if we just do that, which isn't a useful metric.

Let's start by excluding zeros.

```python
non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]

plt.figure(figsize=(12,4.5))
plt.hist(non_zero_fb_enagagements, bins=50)
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()
```

That's still a big number at the bottom, and so not a useful score.

Next, we exclude the outliers: cap at the 99.9th percentile (i.e. 65,258.53), so that 0.1% of articles should receive the maximum score.

```python
non_zero_fb_enagagements_without_outliers = non_zero_fb_enagagements.clip_upper(65258.53)

plt.figure(figsize=(12,4.5))
plt.hist(non_zero_fb_enagagements_without_outliers, bins=50)
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

That's looking a bit more interesting.

After some exploration, to avoid too much emphasis on the lower end of the scale, we move the numbers to the right a bit by adding on the median.

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   114767.00
    mean         5.28
    std          1.81
    min          3.53
    25%          3.76
    50%          4.63
    75%          6.34
    max         11.69
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   114767.00
    mean        11.18
    std         11.11
    min          0.00
    25%          2.00
    50%          7.00
    75%         18.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Now look at how the shares distribute to score:

```python
plt.figure(figsize=(15,8))

shares = np.arange(1, 60000)
plt.plot(shares, scale_engagements(shares))
plt.xlabel("shares")
plt.ylabel("score")
plt.axhline(scale_engagements(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axhline(scale_engagements(median), label=f'Median ({median:,.0f})', color='green')
plt.axhline(scale_engagements(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axhline(scale_engagements(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axhline(scale_engagements(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')

plt.legend(frameon=True, shadow=True)

```

```python
proposed.groupby("response_score").fb_engagements.agg([np.size, np.min, np.max])
```

      size
      amin
      amax

      response_score

      0.00
      30990.00
      0.00
      1.00

      1.00
      17243.00
      2.00
      7.00

      2.00
      8865.00
      8.00
      14.00

      3.00
      6443.00
      15.00
      22.00

      4.00
      5552.00
      23.00
      32.00

      5.00
      4487.00
      33.00
      43.00

      6.00
      4236.00
      44.00
      57.00

      7.00
      3876.00
      58.00
      73.00

      8.00
      3484.00
      74.00
      92.00

      9.00
      3181.00
      93.00
      114.00

      10.00
      3098.00
      115.00
      141.00

      11.00
      2912.00
      142.00
      172.00

      12.00
      2667.00
      173.00
      208.00

      13.00
      2621.00
      209.00
      251.00

      14.00
      2492.00
      252.00
      301.00

      15.00
      2466.00
      302.00
      361.00

      16.00
      2253.00
      362.00
      431.00

      17.00
      2238.00
      432.00
      513.00

      18.00
      2082.00
      514.00
      610.00

      19.00
      1965.00
      611.00
      724.00

      20.00
      1884.00
      725.00
      858.00

      21.00
      1852.00
      859.00
      1017.00

      22.00
      1724.00
      1018.00
      1203.00

      23.00
      1554.00
      1204.00
      1422.00

      24.00
      1568.00
      1423.00
      1681.00

      25.00
      1437.00
      1682.00
      1985.00

      26.00
      1247.00
      1986.00
      2343.00

      27.00
      1184.00
      2344.00
      2765.00

      28.00
      1227.00
      2766.00
      3262.00

      29.00
      1195.00
      3263.00
      3846.00

      30.00
      1039.00
      3847.00
      4535.00

      31.00
      1056.00
      4536.00
      5341.00

      32.00
      955.00
      5346.00
      6299.00

      33.00
      908.00
      6301.00
      7423.00

      34.00
      839.00
      7425.00
      8747.00

      35.00
      674.00
      8752.00
      10305.00

      36.00
      619.00
      10306.00
      12140.00

      37.00
      636.00
      12150.00
      14296.00

      38.00
      570.00
      14302.00
      16834.00

      39.00
      495.00
      16846.00
      19838.00

      40.00
      446.00
      19840.00
      23363.00

      41.00
      392.00
      23366.00
      27511.00

      42.00
      356.00
      27520.00
      32388.00

      43.00
      274.00
      32406.00
      38120.00

      44.00
      213.00
      38171.00
      44867.00

      45.00
      196.00
      44981.00
      52791.00

      46.00
      121.00
      52991.00
      62183.00

      47.00
      107.00
      62600.00
      73282.00

      48.00
      65.00
      73632.00
      86142.00

      49.00
      69.00
      86520.00
      101462.00

      50.00
      183.00
      102024.00
      1486144.00

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   138245.00
    mean         9.43
    std         91.85
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        280.00
    99.5%      609.78
    99.9%     1200.76
    max       6989.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   4133.00
    mean     315.39
    std      430.96
    min        4.00
    25%       72.00
    50%      169.00
    75%      400.00
    90%      784.00
    95%     1061.00
    99%     1644.00
    99.5%   2144.40
    99.9%   5161.23
    max     6989.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      34.00
      5.20
      0.70
      1.24
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      100.00
      12.69
      1.75
      4.16
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      184.00
      8.35
      1.28
      1.45
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      40.00
      4.23
      1.01
      0.77
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      204.00
      11.60
      1.46
      3.81
      12.02
      12.02
      12.02
      12.02

      cnn_com
      206.00
      12.52
      1.48
      2.15
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      162.00
      11.48
      1.54
      0.79
      11.85
      11.85
      11.85
      11.85

      economist_com
      60.00
      7.35
      1.75
      0.67
      7.99
      7.99
      7.99
      7.99

      evolvepolitics_com
      48.00
      4.87
      0.79
      1.71
      5.13
      5.13
      5.13
      5.13

      foxnews_com
      184.00
      10.48
      0.93
      0.88
      10.61
      10.61
      10.61
      10.61

      ft_com
      92.00
      7.06
      2.17
      0.54
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      174.00
      10.95
      1.21
      1.86
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      135.00
      9.26
      2.25
      0.67
      10.07
      10.07
      10.07
      10.07

      indy100_com
      157.00
      5.76
      1.94
      1.06
      4.69
      7.04
      7.04
      7.04

      libdemvoice_org
      175.00
      4.55
      0.50
      1.96
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      65.00
      8.54
      1.65
      1.52
      9.15
      9.15
      9.15
      9.15

      nbcnews_com
      122.00
      8.53
      1.61
      0.60
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      17.00
      6.35
      0.00
      6.35
      6.35
      6.35
      6.35
      6.35

      npr_org
      162.00
      8.48
      2.22
      0.78
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      68.00
      12.02
      1.98
      1.04
      12.53
      12.53
      12.53
      12.53

      order-order_com
      319.00
      4.32
      1.70
      0.39
      2.79
      5.29
      5.78
      5.78

      propublica_org
      23.00
      6.06
      0.65
      3.10
      6.20
      6.20
      6.20
      6.20

      reuters_com
      100.00
      9.46
      1.08
      3.06
      9.66
      9.66
      9.66
      9.66

      rt_com
      157.00
      9.25
      2.20
      0.68
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      173.00
      4.50
      1.12
      0.42
      4.94
      5.03
      5.03
      5.03

      telegraph_co_uk
      89.00
      9.56
      2.02
      0.85
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      281.00
      4.56
      1.91
      0.93
      2.59
      5.59
      6.21
      6.21

      theguardian_com
      117.00
      11.66
      1.93
      0.81
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      81.00
      6.63
      0.95
      1.03
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      83.00
      10.75
      2.36
      0.76
      11.42
      11.42
      11.42
      11.42

      westmonster_com
      83.00
      4.71
      1.24
      0.78
      5.22
      5.22
      5.22
      5.22

      yournewswire_com
      238.00
      4.21
      2.20
      0.40
      1.90
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.38500569152790032, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     134118
    1.00         97
    2.00         76
    3.00         81
    4.00        130
    5.00         80
    6.00         70
    7.00        271
    8.00        336
    9.00        328
    10.00       189
    11.00       183
    12.00        69
    13.00        85
    14.00       347
    15.00       233
    16.00       333
    17.00       352
    18.00       232
    19.00       291
    20.00       344
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      14.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  8.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    libdemvoice_org                     7.00
    mirror_co_uk                       14.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     8.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    1.1666666666666667

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     60540
    2.00      8026
    3.00      5541
    4.00      3980
    5.00      1009
    6.00       690
    7.00       558
    8.00       564
    9.00       698
    10.00      297
    11.00      286
    12.00      296
    13.00       99
    14.00       40
    15.00       17
    32.00        2
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk      81.00
    bbc_co_uk                          15276.00
    breitbart_com                       2485.00
    brexitcentral_com                     51.00
    buzzfeed_com                        9500.00
    cnn_com                            12161.00
    dailymail_co_uk                    13313.00
    economist_com                        390.00
    evolvepolitics_com                    70.00
    foxnews_com                         8882.00
    ft_com                              3148.00
    huffingtonpost_com                  8190.00
    independent_co_uk                   4312.00
    indy100_com                          616.00
    libdemvoice_org                      176.00
    mirror_co_uk                        2241.00
    nbcnews_com                         2071.00
    newstatesman_com                     155.00
    npr_org                             2341.00
    nytimes_com                         9668.00
    order-order_com                      330.00
    propublica_org                        35.00
    reuters_com                         7080.00
    rt_com                              4172.00
    skwawkbox_org                        174.00
    telegraph_co_uk                     6396.00
    thecanary_co                         312.00
    theguardian_com                    12391.00
    thetimes_co_uk                      8652.00
    washingtonpost_com                  9487.00
    westmonster_com                      326.00
    yournewswire_com                     372.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    7dff0d3d3e8ae6a4f43d582b294bd03c0a4746f3    1.00
    663d3c01e00503ac647c38e5d3d775cafe7e2ca0    1.00
    8bb899dbd63148a4a173882bbbc940492737d0b7    1.00
    e8184c4ea8498103945968fcacd5c3c00f883cba    0.00
    0d8ec825a1f491a0c526859f608ba9b15c7a8ba2    2.00
    3aafd4740f1f17e6fc95ff150f69cb59558e59d8    4.00
    6a841b83a0dd644aa6ea78ffbd54f58b1a66a7b6    6.00
    96d72401aedbc06d4bb5cca6362d4ec8ed793355    1.00
    56aa931d7c654815b53f4b043c87c0ac0861e021    0.00
    3e5be0e746d28dfc32e30fe2effee69d5961c37f    0.00
    80d87401d456b2ee62023bbe3743aa5ba364f247    0.00
    b18912c86234a70ba7610776efa576d2aa2e0177    1.00
    e9070ef52ee653532ad3a97bd0b0706f7d1df144    0.00
    82e7c8ddc8464e4d52f86c486b16a8d101dc9a7d    0.00
    f8a7d027d95b478702139fff1ee0e5833b66a8bb    1.00
    11cb1da2c791987aadfacd179add5f2673f3291e    2.00
    94f2f78412c11739c4fca490d24a104edcac33df   11.00
    d354f70d22c17d66af3b7771dd87eb5799925ffc    2.00
    aeb28ab277ead52d186829ae058a00df46648172    1.00
    536ccbab72a7c359e098e3f6feb9b070f2b73339    1.00
    0a9bb1e943a406a233bb7f91a63dedbb9c20869b    1.00
    0cb25e9285a97aae867dacae2e318dd003766b62    1.00
    5595a40184777899bd899521bf823962ea09339c    0.00
    27f0fc09be05d1a249804b7e7e050ac3614c6aa1    1.00
    3315505b6ff44b587f99b96e4301d3524c21948b    0.00
    3c7d6c3c3ac201a7b43d29b60e3ff94135d8f6bc    0.00
    451d3e7b5e18e187395f26f7d51e85f23c61c731    1.00
    81bd114418c8eb223d0107c3ff69dd607d6be209    1.00
    312b27a59e466dbdfad9c05fa845458e2bed858a    0.00
    685a841b5b1a2297972bd0f2c9a7ecbae88c7344    0.00
                                                ...
    798383f7dda9ff9a9cb6b7ad768fc11a0f3615c9    1.00
    a10a0bc69d1151c5a6b01e3e345a31f77d87b14e    0.00
    7a3f9641b6501450c654b4e00b359358f27673bf    1.00
    613c250df7e5e6d01d85132dc7fbbb198c2fac39    2.00
    c57a29f5af8745f2e31641db609494c3e4b48bf1    1.00
    a5f69b61e12bc0b7aa5cf7b22842fddba2e9709e    1.00
    3c4aa73c293f6a9a50bf263a89af953ebad053eb    3.00
    2309b756a280f85a0915cb4c151431b1354669ae    3.00
    163bb5d69fd0de2119aafeff09fddbb9691e61e9    1.00
    82d0f2db16969def0a48180f1a7f6d7be99afbf9    5.00
    61f3d972e29674a24de5b2cb52c6464290598c7b    0.00
    72d9c8408d39f939f6a64196947030b0eb87993f    3.00
    24f282b97a2924f66c39b07f212e43978ead177a    1.00
    0b30e4e88f50225a34f8672188b291a2cc76c838    1.00
    f526104dc7e0d832a3c68631b23fcc28eb9dd418    4.00
    4720496ece45476440d35dd2f7739c7919e60f5b    0.00
    e5dd1c9ada471cce035e09f630ce2e8e011f8f6c    1.00
    1ab635a7aa2659cb95dd6ab937c15c0c62a94ccc    0.00
    869ce9a17451e9dec5f98028cf017a6a2f1863d5    1.00
    46983e4f0dea39028762ca7337a7c9352ee7928f    0.00
    a4f2800eac25aad77eebda695f38cabbeeb9706d    1.00
    d17e10c6bbf77b350ec4abef3f727debbe9a6d2a   12.00
    c558f64e381647d7fdb7b78127bd5dfd93d9b518    1.00
    88c19249aa141882438509d533370f5ce85f64a9    3.00
    6fcbe7cdcf62bc3304d16955dd696cd23a8bf6bc    0.00
    efc85592b8f3ea27098c32456433a35994cac1b4    1.00
    946c9dce19c585c7dcde90a8d67cbfdf1469d3ee    1.00
    191fad27ec178a6d9461208643fe8442f894afd1    0.00
    f45fd57b13f75bf705c1285bffc81d525219b1bb    0.00
    954d40411fb8a312e02982803c8bc561c91deb70    0.00
    Name: front_score, Length: 138245, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    42647872.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      81.00
      0.44
      0.50
      0.00
      0.00
      0.00
      1.00
      1.00

      bbc_co_uk
      12832.00
      0.65
      3.05
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2579.00
      0.81
      0.98
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      51.00
      0.90
      0.30
      0.00
      1.00
      1.00
      1.00
      1.00

      buzzfeed_com
      2187.00
      0.23
      0.42
      0.00
      0.00
      0.00
      0.00
      1.00

      cnn_com
      4063.00
      2.51
      4.33
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      24069.00
      0.53
      1.51
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      718.00
      1.79
      1.47
      0.00
      0.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      72.00
      0.97
      0.17
      0.00
      1.00
      1.00
      1.00
      1.00

      foxnews_com
      7072.00
      0.62
      1.82
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      3349.00
      0.58
      0.91
      0.00
      0.00
      0.00
      2.00
      2.00

      huffingtonpost_com
      4311.00
      1.22
      1.84
      0.00
      0.00
      0.00
      4.00
      4.00

      independent_co_uk
      6046.00
      0.71
      1.28
      0.00
      0.00
      0.00
      0.00
      3.00

      indy100_com
      628.00
      0.14
      0.35
      0.00
      0.00
      0.00
      0.00
      1.00

      libdemvoice_org
      176.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      2815.00
      0.18
      0.39
      0.00
      0.00
      0.00
      0.00
      1.00

      nbcnews_com
      2227.00
      2.09
      2.00
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      161.00
      0.71
      0.45
      0.00
      0.00
      1.00
      1.00
      1.00

      npr_org
      2018.00
      1.41
      1.50
      0.00
      0.00
      0.00
      3.00
      3.00

      nytimes_com
      5298.00
      1.32
      2.20
      0.00
      0.00
      0.00
      5.00
      5.00

      order-order_com
      331.00
      0.78
      0.42
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      35.00
      0.94
      0.24
      0.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      5812.00
      0.32
      0.73
      0.00
      0.00
      0.00
      0.00
      2.00

      rt_com
      2302.00
      1.07
      1.00
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      174.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      7897.00
      0.53
      0.89
      0.00
      0.00
      0.00
      2.00
      2.00

      thecanary_co
      321.00
      0.96
      0.19
      0.00
      1.00
      1.00
      1.00
      1.00

      theguardian_com
      8960.00
      0.55
      1.16
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      8663.00
      0.06
      0.23
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      22289.00
      0.20
      0.75
      0.00
      0.00
      0.00
      0.00
      3.00

      westmonster_com
      336.00
      0.27
      0.45
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      372.00
      0.02
      0.13
      0.00
      0.00
      0.00
      0.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     328420.00
    bbc_co_uk                          42647872.00
    breitbart_com                       3487964.00
    brexitcentral_com                      7866.00
    buzzfeed_com                        2591775.00
    cnn_com                            27763464.00
    dailymail_co_uk                    11651715.00
    economist_com                       8154368.00
    evolvepolitics_com                   107007.00
    foxnews_com                        15503059.00
    ft_com                              3584700.00
    huffingtonpost_com                  9593243.00
    independent_co_uk                   7132250.00
    indy100_com                          209247.00
    libdemvoice_org                        8518.00
    mirror_co_uk                        2740528.00
    nbcnews_com                         9049400.00
    newstatesman_com                     153519.00
    npr_org                             6060772.00
    nytimes_com                        14186079.00
    order-order_com                       43213.00
    propublica_org                       335889.00
    reuters_com                         3777573.00
    rt_com                              4502330.00
    skwawkbox_org                          3650.00
    telegraph_co_uk                     4239098.00
    thecanary_co                         150704.00
    theguardian_com                     7555864.00
    thetimes_co_uk                       656207.00
    washingtonpost_com                  5920796.00
    westmonster_com                       11380.00
    yournewswire_com                      24569.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.70
    bbc_co_uk                          17.57
    breitbart_com                      15.06
    brexitcentral_com                   8.97
    buzzfeed_com                       14.77
    cnn_com                            17.14
    dailymail_co_uk                    16.27
    economist_com                      15.91
    evolvepolitics_com                 11.58
    foxnews_com                        16.56
    ft_com                             15.09
    huffingtonpost_com                 16.08
    independent_co_uk                  15.78
    indy100_com                        12.25
    libdemvoice_org                     9.05
    mirror_co_uk                       14.82
    nbcnews_com                        16.02
    newstatesman_com                   11.94
    npr_org                            15.62
    nytimes_com                        16.47
    order-order_com                    10.67
    propublica_org                     12.72
    reuters_com                        15.14
    rt_com                             15.32
    skwawkbox_org                       8.20
    telegraph_co_uk                    15.26
    thecanary_co                       11.92
    theguardian_com                    15.84
    thetimes_co_uk                     13.39
    washingtonpost_com                 15.59
    westmonster_com                     9.34
    yournewswire_com                   10.11
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.79
    bbc_co_uk                          10.66
    breitbart_com                       8.16
    brexitcentral_com                   2.06
    buzzfeed_com                        7.86
    cnn_com                            10.23
    dailymail_co_uk                     9.36
    economist_com                       9.01
    evolvepolitics_com                  4.67
    foxnews_com                         9.65
    ft_com                              8.18
    huffingtonpost_com                  9.17
    independent_co_uk                   8.87
    indy100_com                         5.34
    libdemvoice_org                     2.14
    mirror_co_uk                        7.92
    nbcnews_com                         9.11
    newstatesman_com                    5.03
    npr_org                             8.71
    nytimes_com                         9.56
    order-order_com                     3.77
    propublica_org                      5.82
    reuters_com                         8.24
    rt_com                              8.41
    skwawkbox_org                       1.29
    telegraph_co_uk                     8.35
    thecanary_co                        5.02
    theguardian_com                     8.93
    thetimes_co_uk                      6.49
    washingtonpost_com                  8.69
    westmonster_com                     2.43
    yournewswire_com                    3.20
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   3.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        8.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     4.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   3.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    libdemvoice_org                     3.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    7.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        7.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     4.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     19800
    1.00     18359
    2.00     12406
    3.00      7252
    4.00      5659
    5.00      4814
    6.00      4111
    7.00      3704
    8.00      3257
    9.00      2964
    10.00     2698
    11.00     2548
    12.00     2324
    13.00     2352
    14.00     2250
    15.00     2039
    16.00     1942
    17.00     1853
    18.00     1703
    19.00     1651
    20.00     1467
    21.00     1480
    22.00     1371
    23.00     1314
    24.00     1240
    25.00     1224
    26.00     1143
    27.00     1102
    28.00     1141
    29.00     1115
             ...
    65.00      133
    66.00      117
    67.00       99
    68.00       86
    69.00       76
    70.00       72
    71.00       50
    72.00       42
    73.00       52
    74.00       40
    75.00       35
    76.00       31
    77.00       36
    78.00       28
    79.00       27
    80.00       17
    81.00       22
    82.00       24
    83.00       21
    84.00       22
    85.00       13
    86.00       10
    87.00        8
    88.00        9
    89.00        6
    91.00        2
    92.00        2
    93.00        4
    94.00        6
    95.00        1
    Name: attention_index, Length: 95, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      3f66918e16e7430d7aa13244df084a801f52d862
      http://www.bbc.co.uk/news/uk-england-london-40...
      Vehicle hits pedestrians in Finsbury Park
      2017-06-19 00:21:00.909
      2017-06-19 00:17:28
      268459.00
      575.75
      2017-06-19T06:23:59.871Z
      62981.00
      156410.00
      49068.00
      ...
      True
      42352036.00
      2017-06-19T05:20:41.000Z
      96
      50.00
      20.00
      10.00
      15.00
      45.00
      95.00

      ba671a1169cb7269431511acfa3d2468e3ab3d31
      http://www.bbc.co.uk/news/uk-england-london-40...
      'Serious fire' in west London tower block
      2017-06-14 01:45:01.257
      2017-06-14 01:43:03
      534172.00
      1356.58
      2017-06-14T07:12:00.581Z
      108074.00
      344028.00
      82070.00
      ...
      True
      42229337.00
      2017-06-14T06:56:51.000Z
      96
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      ec12fc2f7d45a68e7e361d4ad3fd654d56696e4a
      http://money.cnn.com/2017/06/23/media/white-ho...
      With cameras banned, CNN sends sketch artist t...
      2017-06-23 19:50:05.637
      2017-06-23 19:20:57
      138222.00
      327.33
      2017-06-23T21:16:02.827Z
      20559.00
      103899.00
      13764.00
      ...
      True
      27691050.00
      2017-06-23T20:02:55.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      13b55256066011820af20d76a366269878b61dd4
      http://www.cnn.com/2017/06/17/politics/cuba-tr...
      Havana: US in no 'condition to lecture us' on ...
      2017-06-17 19:39:16.930
      2017-06-17 19:35:58
      143341.00
      330.46
      2017-06-19T00:44:00.558Z
      13720.00
      113386.00
      16235.00
      ...
      True
      27664004.00
      2017-06-19T00:00:01.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      45e93af5a94cb7315465aeab9e10921a223fa7fe
      http://www.cnn.com/2017/06/16/us/michelle-cart...
      Texting suicide trial: Decision coming Friday
      2017-06-16 13:27:19.059
      2017-06-16 13:23:18
      222388.00
      573.30
      2017-06-16T15:56:02.213Z
      50970.00
      152144.00
      19274.00
      ...
      True
      27650967.00
      2017-06-16T15:22:39.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      9416d045643bdea7777098092253ecd807f5d0bd
      http://www.cnn.com/2017/06/17/us/missing-sailo...
      Missing Fitzgerald sailors found in flooded co...
      2017-06-18 02:18:16.117
      2017-06-18 02:14:27
      169181.00
      525.58
      2017-06-18T04:08:00.547Z
      18523.00
      135716.00
      14942.00
      ...
      True
      27659675.00
      2017-06-18T03:49:56.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      72e0b54da34d718d5078cd585f632e2e343c230f
      http://money.cnn.com/2017/06/29/media/mika-brz...
      Trump tweets shocking assault on Brzezinski, S...
      2017-06-29 14:10:09.750
      2017-06-29 13:57:41
      147817.00
      342.17
      2017-06-29T15:36:00.984Z
      59702.00
      76199.00
      11916.00
      ...
      True
      27726444.00
      2017-06-29T14:10:00.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      c5f69004e317f04fdbddd42f053d584f2a7a937a
      http://www.bbc.co.uk/news/uk-politics-40403434
      Conservatives agree pact with DUP to support M...
      2017-06-26 10:12:03.300
      2017-06-26 10:11:07
      60778.00
      1319.08
      2017-06-26T13:02:01.797Z
      23885.00
      29181.00
      7712.00
      ...
      True
      42500083.00
      2017-06-26T10:20:46.000Z
      96
      46.00
      20.00
      12.00
      15.00
      47.00
      93.00

      f2fc4fed26fb2c85439c4a522ca7b912fb45a7ac
      http://www.cnn.com/2017/06/14/politics/alexand...
      Rep. Scalise shot in Virginia
      2017-06-14 12:12:16.440
      2017-06-14 12:06:24
      100739.00
      344.48
      2017-06-14T13:14:01.184Z
      45271.00
      42509.00
      12959.00
      ...
      True
      27635745.00
      2017-06-14T12:10:29.000Z
      105
      49.00
      20.00
      9.00
      15.00
      44.00
      93.00

      391210d281522666180f138db4a4afdf8e570c2a
      http://www.bbc.co.uk/news/uk-40298473
      London fire: Tower victims 'may never be ident...
      2017-06-16 04:42:01.959
      2017-06-16 04:39:31
      64446.00
      152.13
      2017-06-16T10:58:00.580Z
      8740.00
      48942.00
      6764.00
      ...
      True
      42294330.00
      2017-06-16T10:45:24.000Z
      96
      47.00
      20.00
      11.00
      15.00
      46.00
      93.00

      ad85ebce93e51c550f95211d586d630b4bbc4c15
      http://www.cnn.com/2017/06/12/politics/donald-...
      Donald Trump just held the weirdest Cabinet me...
      2017-06-12 18:00:25.814
      2017-06-12 17:55:41
      312063.00
      17788.25
      2017-06-14T00:48:00.400Z
      118752.00
      157634.00
      35677.00
      ...
      True
      27620704.00
      2017-06-12T18:30:01.000Z
      105
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      d44dad6394bf1a8b082c3a6aae8e8224f0d3317f
      https://www.buzzfeed.com/katiejmbaker/how-accu...
      How Accusing A Powerful Man of Rape Drove A Co...
      2017-06-22 11:42:15.771
      2017-06-21 19:42:29
      159741.00
      159.14
      2017-06-22T17:52:02.091Z
      26662.00
      109005.00
      24074.00
      ...
      True
      2566810.00
      2017-06-22T15:08:00.000Z
      147
      50.00
      19.00
      12.00
      11.00
      42.00
      92.00

      d7a83a201eb18d9bb2519f26fd165e050991212d
      http://www.bbc.co.uk/news/live/uk-40147014
      Latest updates: London Bridge incident
      2017-06-03 21:51:01.607
      2017-06-03 21:49:50
      43622.00
      33.46
      2017-06-05T05:52:00.245Z
      6049.00
      33268.00
      4305.00
      ...
      True
      42030172.00
      2017-06-06T06:04:13.000Z
      96
      44.00
      20.00
      13.00
      15.00
      48.00
      92.00

      ff2319485fe811a76650106d805ad8f5e111b560
      http://www.bbc.co.uk/news/world-us-canada-4033...
      US student sent home from N Korea dies
      2017-06-19 20:39:11.797
      2017-06-19 20:38:26
      73632.00
      392.96
      2017-06-19T21:16:01.246Z
      12664.00
      56480.00
      4488.00
      ...
      True
      42364593.00
      2017-06-19T20:45:54.000Z
      96
      48.00
      20.00
      8.00
      15.00
      43.00
      91.00

      96bb0ec657ef482c9b2274e70aec5a3f2a16f517
      http://www.cnn.com/2017/06/19/politics/otto-wa...
      Former North Korea detainee Otto Warmbier dies
      2017-06-19 20:48:17.719
      2017-06-19 20:45:04
      124110.00
      827.73
      2017-06-20T05:02:00.797Z
      30469.00
      84401.00
      9240.00
      ...
      True
      27668640.00
      2017-06-19T20:56:01.000Z
      105
      50.00
      20.00
      6.00
      15.00
      41.00
      91.00

      054797b69042e86657db71128c4ce1f2587d483c
      http://www.huffingtonpost.com/entry/philando-c...
      There Is No Justice In America For Black Peopl...
      2017-06-16 22:36:18.257
      2017-06-16 13:23:18
      92735.00
      72.83
      2017-06-17T03:54:01.762Z
      7592.00
      69303.00
      15840.00
      ...
      True
      9547916.00
      2017-06-18T10:00:22.000Z
      215
      49.00
      17.00
      10.00
      13.00
      40.00
      89.00

      42880945bf2d236b3b499bca975fb190fe0d814f
      http://www.cnn.com/2017/06/29/politics/lawmake...
      GOP lawmakers blast Trump's 'Morning Joe' tweets
      2017-06-29 15:06:19.332
      2017-06-29 14:59:22
      69742.00
      299.25
      2017-06-29T18:46:00.835Z
      22331.00
      41285.00
      6126.00
      ...
      True
      27728995.00
      2017-06-29T18:30:01.000Z
      105
      47.00
      20.00
      7.00
      15.00
      42.00
      89.00

      34e5173cca66c7c6cf0e29a4547730cb4d2a84bd
      https://www.buzzfeed.com/davidmack/adam-west-dies
      Adam West, Star Of "Batman" And "Family Guy," ...
      2017-06-10 15:54:22.106
      2017-06-10 15:47:47
      82577.00
      146.46
      2017-06-12T23:56:00.681Z
      5108.00
      67441.00
      10028.00
      ...
      True
      2552440.00
      2017-06-10T16:06:28.000Z
      147
      48.00
      19.00
      11.00
      11.00
      41.00
      89.00

      963766e28985b4b91b6bc5865e23b0b591de2ee0
      http://www.huffingtonpost.com/entry/kellyanne-...
      Kellyanne Conway Defends Medicaid Cuts, Says A...
      2017-06-25 16:33:13.540
      2017-06-25 06:54:16
      238597.00
      416.00
      2017-06-25T22:50:00.507Z
      75871.00
      141473.00
      21253.00
      ...
      True
      9571073.00
      2017-06-25T18:00:32.000Z
      215
      50.00
      17.00
      9.00
      13.00
      39.00
      89.00

      9fefd3fd2927cc99c610a7d5f006f0c754bf25ea
      https://www.buzzfeed.com/danrather/dan-rather-...
      Dan Rather On Why America Needs More Empathy
      2017-06-23 05:40:03.972
      2017-06-22 15:49:53
      67425.00
      35.58
      2017-06-23T14:00:02.189Z
      3543.00
      55014.00
      8868.00
      ...
      True
      2569052.00
      2017-06-23T13:38:23.000Z
      147
      47.00
      19.00
      12.00
      11.00
      42.00
      89.00

      c647c83f01035dbdcc2b070ba7ecba88e3336df4
      http://www.huffingtonpost.com/entry/congress-m...
      After 152 Mass Shootings In 2017, It Took An A...
      2017-06-14 18:15:22.292
      2017-06-14 06:41:10
      86579.00
      151.58
      2017-06-15T00:18:00.918Z
      15148.00
      55704.00
      15727.00
      ...
      True
      9537965.00
      2017-06-15T00:00:27.000Z
      215
      49.00
      17.00
      10.00
      13.00
      40.00
      89.00

      296d5063aed71b4a169036cf00577cd0dff2909f
      http://www.bbc.co.uk/news/world-middle-east-40...
      Four countries cut links with Qatar
      2017-06-05 03:18:01.472
      2017-06-05 03:16:17
      32246.00
      30.79
      2017-06-05T03:32:00.859Z
      7424.00
      20636.00
      4186.00
      ...
      True
      41984942.00
      2017-06-05T03:19:49.000Z
      96
      42.00
      20.00
      11.00
      15.00
      46.00
      88.00

      e567ffdc11ecda98cee5375597c7f6adc3f33164
      https://www.nytimes.com/2017/06/26/us/politics...
      Senate Republicans Revise Health Bill to Penal...
      2017-06-26 19:03:08.254
      2017-06-26 19:00:21
      109231.00
      277.04
      2017-06-26T21:32:01.762Z
      23461.00
      68971.00
      16799.00
      ...
      True
      14149338.00
      2017-06-26T20:21:25.000Z
      120
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      88d97839967e8d91fd44aba7b518bd284a78586d
      http://www.cnn.com/2017/06/25/us/police-shooti...
      Three trials, no convictions in fatal police s...
      2017-06-25 10:06:14.535
      2017-06-25 10:03:21
      42763.00
      129.57
      2017-06-25T22:44:00.413Z
      6898.00
      29438.00
      6427.00
      ...
      True
      27701396.00
      2017-06-25T21:45:00.000Z
      105
      44.00
      20.00
      9.00
      15.00
      44.00
      88.00

      0d046b0c4a525aa3763aa5197137e8488893d31e
      http://www.bbc.co.uk/news/world-us-canada-4025...
      US appeals court blocks 'travel ban'
      2017-06-12 17:15:03.277
      2017-06-12 17:14:12
      58503.00
      2172.08
      2017-06-12T23:51:59.523Z
      2794.00
      52797.00
      2912.00
      ...
      True
      42174566.00
      2017-06-12T17:16:46.000Z
      96
      46.00
      20.00
      7.00
      15.00
      42.00
      88.00

      52f48b170d2f9c05ad808d4f941c06c70d4ce7f3
      http://www.cnn.com/2017/06/15/politics/otto-wa...
      North Korea claims Otto Warmbier released on h...
      2017-06-15 08:06:15.656
      2017-06-15 08:04:55
      39036.00
      80.53
      2017-06-15T23:52:00.548Z
      10271.00
      24688.00
      4077.00
      ...
      True
      27646995.00
      2017-06-16T05:00:00.000Z
      105
      44.00
      20.00
      9.00
      15.00
      44.00
      88.00

      381031bbe0f2f0eee70ba7e1dd5ca919a3f49a5d
      https://www.buzzfeed.com/albertsamaha/kids-are...
      Kids Are Quoting Trump To Bully Their Classmat...
      2017-06-06 15:57:18.135
      2017-06-06 15:55:37
      59546.00
      26.87
      2017-06-06T16:58:01.397Z
      13086.00
      36247.00
      10213.00
      ...
      True
      2540569.00
      2017-06-06T16:31:56.000Z
      147
      46.00
      19.00
      12.00
      11.00
      42.00
      88.00

      94407f80f29dab1b96e1c1b15b7116093ac753b4
      http://www.huffingtonpost.com/entry/new-dashca...
      New Dashcam Footage Shows Philando Castile War...
      2017-06-20 20:35:08.614
      2017-06-20 20:30:47
      133442.00
      187.00
      2017-06-20T23:00:00.948Z
      46503.00
      71940.00
      14999.00
      ...
      True
      9555473.00
      2017-06-20T21:20:13.000Z
      215
      50.00
      17.00
      8.00
      13.00
      38.00
      88.00

      d46f831ddd3ff685c868f37349823ce10580e269
      https://www.nytimes.com/2017/06/29/business/me...
      Trump Continues Media Hostilities With Persona...
      2017-06-29 14:24:00.610
      2017-06-29 14:21:31
      225957.00
      451.79
      2017-06-29T15:14:00.540Z
      85909.00
      120760.00
      19288.00
      ...
      True
      14163117.00
      2017-06-29T14:25:59.000Z
      120
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      f7c958e2526db034a2f23b06af7940a7e46ac0e1
      https://www.nytimes.com/2017/06/01/climate/ame...
      Bucking Trump, These Cities, States and Compan...
      2017-06-02 00:18:00.428
      2017-06-02 00:16:58
      120416.00
      108.48
      2017-06-03T11:55:59.399Z
      5948.00
      103932.00
      10536.00
      ...
      True
      14030628.00
      2017-06-02T01:25:00.000Z
      120
      50.00
      20.00
      4.00
      14.00
      38.00
      88.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      890bda1ff249f5dc7f20c92018e99f5a38cde4a4
      https://www.washingtonpost.com/sports/national...
      Ramirez, Leon homer, Red Sox beat Angels 9-4 o...
      2017-06-24 02:45:22.045
      2017-06-24 02:37:31
      1.00
      0.02
      2017-06-24T11:14:01.967Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      d638e69d8285de641043e6266ca8736cb90635f2
      https://www.washingtonpost.com/national/sherif...
      Sheriff’s office: Florida home set on fire to ...
      2017-06-13 00:57:12.567
      2017-06-13 00:56:33
      0.00
      0.00
      2017-06-13T01:10:01.560Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e5ebd9b1492b1b083f2a82ae275c82d959391d18
      https://www.washingtonpost.com/sports/mystics/...
      Moore scores 18 of 22 points in 1st half, Lynx...
      2017-06-24 02:45:18.521
      2017-06-24 02:38:19
      0.00
      0.00
      2017-06-24T16:24:02.051Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5b08ad9043cd54aef1c3eb96c95c8db82d8a5303
      https://www.washingtonpost.com/sports/mystics/...
      WNBA Capsules
      2017-06-24 02:45:19.155
      2017-06-24 02:40:12
      0.00
      0.00
      2017-06-24T16:24:02.052Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6f7412f5f2ecbd52b02d2bc9ce4971dc82dd364a
      https://www.washingtonpost.com/sports/national...
      Longoria, Peterson lead Rays over Orioles 15-5
      2017-06-24 02:45:15.953
      2017-06-24 02:41:35
      0.00
      0.00
      2017-06-24T16:24:02.049Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      26455a43b5b6a1e51965618bf485b1e273078782
      https://www.washingtonpost.com/sports/dcunited...
      Moor, Giovinco score, Toronto FC beats Revolut...
      2017-06-24 02:45:15.587
      2017-06-24 02:43:45
      0.00
      0.00
      2017-06-24T16:24:01.412Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      83c68f3855ed7a50ff3b855398daaf16793f05df
      https://www.washingtonpost.com/national/boyhoo...
      ‘Boyhood’ actress gets deferred sentence for d...
      2017-06-24 02:54:11.763
      2017-06-24 02:45:15
      0.00
      0.00
      2017-06-24T16:34:01.923Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6df7029db4adf1a9b6edc9978a8d20fa4e5be928
      https://www.washingtonpost.com/entertainment/m...
      Billy Joel postpones concert because of viral ...
      2017-06-05 17:24:15.915
      2017-06-05 17:20:13
      0.00
      0.00
      2017-06-06T15:06:00.596Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f8d4f6ed43aa72554e650225dc4446c4f23c3dcf
      https://www.washingtonpost.com/sports/national...
      Oakland’s Cotton, Pinder leave game because of...
      2017-06-24 02:54:08.525
      2017-06-24 02:48:18
      0.00
      0.00
      2017-06-24T16:34:01.918Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      7388d6fcd01859383b9cd33245dd7c443d03bebb
      https://www.washingtonpost.com/national/albuqu...
      Albuquerque TV truck stolen while crew worked ...
      2017-06-24 02:54:11.193
      2017-06-24 02:51:11
      0.00
      0.00
      2017-06-24T16:34:01.922Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      06bde8d17913e9c1a451b576f2a1a131322adb95
      https://www.washingtonpost.com/sports/national...
      Mejia, 3 relievers hold Indians to 4 hits in T...
      2017-06-24 02:54:08.664
      2017-06-24 02:52:29
      0.00
      0.00
      2017-06-24T16:34:01.920Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      d34022437d3a5b727d7a5dddbff69978edb2cb88
      https://www.washingtonpost.com/national/correc...
      Police arrest 2, seek 2 in attack on French Qu...
      2017-06-27 23:15:14.940
      2017-06-27 23:10:15
      0.00
      0.00
      2017-06-28T09:48:02.017Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e4afb9511e15df7c456933d941e01527526d0ba7
      http://www.dailymail.co.uk/tvshowbiz/article-4...
      Vetern actor John Wood slams ‘cheap’ reality TV
      2017-06-21 00:39:20.416
      2017-06-21 00:36:40
      1.00
      0.02
      2017-06-21T01:54:02.342Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      90fd81315c4a558c66b7f0a288d2e1e8b0425526
      https://www.washingtonpost.com/national/castil...
      Castile protesters target city where officer w...
      2017-06-19 00:09:18.158
      2017-06-19 00:05:16
      0.00
      0.00
      2017-06-19T00:22:01.397Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      e596f2d2450708cebfea51dbb852d8ee2fbc486d
      http://www.bbc.co.uk/news/av/world-us-canada-4...
      Florida shootings: Police say seven survivors ...
      2017-06-05 17:24:01.463
      2017-06-05 17:20:37
      1.00
      0.02
      2017-06-05T20:30:01.072Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      658544b592d7448b14d47540d36a1b6fe2a637df
      https://www.washingtonpost.com/sports/national...
      Seager drives in 3 as Mariners beat Rangers 7-3
      2017-06-19 00:09:13.608
      2017-06-19 00:05:50
      0.00
      0.00
      2017-06-19T00:22:01.387Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      44f153a8a1259ea0188b51acf3be607569de53a9
      https://www.washingtonpost.com/national/energy...
      Utah blaze caused by torch used to kill weeds
      2017-06-21 00:57:44.124
      2017-06-21 00:35:18
      1.00
      0.02
      2017-06-21T23:54:01.866Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f9c0cc2f39ec2a2c4f574e5f6c8d262b52c53c3b
      https://www.washingtonpost.com/sports/capitals...
      Blackhawks deal Panarin and Hjalmarsson, get S...
      2017-06-24 03:09:05.719
      2017-06-24 03:01:17
      0.00
      0.00
      2017-06-24T16:48:02.101Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      46871120ffb8edea4dd8e512fd10283492cf3c75
      http://www.dailymail.co.uk/sport/football/arti...
      Huddersfield 'will bring something new the Pre...
      2017-06-14 10:27:23.385
      2017-06-14 10:25:44
      1.00
      0.08
      2017-06-14T10:40:00.925Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      828056419a1f6e2fd1d960800c71ad99e7b915bc
      https://www.washingtonpost.com/politics/congre...
      The Latest: Mattis ‘shocked’ by state of comba...
      2017-06-13 01:00:06.318
      2017-06-13 00:56:30
      1.00
      0.02
      2017-06-13T05:22:01.817Z
      0.00
      0.00
      1.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f38eb6ac33de53a3f93ed14a2f6848cf707d73ff
      https://www.washingtonpost.com/local/winning-n...
      Winning numbers drawn in ‘Cash 5 Night’ game
      2017-06-27 03:27:15.131
      2017-06-27 03:21:30
      0.00
      0.00
      2017-06-27T03:40:02.220Z
      0.00
      0.00
      0.00
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4a644524bcf5583a0820bbcbbef7b696dd8c87d0
      https://www.ft.com/content/0be07d54-101a-3943-...
      Queen’s Speech signals May’s quest for ‘trust ...
      2017-06-21 11:00:06.580
      2017-06-21 10:48:40
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      1596
      nan
      0.00
      1.00
      0.00
      1.00
      nan

      9dee6b1e770e32d6cf192e8dfa93b50d3bd79bfa
      https://www.ft.com/content/b91aef3e-5669-11e7-...
      Queen’s Speech sets out Brexit programme
      2017-06-21 11:00:07.574
      2017-06-21 10:51:28
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      1596
      nan
      0.00
      1.00
      0.00
      1.00
      nan

      4ed309b88c95f577ee9604d69326607f8ae8b438
      http://www.independent.co.uk/life-style/fashio...
      Halima Aden is the hijab-wearing model defying...
      2017-06-21 11:00:04.351
      2017-06-21 10:54:37
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      386
      nan
      0.00
      1.00
      0.00
      1.00
      nan

      308dff9aa0376d0f00afbb64de1516535614537e
      http://www.independent.co.uk/sport/general/box...
      Who will be in Conor McGregor's corner for his...
      2017-06-21 11:00:17.565
      2017-06-21 10:54:45
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      386
      nan
      0.00
      0.00
      0.00
      0.00
      nan

      6b9c079eab5b76182ebf10e288e3e17139dd761b
      http://www.independent.co.uk/news/uk/politics/...
      Queen’s Speech: Dennis Skinner jokes 'get your...
      2017-06-21 11:00:16.779
      2017-06-21 10:55:08
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      386
      nan
      0.00
      1.00
      0.00
      1.00
      nan

      fdccd726d595c1c1f4415a5dcc275031b7059195
      http://www.libdemvoice.org/lets-be-the-party-o...
      Let’s be the party of the empowered individual
      2017-06-21 11:00:11.915
      2017-06-21 10:55:59
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      True
      8482.00
      2017-06-21T10:55:59.000Z
      344992
      nan
      7.00
      1.00
      3.00
      11.00
      nan

      4c8dbbc31a9f3d10446feace17fc40cb16a4bbab
      https://www.washingtonpost.com/politics/pollin...
      Is Mayor Bowser doing an excellent job, good, ...
      2017-06-21 11:00:13.150
      2017-06-21 10:56:22
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      191
      nan
      0.00
      0.00
      0.00
      0.00
      nan

      1abf220586aa0e31ea0dc2c1ed4bff63bf07cdc3
      https://www.washingtonpost.com/politics/pollin...
      Do you approve or disapprove of the way that M...
      2017-06-21 11:00:10.526
      2017-06-21 10:58:04
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      False
      nan
      NaN
      191
      nan
      0.00
      0.00
      0.00
      0.00
      nan

      3d4b7794605f355bed5c828735950c5d8be3ef53
      https://www.rt.com/politics/393341-most-russia...
      Russians overwhelmingly back Putin policies – ...
      2017-06-21 11:00:13.302
      2017-06-21 10:58:52
      nan
      0.00
      NaN
      nan
      nan
      nan
      ...
      True
      4486052.00
      2017-06-22T16:15:00.000Z
      365
      nan
      0.00
      3.00
      12.00
      15.00
      nan

138245 rows × 25 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      7a429951e59186566f7dd6b638b05506c5176a93
      http://www.bbc.co.uk/news/live/election-2017-4...
      Election Live
      2017-06-06 11:06:01.620
      2017-06-06 11:03:23
      0.00
      0.00
      2017-06-07T11:53:59.853Z
      0.00
      0.00
      0.00
      ...
      42103917.00
      2017-06-09T01:49:34.000Z
      96
      0.00
      20.00
      11.00
      15.00
      46.00
      46.00
      46.00

      84f5fd4a5be0159b455ecc099b2fcd0e42e267f5
      https://www.buzzfeed.com/nidhisubbaraman/insid...
      Inside A Controversial Clinic That Teaches Peo...
      2017-06-28 09:10:13.503
      2017-06-26 14:57:44
      2.00
      0.00
      2017-06-28T09:24:03.146Z
      0.00
      0.00
      2.00
      ...
      2584728.00
      2017-06-29T06:04:00.000Z
      147
      1.00
      19.00
      13.00
      12.00
      44.00
      45.00
      43.00

      cd584979a93abc337cda37bbdf01912caa0381d9
      https://www.buzzfeed.com/claudiarosenbaum/bill...
      Bill Cosby's Rape Trial Is About To Start And ...
      2017-06-03 14:33:16.882
      2017-06-01 01:03:34
      1.00
      0.08
      2017-06-03T14:46:00.622Z
      0.00
      0.00
      1.00
      ...
      2536263.00
      2017-06-03T21:00:00.000Z
      147
      0.00
      19.00
      12.00
      11.00
      42.00
      42.00
      42.00

      0e3250b88c58ded0ebc4ab2d90e0b541a7903cf8
      https://www.buzzfeed.com/buzzfeednews/live-upd...
      Live Updates: Britain Heads To The Polls
      2017-06-08 08:09:17.133
      2017-06-07 15:47:54
      1.00
      0.02
      2017-06-08T09:24:00.945Z
      0.00
      0.00
      1.00
      ...
      2548289.00
      2017-06-09T10:09:32.000Z
      147
      0.00
      19.00
      12.00
      11.00
      42.00
      42.00
      42.00

      d27cc0ade64ffced5b2b1e29b71080e9f0de3d84
      https://www.buzzfeed.com/tomwarren/secrets-of-...
      The Death Of The Spy In The Bag Is Linked To R...
      2017-06-20 16:57:12.742
      2017-06-15 15:33:12
      6.00
      0.50
      2017-06-20T17:10:02.173Z
      0.00
      1.00
      5.00
      ...
      2561814.00
      2017-06-20T17:44:00.000Z
      147
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      dd585ff206f769b8310cc36c58d05374dc1a379c
      https://www.buzzfeed.com/janebradley/scientist...
      Scientist Who Helped Connect Litvinenko's Murd...
      2017-06-19 17:18:20.157
      2017-06-11 14:28:46
      10.00
      0.83
      2017-06-19T17:32:01.728Z
      0.00
      1.00
      9.00
      ...
      2561137.00
      2017-06-19T18:24:00.000Z
      147
      2.00
      19.00
      12.00
      11.00
      42.00
      44.00
      40.00

      54b1489aa247c088391589971ac55fd576fd932e
      https://www.buzzfeed.com/alysonmartin/workers-...
      Workers Are Being Fired For Using Medical Cann...
      2017-06-10 14:03:20.115
      2017-06-07 19:09:24
      6.00
      0.50
      2017-06-10T14:16:00.647Z
      0.00
      1.00
      5.00
      ...
      2552235.00
      2017-06-10T14:37:00.000Z
      147
      1.00
      19.00
      11.00
      11.00
      41.00
      42.00
      40.00

      08e81c12ae1d5ed846572488c04cb48a4aed3120
      http://www.huffingtonpost.com/entry/jason-chaf...
      As Jason Chaffetz Retires From Congress, Here ...
      2017-07-01 13:15:15.581
      2017-06-29 13:47:08
      6.00
      0.50
      2017-07-01T13:28:03.393Z
      0.00
      0.00
      6.00
      ...
      9588082.00
      2017-07-01T15:00:48.000Z
      215
      1.00
      17.00
      10.00
      13.00
      40.00
      41.00
      39.00

      c5426af122810f4643a4e6acc45f225f7625492c
      http://www.huffingtonpost.com/entry/covered-ca...
      Trump Says Obamacare Is 'Imploding.' That's Ne...
      2017-06-10 12:15:14.571
      2017-06-06 07:51:32
      5.00
      0.42
      2017-06-10T12:28:00.860Z
      0.00
      0.00
      5.00
      ...
      9525206.00
      2017-06-10T17:30:21.000Z
      215
      1.00
      17.00
      10.00
      13.00
      40.00
      41.00
      39.00

      6fa1b04f04eb3998fbb7dca8aa172c410b085c46
      https://www.buzzfeed.com/hayesbrown/qatar-the-...
      Here's Why The Hell Everyone Is Talking About ...
      2017-06-07 17:00:24.875
      2017-06-06 21:13:04
      56.00
      0.58
      2017-06-07T17:14:01.577Z
      3.00
      32.00
      21.00
      ...
      2555275.00
      2017-06-11T22:34:00.000Z
      147
      6.00
      19.00
      15.00
      11.00
      45.00
      51.00
      39.00

      69599ebf0697f0f64b6a22d09a9f0e51cd1fa726
      http://www.huffingtonpost.com/entry/march-agai...
      The 'March Against Sharia' Protests Are Really...
      2017-06-10 12:06:17.670
      2017-06-08 05:55:58
      2.00
      0.17
      2017-06-10T12:20:00.737Z
      0.00
      0.00
      2.00
      ...
      9524305.00
      2017-06-10T14:00:20.000Z
      215
      1.00
      17.00
      10.00
      13.00
      40.00
      41.00
      39.00

      1626385c4f54d5faf2a9ea677374b41bc8ad4c14
      http://www.huffingtonpost.com/entry/medicaid-c...
      Cerebral Palsy Didn't Stop This College Junior...
      2017-06-17 12:06:15.532
      2017-06-14 16:19:57
      2.00
      0.17
      2017-06-17T12:20:01.495Z
      0.00
      1.00
      1.00
      ...
      9545617.00
      2017-06-17T16:00:23.000Z
      215
      1.00
      17.00
      10.00
      13.00
      40.00
      41.00
      39.00

      4fa2031b5d188714f45db8dda40279dc0e87f815
      https://www.buzzfeed.com/jwmccormack/the-new-t...
      The New "Twin Peaks" Is A Reckoning, Not A Reboot
      2017-06-04 17:55:04.486
      2017-06-02 18:01:43
      30.00
      0.08
      2017-06-04T18:08:00.895Z
      4.00
      17.00
      9.00
      ...
      2555095.00
      2017-06-11T19:34:00.000Z
      147
      4.00
      19.00
      12.00
      11.00
      42.00
      46.00
      38.00

      6a659f20f1b29794034ce27d90e37330177e1a8f
      http://www.huffingtonpost.com/entry/republican...
      Food Stamps Are Holding Up The Republican Agenda
      2017-07-01 12:42:16.235
      2017-06-30 11:15:42
      8.00
      0.67
      2017-07-01T12:56:03.408Z
      1.00
      1.00
      6.00
      ...
      9588662.00
      2017-07-01T17:30:12.000Z
      215
      2.00
      17.00
      10.00
      13.00
      40.00
      42.00
      38.00

      4b448ade52ff96e054a6d43e9abb10231d430ae0
      https://www.buzzfeed.com/stephaniemlee/weed-br...
      People Get High To Help This Startup Make A We...
      2017-06-17 18:33:31.317
      2017-06-16 21:39:31
      46.00
      0.35
      2017-06-17T20:50:01.293Z
      2.00
      9.00
      35.00
      ...
      2559967.00
      2017-06-17T23:30:00.000Z
      147
      6.00
      19.00
      13.00
      11.00
      43.00
      49.00
      37.00

      fcc2765ab21d6efbefe6567df3d9c9273f9f06c3
      https://www.buzzfeed.com/buzzfeednews/live-upd...
      Live Updates: Trump Responds To Terror Attack ...
      2017-06-05 14:24:17.153
      2017-06-05 14:22:04
      41.00
      0.15
      2017-06-05T20:36:00.688Z
      7.00
      11.00
      23.00
      ...
      2541464.00
      2017-06-07T01:44:00.000Z
      147
      5.00
      19.00
      12.00
      11.00
      42.00
      47.00
      37.00

      13137e842019a89c195b1b74699153678d303d42
      http://www.huffingtonpost.com/entry/blimp-cras...
      Blimp Reportedly Crashes To The Ground At U.S....
      2017-06-15 16:51:21.965
      2017-06-11 14:09:36
      1.00
      0.08
      2017-06-15T17:04:02.147Z
      0.00
      0.00
      1.00
      ...
      9539816.00
      2017-06-15T18:01:34.000Z
      215
      0.00
      17.00
      5.00
      13.00
      35.00
      35.00
      35.00

      d6cb51ff0a4bc8cf47052152dd69e0448dd2b76d
      http://www.huffingtonpost.com/entry/trump-food...
      The Food Stamp Program Is An Overwhelming Succ...
      2017-06-12 01:42:19.329
      2017-06-09 19:09:39
      1.00
      0.08
      2017-06-12T01:56:00.462Z
      0.00
      0.00
      1.00
      ...
      9529876.00
      2017-06-12T19:00:16.000Z
      215
      0.00
      17.00
      4.00
      13.00
      34.00
      34.00
      34.00

      c74109be8992c252f04f993050b453e184a137ca
      https://www.buzzfeed.com/matthewchampion/arian...
      Ariana Grande's Concert For The Victims Of The...
      2017-06-04 08:30:18.360
      2017-06-03 13:35:10
      75.00
      0.92
      2017-06-04T08:44:00.839Z
      0.00
      53.00
      22.00
      ...
      2537990.00
      2017-06-04T16:00:00.000Z
      147
      8.00
      19.00
      11.00
      11.00
      41.00
      49.00
      33.00

      1984d05566d80d2c2c8b1d319a541e21595046b2
      https://www.buzzfeed.com/charliewarzel/twitter...
      Twitter's Pro-Trump Bot Crisis Is Really A Hum...
      2017-06-05 00:03:17.557
      2017-06-04 22:49:32
      113.00
      0.39
      2017-06-05T03:22:01.726Z
      7.00
      21.00
      85.00
      ...
      2553228.00
      2017-06-10T23:44:00.000Z
      147
      9.00
      19.00
      12.00
      11.00
      42.00
      51.00
      33.00

      b1263b3ab5c2ad7b9fdccf8810242d71889bbe3b
      https://www.buzzfeed.com/johnhudson/tillerson-...
      Tillerson And Mattis Say China Needs To Do Mor...
      2017-06-21 21:12:24.499
      2017-06-21 20:56:24
      47.00
      0.15
      2017-06-22T06:44:00.501Z
      11.00
      23.00
      13.00
      ...
      2565964.00
      2017-06-22T06:00:00.000Z
      147
      6.00
      19.00
      9.00
      11.00
      39.00
      45.00
      33.00

      28db6e2ff831e402e7441d45860001d680a5d392
      http://www.huffingtonpost.com/entry/congressio...
      Man Who Shot Republicans At Baseball Practice ...
      2017-06-21 15:06:17.497
      2017-06-20 12:18:25
      2.00
      0.17
      2017-06-21T15:20:02.216Z
      0.00
      0.00
      2.00
      ...
      9558262.00
      2017-06-21T16:30:28.000Z
      215
      1.00
      17.00
      4.00
      13.00
      34.00
      35.00
      33.00

      5fcc4db9ba428af538c54c8f42d261114e487021
      http://www.dailymail.co.uk/news/article-459767...
      Man filmed romping on flight had pregnant fian...
      2017-06-12 21:54:28.106
      2017-06-12 21:52:27
      0.00
      503.14
      2017-06-14T01:22:01.220Z
      0.00
      0.00
      0.00
      ...
      11430190.00
      2017-06-12T22:15:09.000Z
      158
      0.00
      18.00
      1.00
      14.00
      33.00
      33.00
      33.00

      ffa5e3e93bbbf8f8e9bfbe46b7ba0060b0cae98f
      http://www.huffingtonpost.com/entry/senate-hea...
      Secret's Out: The Senate Health Care Bill Is A...
      2017-06-22 16:57:32.341
      2017-06-21 11:28:55
      2.00
      0.17
      2017-06-22T17:10:03.533Z
      0.00
      0.00
      2.00
      ...
      9562485.00
      2017-06-22T23:15:12.000Z
      215
      1.00
      17.00
      3.00
      13.00
      33.00
      34.00
      32.00

      8cf42135b5ed57eacc66f1ce82b2a55cee0cbce5
      https://www.buzzfeed.com/bimadewunmi/the-world...
      Jennifer Lopez Doesn't Need To Be The Best At ...
      2017-06-07 01:40:05.860
      2017-06-06 16:39:11
      138.00
      0.73
      2017-06-07T02:56:00.619Z
      19.00
      100.00
      19.00
      ...
      2541508.00
      2017-06-07T02:24:00.000Z
      147
      10.00
      19.00
      12.00
      11.00
      42.00
      52.00
      32.00

25 rows × 26 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      4c7609641edd2e0a574b3fb536f482e609adc5ec
      https://www.buzzfeed.com/guillermodelpalacio/i...
      31 imágenes para no olvidar que los españoles ...
      2017-06-20 09:27:16.715
      2017-06-20 09:26:43
      116818.00
      43.58
      2017-06-22T21:30:03.836Z
      8026.00
      80705.00
      28087.00
      ...
      nan
      NaN
      147
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      7ba7462db1e44e49212622cab8413b06428a0950
      https://www.buzzfeed.com/rafaelcabrera/15-imag...
      15 Imágenes impactantes de la tormenta que cay...
      2017-06-29 01:48:11.878
      2017-06-29 01:22:04
      167255.00
      850.90
      2017-06-29T21:06:02.588Z
      22752.00
      111032.00
      33471.00
      ...
      nan
      NaN
      147
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      c54f9b5c9d2757ea2b06346eec627b84ac1e78f4
      http://www.huffingtonpost.com/entry/worlds-top...
      World’s Top Religious Leaders Issue Rare Joint...
      2017-06-15 20:57:19.154
      2017-06-15 09:17:18
      448813.00
      158.47
      2017-06-16T13:30:02.716Z
      22045.00
      353405.00
      73363.00
      ...
      nan
      NaN
      215
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      84227db2d31a582373ad12c7b15657a51fd388f7
      https://www.buzzfeed.com/alexandrearagao/ladra...
      Homem que filmou tatuagem na testa de adolesce...
      2017-06-12 20:57:21.242
      2017-06-12 20:56:33
      184604.00
      8801.75
      2017-06-14T00:42:01.983Z
      28753.00
      139238.00
      16613.00
      ...
      nan
      NaN
      147
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      962f960b36f46cbf3191a21031db3e063f8900e1
      https://www.nytimes.com/2017/06/24/science/at-...
      At Nursing Homes, Medicaid Cuts Would Reach Ev...
      2017-06-24 14:48:00.291
      2017-06-24 14:46:11
      108327.00
      73.94
      2017-06-24T21:26:00.754Z
      20984.00
      61227.00
      26116.00
      ...
      nan
      NaN
      120
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      429160ad3031aa2c79f8b3c3a22cc476b92be955
      https://www.washingtonpost.com/news/post-polit...
      D.C., Md. attorneys general to sue Trump, sayi...
      2017-06-12 03:09:04.484
      2017-06-12 03:01:52
      207836.00
      10811.33
      2017-06-12T23:47:59.456Z
      17336.00
      162530.00
      27970.00
      ...
      nan
      NaN
      191
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      9c8cd0a3c834a77649f7f50716faf517c0cb5770
      https://www.indy100.com/article/grenfell-tower...
      The Queen managed to make it to Grenfell Tower...
      2017-06-16 11:40:11.906
      2017-06-16 11:31:50
      325211.00
      321.17
      2017-06-16T17:56:00.155Z
      27130.00
      260540.00
      37541.00
      ...
      nan
      NaN
      5014
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      97720f7c41e93d719725f3a0153c27dfc6a780fb
      https://www.buzzfeed.com/robstott/for-the-firs...
      For The First Time, A Baby Has Been Breastfed ...
      2017-06-22 03:30:16.690
      2017-06-22 03:23:52
      86520.00
      34.21
      2017-06-23T20:36:01.692Z
      9146.00
      71637.00
      5737.00
      ...
      nan
      NaN
      147
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      448e59719a590ffdd6303da7f7d2a5b694ee6f57
      http://yournewswire.com/women-dna-man-sex/
      Women Absorb And Retain DNA From Every Man The...
      2017-06-23 18:20:21.452
      2017-06-23 16:49:46
      704855.00
      196.44
      2017-06-26T18:02:01.121Z
      330366.00
      287725.00
      86764.00
      ...
      nan
      NaN
      22568
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      80241e4ecbe07ee2811ff70f9195bebe5d140e97
      http://www.economist.com/news/europe/21723211-...
      Emmanuel Macron is headed for a record-breakin...
      2017-06-11 20:15:14.195
      2017-06-11 00:00:00
      182133.00
      2917.05
      2017-06-11T23:34:00.492Z
      19866.00
      97001.00
      65266.00
      ...
      nan
      NaN
      1825
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      85d929cdec6b3751da9b4f9c39ab7dd344a7fe43
      https://www.nytimes.com/2017/06/29/us/politics...
      Trump’s Attack on Mika Brzezinski Comes at a T...
      2017-06-29 19:03:03.303
      2017-06-29 19:01:08
      227642.00
      2406.53
      2017-06-30T01:28:02.442Z
      86514.00
      121534.00
      19594.00
      ...
      nan
      NaN
      120
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      87e5088e98973128ed6304376870945a262fba29
      https://www.buzzfeed.com/krystieyandoli/dascha...
      Dascha Polanco's Daughter Is On "Orange Is The...
      2017-06-20 22:40:08.651
      2017-06-20 18:24:11
      89243.00
      143.70
      2017-06-21T21:50:00.341Z
      18458.00
      65648.00
      5137.00
      ...
      nan
      NaN
      147
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      638a437a85bc8583b4b8e9db039c151580595420
      http://www.huffingtonpost.com/entry/gabrielle-...
      Navy Ship Named For Gun Violence Survivor Gabr...
      2017-06-10 19:51:14.736
      2017-06-10 09:28:24
      86077.00
      95.30
      2017-06-11T00:42:00.244Z
      1655.00
      80708.00
      3714.00
      ...
      nan
      NaN
      215
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      cfc98361a6b80f29c938c281ad83014ae6b73a5e
      http://www.foxnews.com/politics/2017/06/14/ste...
      Steve Scalise shooting: 'Political rhetorical ...
      2017-06-14 15:10:03.220
      2017-06-14 14:55:00
      207989.00
      484.84
      2017-06-15T12:02:00.832Z
      29184.00
      151749.00
      27056.00
      ...
      nan
      NaN
      285
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      310c7557be12b110787094f77d4314949daa6f71
      http://www.huffingtonpost.com/entry/pbs-school...
      PBS Runs A Three-Hour Series Glorifying The De...
      2017-06-13 15:06:20.535
      2017-06-13 06:37:02
      76804.00
      295.83
      2017-06-14T15:03:59.875Z
      25441.00
      42441.00
      8922.00
      ...
      nan
      NaN
      215
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      98439dbae2f4a5979c1ddd8b7c8808f79b98b4af
      https://www.washingtonpost.com/news/post-polit...
      Supreme Court sides with Missouri church in ma...
      2017-06-26 14:25:02.248
      2017-06-26 14:18:00
      112079.00
      582.75
      2017-06-29T03:06:01.238Z
      8972.00
      96821.00
      6286.00
      ...
      nan
      NaN
      191
      50.00
      1.00
      1.00
      0.00
      2.00
      52.00
      -48.00

      bff9fe8ac282796853418e50fcbcd903d14147c3
      https://www.washingtonpost.com/national/new-st...
      New study of Seattle’s $15 minimum wage says i...
      2017-06-26 17:30:16.062
      2017-06-26 17:20:09
      82458.00
      298.92
      2017-06-26T21:52:04.069Z
      39867.00
      31638.00
      10953.00
      ...
      nan
      NaN
      191
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      df0bf218acd5c9e976afe0f87b62f0d3190f8b7c
      http://www.foxnews.com/entertainment/2017/06/2...
      Greta Van Susteren out as MSNBC host
      2017-06-29 20:40:02.459
      2017-06-29 20:21:00
      151741.00
      525.25
      2017-07-01T19:06:02.577Z
      25683.00
      118905.00
      7153.00
      ...
      nan
      NaN
      285
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      d64c512d094dd0420f2d225e51a984e2b54ef56c
      https://www.washingtonpost.com/news/monkey-cag...
      The Confederate flag largely disappeared after...
      2017-06-12 10:09:11.824
      2017-06-12 10:00:34
      81882.00
      1640.03
      2017-06-14T00:42:01.562Z
      16653.00
      51151.00
      14078.00
      ...
      nan
      NaN
      191
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      4673d89bb6c77627584524f0d66ce27b5494e03e
      http://www.huffingtonpost.com/entry/what-will-...
      What Will It Take To Dump Trump?
      2017-06-19 03:12:15.780
      2017-06-18 18:51:44
      69827.00
      39.42
      2017-06-20T20:42:01.164Z
      22361.00
      38591.00
      8875.00
      ...
      nan
      NaN
      215
      47.00
      0.00
      0.00
      0.00
      0.00
      47.00
      -47.00

      a81450fd4d03b8785a32205c4e7a821ee62ad2f2
      http://www.nbcnews.com/news/world/shake-rattle...
      Airline captain says to pray during shaking, r...
      2017-06-25 19:15:03.075
      2017-06-25 19:12:43
      74450.00
      150.04
      2017-06-26T12:24:01.322Z
      3726.00
      65996.00
      4728.00
      ...
      nan
      NaN
      826
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      94e194bed94e3ddd5dd593015215a5d7f3ad7f2d
      http://www.foxnews.com/politics/2017/06/19/msn...
      MSNBC's Joy Ann Reid attacks Scalise as he rec...
      2017-06-19 15:40:04.035
      2017-06-19 15:15:00
      78559.00
      207.83
      2017-06-20T02:33:59.719Z
      18985.00
      55333.00
      4241.00
      ...
      nan
      NaN
      285
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      d9487b75fc22dbca814b435ebde19777d35f4bc4
      http://www.dailymail.co.uk/news/article-459327...
      Burns survivor Turia Pitt pregnant with first ...
      2017-06-11 12:42:23.089
      2017-06-11 12:39:59
      70871.00
      85.90
      2017-06-13T04:29:59.854Z
      2175.00
      68064.00
      632.00
      ...
      nan
      NaN
      158
      47.00
      0.00
      0.00
      0.00
      0.00
      47.00
      -47.00

      55190e4387e2fa026ca71bd908f7146962a58785
      http://www.foxnews.com/us/2017/06/16/hero-home...
      Hero homeowner holds escaped Georgia inmates a...
      2017-06-16 05:00:08.262
      2017-06-16 04:15:00
      703291.00
      1258.67
      2017-06-16T16:21:59.877Z
      64699.00
      553933.00
      84659.00
      ...
      nan
      NaN
      285
      50.00
      0.00
      3.00
      0.00
      3.00
      53.00
      -47.00

      eb5636dc24110a84803d8643cde76bb3d8f95e5f
      http://www.cnn.com/2017/06/14/us/kendrick-john...
      Kendrick Johnson gym mat death suit is dismissed
      2017-06-14 22:51:26.918
      2017-06-14 22:46:15
      138913.00
      116.33
      2017-06-15T18:22:00.920Z
      23638.00
      86824.00
      28451.00
      ...
      nan
      NaN
      105
      50.00
      0.00
      3.00
      0.00
      3.00
      53.00
      -47.00

25 rows × 26 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_2017-06-01_2017-06-30.csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}

```python

```