# Exploring data for the attention index

The idea of the attention index is to provide a score that indicates the impact of an article, and can easily be aggregated by subject, publisher or other axis.

The index comprises of two parts:

- **promotion** how important the article was to the publisher, based on the extent to which they chose to editorially promote it
- **response** how readers reacted to the article, based on social engagements

The index will be a number between 0 and 100. 50% is driven by the promotion, and 50% by response:

### Promotion Score

The promotion score should take into account:

- whether the publisher chose make the article a lead article on their primary front (30%)
- how long the publisher chose to retain the article on their front (40%)
- whether they chose to push the article on their facebook brand page (30%)

It should be scaled based on the value of that promotion, so a popular, well-visited site should score higher than one on the fringes. And similarly a powerful, well-followed brand page should score higher than one less followed.

### Response Score

The response score takes into account the number of engagements on Facebook.

The rest of this notebook explores how those numbers could work, starting with the response score because that is easier, I think.

# Setup

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
```

```python
data = pd.read_csv("articles_2017-07-01_2017-07-31.csv", index_col="id", \
                   parse_dates=["published", "discovered"])
data.head()
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      4743bc9bda7765affb3eeda9df300d69991d6230
      https://www.economist.com/news/china/21724659-...
      Xi Jinping talks tough to Hong Kong
      2017-07-01 15:40:06.799
      2017-07-01
      16
      0.166667
      2017-07-01T15:54:04.094Z
      0
      0
      16
      The Economist
      economist_com
      0
      8569
      38.0
      True
      8152997.0
      2017-07-02T12:25:02.000Z
      1825

      38dd54cc0cd7839edf7bbf1e29506840561b44f5
      https://www.economist.com/news/books-and-arts/...
      The man who built an American icon
      2017-07-05 05:00:07.882
      2017-07-01
      1350
      1.752434
      2017-07-05T05:14:01.859Z
      74
      1154
      122
      The Economist
      economist_com
      0
      0
      NaN
      True
      8160441.0
      2017-07-05T04:56:56.000Z
      1825

      f81c09e061c00bc175935a057e5abeead3902eae
      https://www.economist.com/news/asia/21724426-d...
      Inside Pakistan’s sex-toy industry
      2017-07-02 02:25:41.231
      2017-07-01
      2369
      6.333333
      2017-07-02T02:38:03.264Z
      546
      1547
      276
      The Economist
      economist_com
      0
      6173
      38.0
      True
      8182865.0
      2017-07-17T00:06:44.000Z
      1825

      97ef16bfcfa365eec09e577810ec03b53219e0f3
      https://www.economist.com/news/middle-east-and...
      Why justice in Africa is slow and unfair
      2017-07-03 13:05:07.359
      2017-07-01
      23
      0.000000
      2017-07-03T13:18:02.394Z
      4
      8
      11
      The Economist
      economist_com
      0
      1250
      36.0
      False
      NaN
      NaN
      1825

      c517d6b94413031521153477588ab98e70cba196
      https://www.economist.com/news/asia/21724432-f...
      Mahathir Mohamad’s return shows the sorry stat...
      2017-07-04 03:25:06.278
      2017-07-01
      681
      0.000000
      2017-07-04T03:38:03.013Z
      71
      496
      114
      The Economist
      economist_com
      380
      390
      39.0
      False
      NaN
      NaN
      1825

# Response Score

The response score is a number between 0 and 50 that indicates the level of response to an article.

Perhaps in the future we may choose to include other factors, but for now we just include engagements on Facebook. The maximum score of 50 should be achieved by an article that does really well compared with others.

```python
pd.options.display.float_format = '{:.2f}'.format
data.fb_engagements.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count    144853.00
    mean       1538.12
    std       11337.43
    min           0.00
    50%          31.00
    75%         315.00
    90%        2231.80
    95%        5944.40
    99%       27246.28
    99.5%     44891.70
    99.9%    119211.68
    max     1104454.00
    Name: fb_engagements, dtype: float64

There's one article there with 1 million plus engagements, let's just double check that.

```python
data[data.fb_engagements > 1000000]
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      publisher_name
      publisher_id
      mins_as_lead
      mins_on_front
      num_articles_on_front
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank

      id

      992cb676c2187e8826221574608258a1e971820f
      http://yournewswire.com/chester-bennington-mur...
      Police: Chester Bennington Was Murdered
      2017-07-21 15:50:16.330
      2017-07-21 15:39:19
      1040070
      702.33
      2017-07-22T02:56:01.689Z
      352834
      461329
      225907
      Your News Wire
      yournewswire_com
      274
      13313
      30.00
      True
      25631.00
      2017-07-21T17:12:04.000Z
      22568

      7ed58c3df826cf139553961106f60a9abb689382
      https://www.nytimes.com/2017/07/31/us/politics...
      Trump Removes Anthony Scaramucci From Communic...
      2017-07-31 18:36:02.714
      2017-07-31 18:34:10
      1104454
      11094.58
      2017-07-31T19:14:01.740Z
      319735
      680516
      104203
      New York Times
      nytimes_com
      389
      390
      113.00
      True
      14353650.00
      2017-07-31T18:36:16.000Z
      120

Yup, that's a biggie.

```python
data.fb_engagements.mode()
```

    0    0
    dtype: int64

Going back to the enagement counts, we see the mean is 1,538, mode is zero, median is 31, 90th percentile is 2,231, 99th percentile is 27,246, 99.5th percentile is 44,891. The standard deviation is 11,377, significantly higher than the mean, so this is not a normal distribution.

We want to provide a sensible way of allocating this to the 50 buckets we have available. Let's just bucket geometrically first:

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()

plt.figure(figsize=(12,4.5))
plt.hist(data.fb_engagements, bins=50)
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()

```

Well that's not very useful. Almost everything will score less than 0 if we just do that, which isn't a useful metric.

Let's start by excluding zeros.

```python
non_zero_fb_enagagements = data.fb_engagements[data.fb_engagements > 0]

plt.figure(figsize=(12,4.5))
plt.hist(non_zero_fb_enagagements, bins=50)
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()
```

That's still a big number at the bottom, and so not a useful score.

Next, we exclude the outliers: cap at the 99.9th percentile (i.e. 119211), so that 0.1% of articles should receive the maximum score.

```python
non_zero_fb_enagagements_without_outliers = non_zero_fb_enagagements.clip_upper(119211)

plt.figure(figsize=(12,4.5))
plt.hist(non_zero_fb_enagagements_without_outliers, bins=50)
plt.axvline(mean, linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(median, label=f'Median ({median:,.0f})', color='red')
leg = plt.legend()
```

That's a bit better, but still way too clustered at the low end. Let's look at a log normal distribution.

```python
mean = data.fb_engagements.mean()
median = data.fb_engagements.median()
ninety = data.fb_engagements.quantile(.90)
ninetyfive = data.fb_engagements.quantile(.95)
ninetynine = data.fb_engagements.quantile(.99)

plt.figure(figsize=(12,4.5))
plt.hist(np.log(non_zero_fb_enagagements + median), bins=50)
plt.axvline(np.log(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axvline(np.log(median), label=f'Median ({median:,.0f})', color='green')
plt.axvline(np.log(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axvline(np.log(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axvline(np.log(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')
leg = plt.legend()

```

That's looking a bit more interesting.

After some exploration, to avoid too much emphasis on the lower end of the scale, we move the numbers to the right a bit by adding on the median.

```python
log_engagements = (non_zero_fb_enagagements
                   .clip_upper(data.fb_engagements.quantile(.999))
                   .apply(lambda x: np.log(x + median))
                  )
log_engagements.describe()
```

    count   119879.00
    mean         5.22
    std          1.80
    min          3.47
    25%          3.71
    50%          4.57
    75%          6.28
    max         11.69
    Name: fb_engagements, dtype: float64

Use standard feature scaling to bring that to a 1 to 50 range

```python
def scale_log_engagements(engagements_logged):
    return np.ceil(
        50 * (engagements_logged - log_engagements.min()) / (log_engagements.max() - log_engagements.min())
    )

def scale_engagements(engagements):
    return scale_log_engagements(np.log(engagements + median))

scaled_non_zero_engagements = scale_log_engagements(log_engagements)
scaled_non_zero_engagements.describe()
```

    count   119879.00
    mean        11.14
    std         10.96
    min          0.00
    25%          2.00
    50%          7.00
    75%         18.00
    max         50.00
    Name: fb_engagements, dtype: float64

```python
# add in the zeros, as zero
scaled_engagements = pd.concat([scaled_non_zero_engagements, data.fb_engagements[data.fb_engagements == 0]])
```

```python
proposed = pd.DataFrame({"fb_engagements": data.fb_engagements, "response_score": scaled_engagements})
proposed.response_score.plot.hist(bins=50)
```

Now look at how the shares distribute to score:

```python
plt.figure(figsize=(15,8))

shares = np.arange(1, 60000)
plt.plot(shares, scale_engagements(shares))
plt.xlabel("shares")
plt.ylabel("score")
plt.axhline(scale_engagements(mean), linestyle=':', label=f'Mean ({mean:,.0f})', color='green')
plt.axhline(scale_engagements(median), label=f'Median ({median:,.0f})', color='green')
plt.axhline(scale_engagements(ninety), linestyle='--', label=f'90% percentile ({ninety:,.0f})', color='red')
plt.axhline(scale_engagements(ninetyfive), linestyle='-.', label=f'95% percentile ({ninetyfive:,.0f})', color='red')
plt.axhline(scale_engagements(ninetynine), linestyle=':', label=f'99% percentile ({ninetynine:,.0f})', color='red')

plt.legend(frameon=True, shadow=True)

```

```python
proposed.groupby("response_score").fb_engagements.agg([np.size, np.min, np.max])
```

      size
      amin
      amax

      response_score

      0.00
      32308
      0
      1

      1.00
      16259
      2
      6

      2.00
      10562
      7
      13

      3.00
      7444
      14
      21

      4.00
      5671
      22
      30

      5.00
      5105
      31
      41

      6.00
      4410
      42
      54

      7.00
      4143
      55
      70

      8.00
      3682
      71
      88

      9.00
      3399
      89
      109

      10.00
      3261
      110
      134

      11.00
      3134
      135
      164

      12.00
      2809
      165
      199

      13.00
      2702
      200
      240

      14.00
      2518
      241
      288

      15.00
      2565
      289
      346

      16.00
      2397
      347
      413

      17.00
      2276
      414
      493

      18.00
      2138
      494
      586

      19.00
      2065
      587
      697

      20.00
      1935
      698
      827

      21.00
      1868
      828
      980

      22.00
      1724
      981
      1161

      23.00
      1674
      1162
      1374

      24.00
      1525
      1375
      1626

      25.00
      1526
      1627
      1922

      26.00
      1428
      1923
      2271

      27.00
      1402
      2272
      2683

      28.00
      1316
      2684
      3168

      29.00
      1304
      3169
      3740

      30.00
      1172
      3741
      4414

      31.00
      1116
      4415
      5208

      32.00
      993
      5210
      6145

      33.00
      991
      6146
      7250

      34.00
      930
      7253
      8551

      35.00
      735
      8552
      10000

      36.00
      540
      10093
      11888

      37.00
      643
      11895
      14026

      38.00
      481
      14039
      16535

      39.00
      491
      16560
      19483

      40.00
      388
      19506
      22963

      41.00
      366
      22997
      27106

      42.00
      304
      27153
      31958

      43.00
      226
      32000
      37674

      44.00
      188
      37684
      44397

      45.00
      151
      44422
      52352

      46.00
      137
      52366
      61475

      47.00
      94
      61775
      72683

      48.00
      88
      72877
      85746

      49.00
      67
      86089
      100825

      50.00
      202
      101470
      1104454

Looks good to me, lets save that.

```python
data["response_score"] = proposed.response_score
```

### Proposal

The maximum of 50 points is awarded when the engagements are greater than the 99.9th percentile, rolling over the last month.

i.e. where $limit$ is the 99.5th percentile of engagements calculated over the previous month, the response score for article $a$ is:

\begin{align}
basicScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   \log(\min(engagements_a,limit) + median(engagements))      & \text{if } engagements_a > 0
\end{cases} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot \frac{basicScore_a - \min(basicScore)}{\max(basicScore) - \min(basicScore)}       & \text{if } engagements_a > 0
\end{cases} \\
\\
\text{The latter equation can be expanded to:} \\
responseScore_a & =
\begin{cases}
   0 & \text{if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))}       & \text{if } engagements_a > 0
\end{cases} \\
\end{align}

# Promotion Score

The aim of the promotion score is to indicate how important the article was to the publisher, by tracking where they chose to promote it. This is a number between 0 and 50 comprised of:

- 20 points based on whether the article was promoted as the "lead" story on the publisher's home page
- 15 points based on how long the article was promoted anywhere on the publisher's home page
- 15 points based on whether the article was promoted on the publisher's main facebook brand page

The first two should be scaled by the popularity/reach of the home page, for which we use the alexa page rank as a proxy.

The last should be scaled by the popularity/reach of the brand page, for which we use the number of likes the brand page has.

### Lead story (20 points)

```python
data.mins_as_lead.describe([0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   144853.00
    mean         9.79
    std         93.27
    min          0.00
    50%          0.00
    75%          0.00
    90%          0.00
    95%          0.00
    99%        296.92
    99.5%      604.74
    99.9%     1244.00
    max       6045.00
    Name: mins_as_lead, dtype: float64

As expected, the vast majority of articles don't make it as lead. Let's explore how long typically publishers put something as lead for.

```python
lead_articles = data[data.mins_as_lead > 0]
```

```python
lead_articles.mins_as_lead.describe([0.25, 0.5, 0.75, 0.9, 0.95, 0.99, 0.995, 0.999])
```

    count   4186.00
    mean     338.94
    std      435.30
    min        4.00
    25%       90.00
    50%      189.00
    75%      434.00
    90%      824.00
    95%     1099.75
    99%     1824.90
    99.5%   2526.32
    99.9%   4459.19
    max     6045.00
    Name: mins_as_lead, dtype: float64

```python
lead_articles.mins_as_lead.plot.hist(bins=50)
```

For lead, it's a  significant thing for an article to be lead at all, so although we want to penalise articles that were lead for a very short time, mostly we want to score the maximum even if it wasn't lead for ages. So we'll give maximum points when something has been lead for an hour.

```python
lead_articles.mins_as_lead.clip_upper(60).plot.hist(bins=50)
```

We also want to scale this by the alexa page rank, such that the maximum score of 20 points is for an article that was on the front for 4 hours for the most popular site.

So lets explore the alexa nunbers.

```python
alexa_ranks = data.groupby(by="publisher_id").alexa_rank.mean().sort_values()
alexa_ranks
```

    publisher_id
    bbc_co_uk                               96
    cnn_com                                105
    nytimes_com                            120
    theguardian_com                        142
    buzzfeed_com                           147
    dailymail_co_uk                        158
    washingtonpost_com                     191
    huffingtonpost_com                     215
    foxnews_com                            285
    rt_com                                 365
    telegraph_co_uk                        370
    independent_co_uk                      386
    reuters_com                            497
    npr_org                                594
    mirror_co_uk                           706
    nbcnews_com                            826
    breitbart_com                          994
    ft_com                                1596
    economist_com                         1825
    indy100_com                           5014
    thetimes_co_uk                        6435
    newstatesman_com                     12769
    thecanary_co                         15686
    propublica_org                       16066
    yournewswire_com                     22568
    order-order_com                      32515
    anotherangryvoice_blogspot_co_uk     77827
    westmonster_com                      97775
    evolvepolitics_com                  119412
    skwawkbox_org                       152475
    libdemvoice_org                     344992
    brexitcentral_com                   469149
    Name: alexa_rank, dtype: int64

```python
alexa_ranks.plot.bar(figsize=[10,5])
```

Let's try the simple option first: just divide the number of minutes as lead by the alexa rank. What's the scale of numbers we get then.

```python
lead_proposal_1 = lead_articles.mins_as_lead.clip_upper(60) / lead_articles.alexa_rank
lead_proposal_1.plot.hist()
```

Looks like there's too much of a cluster around 0. Have we massively over penalised the publishers with a high alexa rank?

```python
lead_proposal_1.groupby(data.publisher_id).mean().plot.bar(figsize=[10,5])
```

Yes. Let's try taking the log of the alexa rank and see if that looks better.

```python
lead_proposal_2 = (lead_articles.mins_as_lead.clip_upper(60) / np.log(lead_articles.alexa_rank))
lead_proposal_2.plot.hist()
```

```python
lead_proposal_2.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      41.00
      5.33
      0.00
      5.33
      5.33
      5.33
      5.33
      5.33

      bbc_co_uk
      108.00
      12.65
      2.09
      1.10
      13.15
      13.15
      13.15
      13.15

      breitbart_com
      202.00
      8.36
      1.33
      0.72
      8.69
      8.69
      8.69
      8.69

      brexitcentral_com
      33.00
      4.44
      0.71
      0.69
      4.59
      4.59
      4.59
      4.59

      buzzfeed_com
      212.00
      11.75
      1.35
      0.80
      12.02
      12.02
      12.02
      12.02

      cnn_com
      205.00
      12.36
      1.89
      2.15
      12.89
      12.89
      12.89
      12.89

      dailymail_co_uk
      184.00
      11.23
      1.94
      0.99
      11.85
      11.85
      11.85
      11.85

      economist_com
      42.00
      7.22
      2.13
      1.20
      7.99
      7.99
      7.99
      7.99

      evolvepolitics_com
      68.00
      4.80
      0.95
      0.77
      5.13
      5.13
      5.13
      5.13

      foxnews_com
      197.00
      10.31
      1.31
      0.88
      10.61
      10.61
      10.61
      10.61

      ft_com
      105.00
      7.31
      1.91
      1.22
      8.14
      8.14
      8.14
      8.14

      huffingtonpost_com
      158.00
      10.96
      1.19
      1.86
      11.17
      11.17
      11.17
      11.17

      independent_co_uk
      158.00
      9.58
      1.68
      1.51
      10.07
      10.07
      10.07
      10.07

      indy100_com
      121.00
      5.70
      2.04
      1.06
      4.58
      7.04
      7.04
      7.04

      libdemvoice_org
      121.00
      4.67
      0.27
      2.27
      4.71
      4.71
      4.71
      4.71

      mirror_co_uk
      340.00
      8.39
      1.80
      0.76
      9.15
      9.15
      9.15
      9.15

      nbcnews_com
      102.00
      8.72
      1.05
      2.23
      8.93
      8.93
      8.93
      8.93

      newstatesman_com
      77.00
      6.19
      0.84
      0.42
      6.35
      6.35
      6.35
      6.35

      npr_org
      151.00
      8.89
      1.65
      0.63
      9.39
      9.39
      9.39
      9.39

      nytimes_com
      58.00
      12.31
      1.40
      2.09
      12.53
      12.53
      12.53
      12.53

      order-order_com
      225.00
      4.67
      1.52
      0.39
      3.75
      5.78
      5.78
      5.78

      propublica_org
      22.00
      6.20
      0.00
      6.20
      6.20
      6.20
      6.20
      6.20

      reuters_com
      102.00
      8.96
      2.00
      0.64
      9.66
      9.66
      9.66
      9.66

      rt_com
      170.00
      9.22
      2.42
      0.85
      10.17
      10.17
      10.17
      10.17

      skwawkbox_org
      136.00
      4.63
      0.97
      0.34
      5.03
      5.03
      5.03
      5.03

      telegraph_co_uk
      101.00
      9.97
      1.05
      0.85
      10.15
      10.15
      10.15
      10.15

      thecanary_co
      230.00
      4.86
      1.81
      0.93
      3.21
      6.21
      6.21
      6.21

      theguardian_com
      149.00
      11.49
      2.02
      1.01
      12.11
      12.11
      12.11
      12.11

      thetimes_co_uk
      70.00
      6.83
      0.14
      5.70
      6.84
      6.84
      6.84
      6.84

      washingtonpost_com
      82.00
      11.04
      1.78
      0.76
      11.42
      11.42
      11.42
      11.42

      westmonster_com
      70.00
      4.76
      1.25
      0.78
      5.22
      5.22
      5.22
      5.22

      yournewswire_com
      146.00
      5.75
      0.76
      0.50
      5.99
      5.99
      5.99
      5.99

```python
lead_proposal_2.groupby(data.publisher_id).min().plot.bar(figsize=[10,5])
```

That looks about right, as long as the smaller publishers were closer to zero. So let's apply feature scaling to this, to give a number between 1 and 20. (Anything not as lead will pass though as zero.)

```python
def rescale(series):
    return (series - series.min()) / (series.max() - series.min())

lead_proposal_3 = np.ceil(20 * rescale(lead_proposal_2))
```

```python
lead_proposal_2.min(), lead_proposal_2.max()
```

    (0.3351555762355905, 13.145359968846892)

```python
lead_proposal_3.plot.hist()
```

```python
lead_proposal_3.groupby(data.publisher_id).median().plot.bar(figsize=[10,5])
```

```python
data["lead_score"] = pd.concat([lead_proposal_3, data.mins_as_lead[data.mins_as_lead==0]])
```

```python
data.lead_score.value_counts().sort_index()
```

    0.00     140668
    1.00         35
    2.00         53
    3.00         60
    4.00         60
    5.00         70
    6.00         49
    7.00        226
    8.00        316
    9.00        296
    10.00       238
    11.00       163
    12.00        63
    13.00       112
    14.00       566
    15.00       230
    16.00       388
    17.00       343
    18.00       243
    19.00       331
    20.00       343
    Name: lead_score, dtype: int64

```python
data.lead_score.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          20.00
    breitbart_com                      14.00
    brexitcentral_com                   7.00
    buzzfeed_com                       19.00
    cnn_com                            20.00
    dailymail_co_uk                    18.00
    economist_com                      12.00
    evolvepolitics_com                  8.00
    foxnews_com                        17.00
    ft_com                             13.00
    huffingtonpost_com                 17.00
    independent_co_uk                  16.00
    indy100_com                        11.00
    libdemvoice_org                     7.00
    mirror_co_uk                       14.00
    nbcnews_com                        14.00
    newstatesman_com                   10.00
    npr_org                            15.00
    nytimes_com                        20.00
    order-order_com                     9.00
    propublica_org                     10.00
    reuters_com                        15.00
    rt_com                             16.00
    skwawkbox_org                       8.00
    telegraph_co_uk                    16.00
    thecanary_co                       10.00
    theguardian_com                    19.00
    thetimes_co_uk                     11.00
    washingtonpost_com                 18.00
    westmonster_com                     8.00
    yournewswire_com                    9.00
    Name: lead_score, dtype: float64

In summary then, score for article $a$ is:

$$
unscaledLeadScore_a = \frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)}\\
leadScore_a = 19 \cdot
\frac{unscaledLeadScore_a - \min(unscaledLeadScore)}
{\max(unscaledLeadScore) - \min(unscaledLeadScore)}
+ 1
$$

Since the minium value of $minsAsLead$ is 1, $\min(unscaledLeadScore)$ is pretty insignificant. So we can simplify this to:

$$
leadScore_a = 20 \cdot
\frac{unscaledLeadScore_a }
     {\max(unscaledLeadScore)}
$$

or:

$$
leadScore_a = 20 \cdot
\frac{\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} }
     {\frac{60}{\log(\max(alexaRank))}}
$$

$$
leadScore_a = \left( 20 \cdot
\frac{\min(minsAsLead_a, 60)}{\log(alexaRank_a)} \cdot
     {\frac{\log(\max(alexaRank))}{60}} \right)
$$

## Time on front score (15 points)

This is similar to time as lead, so lets try doing the same calculation, except we also want to factor in the number of slots on the front:

$$frontScore_a =
15
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right)
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right)$$

```python
(data.alexa_rank * data.num_articles_on_front).min() / 1440
```

    2.4500000000000002

```python
time_on_front_proposal_1 = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15)
```

```python
time_on_front_proposal_1.plot.hist(figsize=(15, 7), bins=15)
```

```python
time_on_front_proposal_1.value_counts().sort_index()
```

    1.00     66222
    2.00      7925
    3.00      5204
    4.00      3679
    5.00      1086
    6.00       593
    7.00       522
    8.00       456
    9.00       776
    10.00      329
    11.00      203
    12.00      381
    13.00      212
    14.00       91
    15.00       36
    dtype: int64

```python
time_on_front_proposal_1.groupby(data.publisher_id).sum()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk      41.00
    bbc_co_uk                          15610.00
    breitbart_com                       2651.00
    brexitcentral_com                     39.00
    buzzfeed_com                        9439.00
    cnn_com                            11926.00
    dailymail_co_uk                    13394.00
    economist_com                        279.00
    evolvepolitics_com                    77.00
    foxnews_com                         8143.00
    ft_com                              2977.00
    huffingtonpost_com                  7960.00
    independent_co_uk                   4520.00
    indy100_com                          567.00
    libdemvoice_org                      121.00
    mirror_co_uk                        9980.00
    nbcnews_com                         1721.00
    newstatesman_com                     587.00
    npr_org                             2387.00
    nytimes_com                         9051.00
    order-order_com                      228.00
    propublica_org                        29.00
    reuters_com                         6889.00
    rt_com                              4125.00
    skwawkbox_org                        138.00
    telegraph_co_uk                     6268.00
    thecanary_co                         253.00
    theguardian_com                    12603.00
    thetimes_co_uk                      8934.00
    washingtonpost_com                  8749.00
    westmonster_com                      312.00
    yournewswire_com                     341.00
    dtype: float64

That looks good to me.

```python
data["front_score"] = np.ceil(data.mins_on_front.clip_upper(1440) / (data.alexa_rank * data.num_articles_on_front) * (2.45) * 15).fillna(0)
```

```python
data.front_score
```

    id
    4743bc9bda7765affb3eeda9df300d69991d6230    1.00
    38dd54cc0cd7839edf7bbf1e29506840561b44f5    0.00
    f81c09e061c00bc175935a057e5abeead3902eae    1.00
    97ef16bfcfa365eec09e577810ec03b53219e0f3    1.00
    c517d6b94413031521153477588ab98e70cba196    1.00
    3a057eabb00096d22198ac651a508129e90758a9    1.00
    83ec3cfda04c182f146da1e3db7eaad46cdbd173    0.00
    9755a8d0a6f625cd4187aaf22e2610aa33666fff    0.00
    62824f83459a95c92fada56c99673745bedf651d    0.00
    c0588bed68ed8697864ef4bb6af39e79e458d592    0.00
    1869ef3d6e4577f37a102249ad2eec01ab456175    1.00
    0ceb02edb0181c887a0518acaf6c10a3cb9930a7    3.00
    d14ba5ddd9ce349c3b63094dcbf6b75b25efb344    0.00
    b895763249a479c9902126faaf10b6b88297f170    1.00
    a4118a6f4e08f55bdf48d0080aef3e8b788517d2    0.00
    81e2470b9293770835f24bfd72daf85b331bbbd2    0.00
    e09a740f38f384d876f3603884512f56e6eb15bb   10.00
    4710746ced63ca172f7b6229323c8c6a8c941852    1.00
    0816041a6b4b8d54b5d55e800ab02025ab99d0ac    0.00
    4d0218f80269aca6f60256cb81aa097a39258cc3    0.00
    4ea8b47e9821663813eaae737abf541346da62ff    0.00
    75838ca4600abcc5fdebea3190e7fecd27d556f4    0.00
    35bb44425c828bd7bee4b8440b90ea20b8ab9de8    0.00
    fc71b24733619154cfd8fc0ae4c0347152a7f508    0.00
    99f388b07f9949aed5efd0139106e47c4c4eedfa    1.00
    aebbd20e24975bf87cfc629cb679d210bc7662d5    1.00
    c37ef0e0cb3d2bc8292c232b1d6a32c9be493042    2.00
    87f0a6b25205c21bfc0d28c8c998a69432bf1fcd    1.00
    a500e0bc307c9e4bd4b202a96d0d312d2efe042c    1.00
    b586b2cd6f26d4eb06dcf03bb505ed6d9a4a7d33    1.00
                                                ...
    0933a4247c11cf612687cc7b75f94958a4c33bec    0.00
    ee9ba2610cef6b1cbd1107da586b92df81716974    0.00
    3d1a5170a1f6aaf2469d1de12a0633f761614065    0.00
    305eef2939be65247b0eae8e2a040174670e4dc7    1.00
    3b54250abbc859d6b92bb950a1bfccd1915af75d    0.00
    a00794e3fb7e828db55ed26a1a0e1fb2d18a4b66    3.00
    313c9b2b719e1c518e8cac4a53f41b852c1b0c80    1.00
    fba53a0f262c9a12a271effde9d4fe870df512b9    2.00
    74c10e1d03fed6ee633ed7bc746579974dadbe3a    0.00
    2a6de4dc8d178f70a9202220299de079de313aa2    0.00
    b6295ff249c4ad3b47161d27d5120d3cdf6a1176    1.00
    38c9ded41c1904cb5c133a7c4001805b11951945    9.00
    0bda234055dbc02994f00322b69236027460b012    1.00
    0425aa3130128eb5fafdcf2365da5e8b9d33a68f    8.00
    b78ba15a3dd11de1de5f7d1829fac6fedfcfeaa2    1.00
    fe20177bff930ce345579607a1636e2e802bd528    2.00
    987535021c60e6a5b993602ef7e013cc066dead7    1.00
    b884afebe74dd3fd1cd248bc3f4960e6041e98cd    1.00
    ec94a7d0e634f7d35426d9406b86d62d1514b2ce    0.00
    bc81b06b8be3ce77aca8ca0b67187073ca748a4d    0.00
    a8153ec30f0db8f11b15a898940f503f2d953b15    0.00
    308dad84326da97ac033a28fc5480e7aa348fd0e    4.00
    2b90d062790b2abd36bfa50741a4e0d018eda2a8    0.00
    26664930b52df6d3dba9ab58da802d7441722e5b    0.00
    0ffb48b7a83fa76ab9b5ff673f990f30ffd523cd    0.00
    7321023c48823ee5722255cb6cd69815ab2bdfca    0.00
    ca9d256b591c9d3d1667b6241624b06d36a52d53    5.00
    229ac21c8999e59d1920e59051431edf1fe21fc2    1.00
    72afa730a7f89351ed4fd0a9439fc60baca26d77    0.00
    d4912ca7a6fd3c2e520b25dd113de653203411fd    1.00
    Name: front_score, Length: 144853, dtype: float64

## Facebook brand page promotion (15 points)

One way a publisher has of promoting content is to post to their brand page. The significance of doing so is stronger when the brand page has more followers (likes).

$$ facebookPromotionProposed1_a = 15 \left( \frac {brandPageLikes_a} {\max(brandPageLikes)} \right) $$

Now lets explore the data to see if that makes sense. **tr;dr the formula above is incorrect**

```python
data.fb_brand_page_likes.max()
```

    43263975.0

```python
facebook_promotion_proposed_1 = np.ceil((15 * (data.fb_brand_page_likes / data.fb_brand_page_likes.max())).fillna(0))
```

```python
facebook_promotion_proposed_1.value_counts().sort_index().plot.bar()
```

```python
facebook_promotion_proposed_1.groupby(data.publisher_id).describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      publisher_id

      anotherangryvoice_blogspot_co_uk
      41.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      bbc_co_uk
      12254.00
      0.63
      3.00
      0.00
      0.00
      0.00
      0.00
      15.00

      breitbart_com
      2709.00
      0.78
      0.98
      0.00
      0.00
      0.00
      2.00
      2.00

      brexitcentral_com
      39.00
      0.95
      0.22
      0.00
      1.00
      1.00
      1.00
      1.00

      buzzfeed_com
      1807.00
      0.24
      0.43
      0.00
      0.00
      0.00
      0.00
      1.00

      cnn_com
      3760.00
      2.64
      4.41
      0.00
      0.00
      0.00
      10.00
      10.00

      dailymail_co_uk
      23810.00
      0.58
      1.60
      0.00
      0.00
      0.00
      0.00
      5.00

      economist_com
      493.00
      2.05
      1.40
      0.00
      0.00
      3.00
      3.00
      3.00

      evolvepolitics_com
      78.00
      0.96
      0.19
      0.00
      1.00
      1.00
      1.00
      1.00

      foxnews_com
      6188.00
      0.68
      1.90
      0.00
      0.00
      0.00
      0.00
      6.00

      ft_com
      3164.00
      0.54
      0.89
      0.00
      0.00
      0.00
      2.00
      2.00

      huffingtonpost_com
      8937.00
      0.52
      1.35
      0.00
      0.00
      0.00
      0.00
      4.00

      independent_co_uk
      6209.00
      0.65
      1.23
      0.00
      0.00
      0.00
      0.00
      3.00

      indy100_com
      574.00
      0.24
      0.43
      0.00
      0.00
      0.00
      0.00
      1.00

      libdemvoice_org
      121.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      mirror_co_uk
      10712.00
      0.24
      0.42
      0.00
      0.00
      0.00
      0.00
      1.00

      nbcnews_com
      1802.00
      2.11
      2.00
      0.00
      0.00
      4.00
      4.00
      4.00

      newstatesman_com
      596.00
      0.63
      0.48
      0.00
      0.00
      1.00
      1.00
      1.00

      npr_org
      1959.00
      1.39
      1.50
      0.00
      0.00
      0.00
      3.00
      3.00

      nytimes_com
      4784.00
      1.45
      2.27
      0.00
      0.00
      0.00
      5.00
      5.00

      order-order_com
      228.00
      0.82
      0.39
      0.00
      1.00
      1.00
      1.00
      1.00

      propublica_org
      29.00
      1.00
      0.00
      1.00
      1.00
      1.00
      1.00
      1.00

      reuters_com
      5644.00
      0.41
      0.81
      0.00
      0.00
      0.00
      0.00
      2.00

      rt_com
      2238.00
      1.01
      1.00
      0.00
      0.00
      2.00
      2.00
      2.00

      skwawkbox_org
      138.00
      0.99
      0.09
      0.00
      1.00
      1.00
      1.00
      1.00

      telegraph_co_uk
      7500.00
      0.49
      0.86
      0.00
      0.00
      0.00
      0.00
      2.00

      thecanary_co
      253.00
      0.99
      0.09
      0.00
      1.00
      1.00
      1.00
      1.00

      theguardian_com
      8437.00
      0.57
      1.18
      0.00
      0.00
      0.00
      0.00
      3.00

      thetimes_co_uk
      8959.00
      0.05
      0.22
      0.00
      0.00
      0.00
      0.00
      1.00

      washingtonpost_com
      20711.00
      0.21
      0.76
      0.00
      0.00
      0.00
      0.00
      3.00

      westmonster_com
      338.00
      0.35
      0.48
      0.00
      0.00
      0.00
      1.00
      1.00

      yournewswire_com
      341.00
      0.07
      0.25
      0.00
      0.00
      0.00
      0.00
      1.00

That's too much variation: sites like the Guardian, which have a respectable 7.5m likes, should not be scoring a 3. Lets try applying a log to it, and then standard feature scaling again.

```python
data.fb_brand_page_likes.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk     330146.00
    bbc_co_uk                          43263975.00
    breitbart_com                       3546830.00
    brexitcentral_com                      8555.00
    buzzfeed_com                        2627096.00
    cnn_com                            27968018.00
    dailymail_co_uk                    12001069.00
    economist_com                       8207297.00
    evolvepolitics_com                   111081.00
    foxnews_com                        15587601.00
    ft_com                              3606540.00
    huffingtonpost_com                  9658384.00
    independent_co_uk                   7283163.00
    indy100_com                          214607.00
    libdemvoice_org                        8576.00
    mirror_co_uk                        2791300.00
    nbcnews_com                         9115936.00
    newstatesman_com                     153868.00
    npr_org                             6106275.00
    nytimes_com                        14378510.00
    order-order_com                       43793.00
    propublica_org                       342935.00
    reuters_com                         3823671.00
    rt_com                              4536549.00
    skwawkbox_org                          4731.00
    telegraph_co_uk                     4296563.00
    thecanary_co                         151848.00
    theguardian_com                     7635203.00
    thetimes_co_uk                       675784.00
    washingtonpost_com                  5962489.00
    westmonster_com                       12211.00
    yournewswire_com                      25692.00
    Name: fb_brand_page_likes, dtype: float64

```python
np.log(2149)
```

    7.6727578966425103

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max())
```

    publisher_id
    anotherangryvoice_blogspot_co_uk   12.71
    bbc_co_uk                          17.58
    breitbart_com                      15.08
    brexitcentral_com                   9.05
    buzzfeed_com                       14.78
    cnn_com                            17.15
    dailymail_co_uk                    16.30
    economist_com                      15.92
    evolvepolitics_com                 11.62
    foxnews_com                        16.56
    ft_com                             15.10
    huffingtonpost_com                 16.08
    independent_co_uk                  15.80
    indy100_com                        12.28
    libdemvoice_org                     9.06
    mirror_co_uk                       14.84
    nbcnews_com                        16.03
    newstatesman_com                   11.94
    npr_org                            15.62
    nytimes_com                        16.48
    order-order_com                    10.69
    propublica_org                     12.75
    reuters_com                        15.16
    rt_com                             15.33
    skwawkbox_org                       8.46
    telegraph_co_uk                    15.27
    thecanary_co                       11.93
    theguardian_com                    15.85
    thetimes_co_uk                     13.42
    washingtonpost_com                 15.60
    westmonster_com                     9.41
    yournewswire_com                   10.15
    Name: fb_brand_page_likes, dtype: float64

That's more like it, but the lower numbers should be smaller.

```python
np.log(data.fb_brand_page_likes.groupby(data.publisher_id).max() / 1000)
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    5.80
    bbc_co_uk                          10.68
    breitbart_com                       8.17
    brexitcentral_com                   2.15
    buzzfeed_com                        7.87
    cnn_com                            10.24
    dailymail_co_uk                     9.39
    economist_com                       9.01
    evolvepolitics_com                  4.71
    foxnews_com                         9.65
    ft_com                              8.19
    huffingtonpost_com                  9.18
    independent_co_uk                   8.89
    indy100_com                         5.37
    libdemvoice_org                     2.15
    mirror_co_uk                        7.93
    nbcnews_com                         9.12
    newstatesman_com                    5.04
    npr_org                             8.72
    nytimes_com                         9.57
    order-order_com                     3.78
    propublica_org                      5.84
    reuters_com                         8.25
    rt_com                              8.42
    skwawkbox_org                       1.55
    telegraph_co_uk                     8.37
    thecanary_co                        5.02
    theguardian_com                     8.94
    thetimes_co_uk                      6.52
    washingtonpost_com                  8.69
    westmonster_com                     2.50
    yournewswire_com                    3.25
    Name: fb_brand_page_likes, dtype: float64

```python
scaled_fb_brand_page_likes = (data.fb_brand_page_likes / 1000)
facebook_promotion_proposed_2 = np.ceil(\
    (15 * \
     (np.log(scaled_fb_brand_page_likes) / np.log(scaled_fb_brand_page_likes.max()))\
    )\
                                       ).fillna(0)
```

```python
facebook_promotion_proposed_2.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    9.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   4.00
    buzzfeed_com                       12.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    libdemvoice_org                     4.00
    mirror_co_uk                       12.00
    nbcnews_com                        13.00
    newstatesman_com                    8.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     6.00
    propublica_org                      9.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       3.00
    telegraph_co_uk                    12.00
    thecanary_co                        8.00
    theguardian_com                    13.00
    thetimes_co_uk                     10.00
    washingtonpost_com                 13.00
    westmonster_com                     4.00
    yournewswire_com                    5.00
    Name: fb_brand_page_likes, dtype: float64

LGTM. So the equation is

$$ facebookPromotion_a = 15 \left(
\frac {\log(\frac {brandPageLikes_a}{1000})}
{\log(\frac {\max(brandPageLikes)}{1000}))} \right) $$

Now, let's try applying standard feature scaling approch to this, rather than using a magic number of 1,000. That equation would be:

\begin{align}
unscaledFacebookPromotion_a &=
   \log(brandPageLikes_a) \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \min(unscaledFacebookPromotion)}{\max(unscaledFacebookPromotion) - \min(unscaledFacebookPromotion)}  \\
\\
\text{The scaling can be simplified to:} \\
facebookPromotion_a &=
   15 \cdot \frac{unscaledFacebookPromotion_a - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \\
\\
\text{Meaning the overall equation becomes:} \\
facebookPromotion_a &=
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}
   \end{align}

```python
facebook_promotion_proposed_3 = np.ceil(
    (14 *
     (
         (np.log(data.fb_brand_page_likes) - np.log(data.fb_brand_page_likes.min()) ) /
         (np.log(data.fb_brand_page_likes.max()) - np.log(data.fb_brand_page_likes.min()))
     )
    ) + 1
                                       )
```

```python
facebook_promotion_proposed_3.groupby(data.publisher_id).max()
```

    publisher_id
    anotherangryvoice_blogspot_co_uk    8.00
    bbc_co_uk                          15.00
    breitbart_com                      12.00
    brexitcentral_com                   3.00
    buzzfeed_com                       11.00
    cnn_com                            15.00
    dailymail_co_uk                    14.00
    economist_com                      13.00
    evolvepolitics_com                  7.00
    foxnews_com                        14.00
    ft_com                             12.00
    huffingtonpost_com                 13.00
    independent_co_uk                  13.00
    indy100_com                         8.00
    libdemvoice_org                     3.00
    mirror_co_uk                       11.00
    nbcnews_com                        13.00
    newstatesman_com                    7.00
    npr_org                            13.00
    nytimes_com                        14.00
    order-order_com                     5.00
    propublica_org                      8.00
    reuters_com                        12.00
    rt_com                             12.00
    skwawkbox_org                       2.00
    telegraph_co_uk                    12.00
    thecanary_co                        7.00
    theguardian_com                    13.00
    thetimes_co_uk                      9.00
    washingtonpost_com                 13.00
    westmonster_com                     3.00
    yournewswire_com                    4.00
    Name: fb_brand_page_likes, dtype: float64

```python
data["facebook_promotion_score"] = facebook_promotion_proposed_3.fillna(0.0)
```

# Review

```python
data["promotion_score"] = (data.lead_score + data.front_score + data.facebook_promotion_score)
data["attention_index"] = (data.promotion_score + data.response_score)
```

```python
data.promotion_score.plot.hist(bins=np.arange(50), figsize=(15,6))
```

```python
data.attention_index.plot.hist(bins=np.arange(100), figsize=(15,6))
```

```python
data.attention_index.value_counts().sort_index()
```

    0.00     20808
    1.00     18466
    2.00     12736
    3.00      8510
    4.00      6250
    5.00      5138
    6.00      4458
    7.00      3852
    8.00      3509
    9.00      3053
    10.00     2802
    11.00     2645
    12.00     2491
    13.00     2295
    14.00     2183
    15.00     2029
    16.00     1931
    17.00     1794
    18.00     1796
    19.00     1664
    20.00     1526
    21.00     1486
    22.00     1375
    23.00     1395
    24.00     1321
    25.00     1307
    26.00     1292
    27.00     1299
    28.00     1303
    29.00     1269
             ...
    66.00      107
    67.00       88
    68.00       81
    69.00       81
    70.00       65
    71.00       52
    72.00       49
    73.00       44
    74.00       51
    75.00       30
    76.00       41
    77.00       33
    78.00       25
    79.00       15
    80.00       27
    81.00       22
    82.00       14
    83.00       17
    84.00       14
    85.00       15
    86.00       10
    87.00       11
    88.00       18
    89.00        8
    90.00        3
    91.00        8
    92.00        2
    93.00        6
    94.00        2
    97.00        1
    Name: attention_index, Length: 96, dtype: int64

```python
# and lets see the articles with the biggest attention index
data.sort_values("attention_index", ascending=False)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index

      id

      3436e0d8aaf780f6ad32a6d29314b61739e0dfa8
      http://www.bbc.co.uk/news/world-us-canada-4067...
      Linkin Park's Chester Bennington dies
      2017-07-20 18:50:01.384
      2017-07-20 18:47:30.000
      420598
      5551.75
      2017-07-20T23:04:02.021Z
      57540
      303154
      59904
      ...
      True
      43050925.00
      2017-07-20T18:52:47.000Z
      96
      50.00
      20.00
      12.00
      15.00
      47.00
      97.00

      559aad871b7d6d89864e8e0225886398b2a4e560
      http://www.cnn.com/2017/07/03/politics/kris-ko...
      41 states have refused request for voter infor...
      2017-07-04 02:39:18.995
      2017-07-04 02:37:18.000
      213697
      287.57
      2017-07-04T16:42:03.462Z
      31638
      162489
      19570
      ...
      True
      27774724.00
      2017-07-04T05:00:07.000Z
      105
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      6f345d3581d036532d8629dadce4252d0c2506c2
      http://www.bbc.co.uk/news/world-us-canada-4078...
      Scaramucci out as Trump media chief
      2017-07-31 18:45:00.750
      2017-07-31 18:41:30.000
      161023
      1629.83
      2017-07-31T19:10:01.786Z
      42313
      101496
      17214
      ...
      True
      43254466.00
      2017-07-31T18:44:42.000Z
      96
      50.00
      20.00
      9.00
      15.00
      44.00
      94.00

      f7db1b29e8b216743f25bbd615b50ae408c2f885
      http://www.cnn.com/2017/07/27/politics/health-...
      Health care bill: Senate vote-a-rama and 'skin...
      2017-07-27 10:12:13.636
      2017-07-27 10:06:14.000
      88400
      162.67
      2017-07-28T12:50:02.486Z
      15521
      60071
      12808
      ...
      True
      27923546.00
      2017-07-28T05:23:05.000Z
      105
      49.00
      20.00
      9.00
      15.00
      44.00
      93.00

      58052ff3b041ff4b1966e97a996c530328536211
      http://www.cnn.com/2017/07/28/politics/john-mc...
      John McCain's maverick moment
      2017-07-28 10:18:18.108
      2017-07-28 10:11:51.000
      99793
      197.94
      2017-07-28T12:34:02.686Z
      17193
      74818
      7782
      ...
      True
      27924574.00
      2017-07-28T11:00:03.000Z
      105
      49.00
      20.00
      9.00
      15.00
      44.00
      93.00

      702215b8e856ea6a876a00d7bc283c461c0c5695
      http://www.cnn.com/2017/07/05/us/hobby-lobby-a...
      Hobby Lobby to pay $3 million fine, forfeit an...
      2017-07-06 00:15:20.139
      2017-07-06 00:07:59.000
      127071
      416.25
      2017-07-06T17:06:01.592Z
      38154
      71121
      17796
      ...
      True
      27792201.00
      2017-07-06T00:31:15.000Z
      105
      50.00
      20.00
      8.00
      15.00
      43.00
      93.00

      1de20bf9754fd551e89eb27281f0b1e5c246c2dd
      https://www.buzzfeed.com/jimderogatis/parents-...
      R. Kelly Is Holding Women Against Their Will I...
      2017-07-17 11:03:51.843
      2017-07-16 18:31:59.000
      225186
      342.37
      2017-07-17T16:20:04.064Z
      98751
      92997
      33438
      ...
      True
      2611073.00
      2017-07-17T16:02:28.000Z
      147
      50.00
      19.00
      13.00
      11.00
      43.00
      93.00

      ae6a1b99ec05a3f63b61422b194ca6a493059f80
      https://www.buzzfeed.com/juliareinstein/nelsan...
      "True Blood" Star Nelsan Ellis Has Died At 39
      2017-07-08 20:15:15.199
      2017-07-08 19:52:33.000
      177086
      843.83
      2017-07-09T05:06:02.794Z
      43015
      120507
      13564
      ...
      True
      2599721.00
      2017-07-08T21:20:00.000Z
      147
      50.00
      19.00
      13.00
      11.00
      43.00
      93.00

      2798d0a22dd362d0b432b4b80968ff1fd32fdfa4
      http://www.cnn.com/2017/07/04/politics/kfile-r...
      How CNN found the Reddit user behind the Trump...
      2017-07-05 01:24:16.090
      2017-07-05 01:19:56.000
      95668
      586.33
      2017-07-05T03:14:02.777Z
      50763
      34701
      10204
      ...
      True
      27782621.00
      2017-07-05T01:33:56.000Z
      105
      49.00
      20.00
      9.00
      15.00
      44.00
      93.00

      7a3d174f2e0eb21f0f6979a35ed627f3b357b800
      http://www.bbc.co.uk/news/world-us-canada-4072...
      Transgender people 'can't serve' US army
      2017-07-26 13:18:02.250
      2017-07-26 13:15:47.000
      161143
      641.22
      2017-07-26T13:44:02.131Z
      45467
      103222
      12454
      ...
      True
      43157715.00
      2017-07-26T13:21:15.000Z
      96
      50.00
      20.00
      7.00
      15.00
      42.00
      92.00

      e9366c1c524a11db92a0482ba964a9ab6c9d8934
      http://www.cnn.com/2017/07/23/us/san-antonio-b...
      8 people found dead in tractor-trailer in San ...
      2017-07-23 08:18:15.575
      2017-07-23 08:15:11.000
      78232
      106.56
      2017-07-23T09:08:01.294Z
      19336
      46306
      12590
      ...
      True
      27899082.00
      2017-07-23T08:20:14.000Z
      105
      48.00
      20.00
      9.00
      15.00
      44.00
      92.00

      d0006e12eda0b000500540ed740f683c9837425a
      http://www.cnn.com/2017/07/08/us/chicago-high-...
      Chicago high school graduation requirement: No...
      2017-07-08 17:00:15.475
      2017-07-08 16:53:19.000
      60594
      325.70
      2017-07-08T19:16:01.985Z
      24225
      30050
      6319
      ...
      True
      27808844.00
      2017-07-08T17:28:31.000Z
      105
      46.00
      20.00
      10.00
      15.00
      45.00
      91.00

      a51df1035653a4908d18a0da005f43f3caaf6bc5
      http://www.cnn.com/2017/07/01/politics/donald-...
      Defiant Trump resumes attacks on Morning Joe h...
      2017-07-01 13:57:15.929
      2017-07-01 13:53:09.000
      70867
      185.92
      2017-07-01T15:12:02.778Z
      33403
      32417
      5047
      ...
      True
      27746580.00
      2017-07-01T14:30:01.000Z
      105
      47.00
      20.00
      9.00
      15.00
      44.00
      91.00

      094abb5cfae2aeb7cef9f1de1b1c52c0b59be475
      https://www.buzzfeed.com/briannasacks/this-bab...
      This Baby Might Have The World's First Genderl...
      2017-07-04 19:09:12.656
      2017-07-04 18:41:52.000
      80486
      162.20
      2017-07-05T19:08:05.120Z
      32688
      43621
      4177
      ...
      True
      2595118.00
      2017-07-04T21:22:00.000Z
      147
      48.00
      19.00
      13.00
      11.00
      43.00
      91.00

      b32d5bcada48909ce936d5195b79ee2c06edc56c
      http://www.bbc.co.uk/news/uk-40692457
      Princes tell documentary of 'fun' childhood wi...
      2017-07-22 23:18:00.985
      2017-07-22 23:15:14.000
      44893
      87.20
      2017-07-23T11:06:01.520Z
      2033
      41537
      1323
      ...
      True
      43095582.00
      2017-07-23T10:09:53.000Z
      96
      45.00
      20.00
      11.00
      15.00
      46.00
      91.00

      5358e6947fbdf5034b5298abfb9e35c2d98f5c1d
      http://www.cnn.com/2017/07/29/asia/north-korea...
      Test shows North Korea missile could hit LA, C...
      2017-07-29 04:36:18.702
      2017-07-29 04:32:39.000
      69501
      133.67
      2017-07-29T15:20:04.815Z
      18842
      40320
      10339
      ...
      True
      27929296.00
      2017-07-29T15:00:17.000Z
      105
      47.00
      20.00
      9.00
      15.00
      44.00
      91.00

      51344d104573f6cb0b56195f02f1e7e59b769b96
      https://www.buzzfeed.com/davidmack/back-and-to...
      A Guide To The The Incredible Video Of John Mc...
      2017-07-28 14:57:14.962
      2017-07-28 14:43:08.000
      74884
      147.54
      2017-07-28T16:36:04.450Z
      11170
      56075
      7639
      ...
      True
      2618912.00
      2017-07-28T15:35:01.000Z
      147
      48.00
      19.00
      13.00
      11.00
      43.00
      91.00

      d0a77cd54e1eb00fe4a65612e51aaea33246ae54
      http://www.cnn.com/2017/07/25/politics/senate-...
      Health care vote: Senate drama on Obamacare re...
      2017-07-25 16:18:23.590
      2017-07-25 16:12:17.000
      63501
      118.25
      2017-07-26T12:20:02.166Z
      9648
      47897
      5956
      ...
      True
      27912279.00
      2017-07-26T07:30:04.000Z
      105
      47.00
      20.00
      9.00
      15.00
      44.00
      91.00

      20b91792fb1df7881576696a4b08c0c45c50af1b
      https://www.buzzfeed.com/caseyrackham/wonder-w...
      "Wonder Woman 2" Is Officially Happening
      2017-07-22 19:35:06.157
      2017-07-18 06:01:23.000
      113870
      137.41
      2017-07-23T01:50:02.459Z
      12936
      96615
      4319
      ...
      True
      2615881.00
      2017-07-22T19:54:02.000Z
      147
      50.00
      19.00
      11.00
      11.00
      41.00
      91.00

      5d7f8181e110224c29217192de97d3862491d706
      https://www.buzzfeed.com/franciswhittaker/trum...
      President Trump Tweeted A Video Of Himself Bea...
      2017-07-02 13:45:16.185
      2017-07-02 13:45:06.000
      45451
      135.10
      2017-07-02T17:00:02.182Z
      19467
      21209
      4775
      ...
      True
      2592251.00
      2017-07-02T14:40:44.000Z
      147
      45.00
      19.00
      15.00
      11.00
      45.00
      90.00

      b36e45f1b21b12b39e1030e5ed39c1e6b852144c
      https://www.buzzfeed.com/erinlarosa/the-cast-o...
      Here's What The Live-Action Cast Of "Aladdin" ...
      2017-07-15 19:25:06.172
      2017-07-15 18:45:37.000
      73752
      1068.33
      2017-07-15T21:14:03.477Z
      17100
      49241
      7411
      ...
      True
      2609349.00
      2017-07-15T19:20:23.000Z
      147
      48.00
      19.00
      12.00
      11.00
      42.00
      90.00

      79c36aadcafb1765d4f95b03c93ce15e0868d882
      http://www.cnn.com/2017/07/08/us/kkk-rally-cha...
      KKK rally draws more counterprosters than Klan...
      2017-07-09 01:00:16.377
      2017-07-09 00:56:20.000
      53024
      53.58
      2017-07-10T14:20:02.417Z
      7691
      40052
      5281
      ...
      True
      27821815.00
      2017-07-10T11:04:02.000Z
      105
      46.00
      20.00
      9.00
      15.00
      44.00
      90.00

      ce542647791f21679348712c7b8f1de943a5579c
      http://www.cnn.com/2017/07/31/health/climate-c...
      Earth to warm 2 degrees by the end of this cen...
      2017-07-31 15:06:16.525
      2017-07-31 15:01:10.000
      48113
      122.17
      2017-07-31T17:09:59.951Z
      12464
      26512
      9137
      ...
      True
      27939022.00
      2017-07-31T16:23:57.000Z
      105
      45.00
      20.00
      9.00
      15.00
      44.00
      89.00

      8d817a571c99e0b4459570eb4b741d30754e838e
      https://www.nytimes.com/2017/07/25/us/politics...
      Pence Breaks Tie as Senate Votes to Begin Deba...
      2017-07-25 19:12:07.038
      2017-07-25 19:08:51.000
      106789
      2359.55
      2017-07-25T21:14:03.052Z
      21208
      73622
      11959
      ...
      True
      14319851.00
      2017-07-25T19:11:13.000Z
      120
      50.00
      20.00
      5.00
      14.00
      39.00
      89.00

      4054d42e80eeeccc75746953f8349cc7f43e258e
      https://www.buzzfeed.com/michaelblackmon/beyon...
      Beyoncé Finally Unveiled Her Twins To The Worl...
      2017-07-14 05:30:07.947
      2017-07-14 05:22:45.000
      83975
      705.19
      2017-07-14T07:08:02.426Z
      11181
      69167
      3627
      ...
      True
      2607990.00
      2017-07-14T05:54:45.000Z
      147
      48.00
      19.00
      11.00
      11.00
      41.00
      89.00

      f6661df6489f45511f21ff109b35a4ca12506bb4
      http://www.cnn.com/2017/07/03/politics/chris-c...
      Chris Christie's sunbathing pics clinch it: He...
      2017-07-03 10:18:12.207
      2017-07-03 10:15:53.000
      91449
      199.75
      2017-07-03T17:12:02.051Z
      24359
      59256
      7834
      ...
      True
      27766821.00
      2017-07-03T11:48:57.000Z
      105
      49.00
      20.00
      5.00
      15.00
      40.00
      89.00

      a773d35bbc29c965338c3299af4ed5ca46e50404
      https://www.buzzfeed.com/albertonardelli/ivank...
      Ivanka Trump Sat In For Her Dad At A G20 Leade...
      2017-07-08 10:33:15.951
      2017-07-08 10:30:55.000
      61337
      196.92
      2017-07-08T13:38:02.443Z
      16590
      41419
      3328
      ...
      True
      2599404.00
      2017-07-08T12:36:23.000Z
      147
      46.00
      19.00
      13.00
      11.00
      43.00
      89.00

      fb3cddfa4de1fd0bffefd5e12c8a2980c8ebf282
      http://www.cnn.com/2017/07/21/politics/sean-sp...
      Sean Spicer resigns
      2017-07-21 16:10:08.997
      2017-07-21 16:07:56.000
      135574
      3829.57
      2017-07-21T17:12:03.386Z
      30849
      89420
      15305
      ...
      True
      27891334.00
      2017-07-21T16:10:08.000Z
      105
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      ca1da832c21a98df4bfad86cd61d5e9809d813dd
      http://www.cnn.com/2017/07/26/politics/trump-m...
      Trump announces US military ban on transgender...
      2017-07-26 13:27:18.100
      2017-07-26 13:22:28.000
      570132
      1487.33
      2017-07-26T15:16:03.316Z
      209303
      317636
      43193
      ...
      True
      27913284.00
      2017-07-26T13:26:09.000Z
      105
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      259c023b40767cc7163355586c7798d5c6e61363
      http://www.cnn.com/2017/07/19/health/gupta-mcc...
      Sen. John McCain had aggressive brain tumor su...
      2017-07-20 00:18:25.389
      2017-07-20 00:13:06.000
      113686
      3604.92
      2017-07-20T03:08:03.600Z
      24482
      79207
      9997
      ...
      True
      27881281.00
      2017-07-20T00:33:04.000Z
      105
      50.00
      20.00
      4.00
      15.00
      39.00
      89.00

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      6ddc5e752591eff7e7f2693fc6e880c1c7525e78
      http://www.huffingtonpost.com/entry/transformi...
      Transforming Anxiety into Personal Purpose
      2017-07-27 18:48:18.872
      2017-07-27 18:39:09.816
      1
      0.02
      2017-07-28T01:14:04.511Z
      0
      0
      1
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6139af2a6b4e17f8359e9e4fd8b8884a22365cb9
      http://www.telegraph.co.uk/world-paraathletics...
      London 2017 World Para athletics championships...
      2017-07-23 16:30:00.406
      2017-07-23 16:25:53.000
      1
      0.02
      2017-07-24T06:10:01.599Z
      0
      0
      1
      ...
      False
      nan
      NaN
      370
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      cce7c5592865dde3cfe7e6230c2661d1fa253c30
      https://www.washingtonpost.com/business/german...
      Germany checking Daimler cars amid diesel emis...
      2017-07-14 11:00:23.419
      2017-07-14 10:51:23.000
      1
      0.02
      2017-07-14T12:16:03.797Z
      0
      0
      1
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      926ef5a04b487daccee5f83d76bae6c03be7b8fc
      https://www.washingtonpost.com/national/appare...
      Apparent mocking email surfaces ahead of bank ...
      2017-07-27 18:48:09.672
      2017-07-27 18:38:25.000
      0
      0.00
      2017-07-28T18:48:05.991Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      43798302e687d5c3c385464472da7b1853057eba
      https://www.washingtonpost.com/business/pakist...
      Pakistani oil and gas regulator fines Shell ov...
      2017-07-07 10:57:12.209
      2017-07-07 10:52:13.000
      1
      0.02
      2017-07-07T13:14:02.274Z
      0
      0
      1
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      09d7423c7b77fd423a7ec2301e934271d36387aa
      http://www.bbc.co.uk/news/uk-england-birmingha...
      Dudley man jailed for 14 years for shovel attack
      2017-07-07 10:57:03.789
      2017-07-07 10:53:04.000
      1
      0.02
      2017-07-07T18:24:02.351Z
      0
      0
      1
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      524c0cee988fea59a7e5862ff72efadd00cc3fbf
      http://www.huffingtonpost.com/entry/defending-...
      Defending Fidel and His Revolutions
      2017-07-27 18:48:16.777
      2017-07-27 18:43:40.144
      0
      0.00
      2017-07-28T18:48:05.999Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0a0487c4e27b57687065183c680cbb649f43b9d1
      http://www.bbc.co.uk/news/av/world-asia-406993...
      Taoists march through Taipei to oppose incense...
      2017-07-23 16:18:04.151
      2017-07-23 16:16:52.000
      0
      0.00
      2017-07-24T04:56:02.352Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      f185ddcfdacc71e92c1f8bc590dea8a0e6939428
      http://www.huffingtonpost.com/entry/turning-op...
      Turning Opacity into Transparency
      2017-07-27 18:48:16.712
      2017-07-27 18:44:08.582
      0
      0.00
      2017-07-28T18:48:05.995Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2b55fd0384938872a8ab665dc43742360cbd974a
      http://www.dailymail.co.uk/tvshowbiz/article-4...
      Roxy Jacenko flaunts $50,000 ring before cockt...
      2017-07-14 11:03:47.010
      2017-07-14 11:00:58.000
      0
      0.00
      2017-07-15T04:50:03.810Z
      0
      0
      0
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2da4d50e160b33d2c545ed26698dbe45b96dd6fa
      https://www.theguardian.com/sport/2017/jul/23/...
      Karl Burke’s Unfortunately has luck on his sid...
      2017-07-23 16:09:00.925
      2017-07-23 16:07:23.000
      0
      0.00
      2017-07-24T05:48:02.030Z
      0
      0
      0
      ...
      False
      nan
      NaN
      142
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      2a20b8ec4bb04fb64c7cdcf697cf9365646daace
      http://www.washingtonpost.com/video/sports/bri...
      Britt McHenry: From ESPN reporter to conservat...
      2017-07-27 19:24:07.067
      2017-07-27 18:51:18.000
      0
      0.00
      2017-07-28T16:18:04.459Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      3727bbe0ee7256f1d1f5623dbb291e0983f404b3
      http://www.huffingtonpost.com/entry/the-global...
      The Global Search for Education:  Dear Parents...
      2017-07-23 16:18:20.108
      2017-07-23 16:07:31.943
      0
      0.00
      2017-07-24T04:56:02.359Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ef61ebc4669eb210a4a1fd6a568a5ffe3b2b69b2
      http://www.huffingtonpost.com/entry/how-can-i-...
      How Can I Be a Fashionable Consumer Activist?
      2017-07-27 19:03:25.163
      2017-07-27 18:51:05.190
      1
      0.02
      2017-07-28T11:48:04.934Z
      0
      0
      1
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      ba5c76761719334cb22402c1d0fcf2cdd6eec00e
      http://www.dailymail.co.uk/news/article-467444...
      Fraudster accountant found GUILTY of scamming ...
      2017-07-07 10:33:20.731
      2017-07-07 10:32:53.000
      1
      0.02
      2017-07-07T18:00:02.942Z
      0
      0
      1
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      28a138b0a5a32c4ae4f06a5f573f5c86934d4227
      http://www.huffingtonpost.com/entry/tired-of-y...
      Tired of Your Glass Being Half Empty?
      2017-07-23 16:18:18.612
      2017-07-23 16:08:38.942
      0
      0.00
      2017-07-24T04:56:02.358Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      125ef5f3b8d17f94cab610eabaa5883a63bfa0b0
      https://www.washingtonpost.com/sports/colleges...
      4 conferences sign on to basketball officiatin...
      2017-07-27 18:57:14.652
      2017-07-27 18:50:17.000
      0
      0.00
      2017-07-28T15:50:04.225Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c20134d8230305270672c866670341d8ac518c16
      https://www.washingtonpost.com/sports/redskins...
      Markov leaves Canadiens to sign in KHL, may pl...
      2017-07-27 18:57:13.416
      2017-07-27 18:50:09.000
      0
      0.00
      2017-07-28T15:50:03.708Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      c46baef79c571904da68fd656a1f25902494e5a5
      https://www.washingtonpost.com/national/sherif...
      Sheriff’s office: 47-year-old inmate dies at C...
      2017-07-23 16:18:09.793
      2017-07-23 16:09:15.000
      0
      0.00
      2017-07-24T04:56:02.353Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      9fbbfe4fbe0f7e749e12fc58ca2b3cc604fa3d89
      http://www.bbc.co.uk/news/av/entertainment-art...
      BBC Proms: Jess Gillam plays Escapades
      2017-07-14 11:06:03.646
      2017-07-14 11:04:41.000
      0
      0.00
      2017-07-15T03:52:03.831Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      6bf665abae2ac915de3cd17e5966ace95ee1c4e2
      https://www.washingtonpost.com/world/the_ameri...
      Attorney General Jeff Sessions tells AP he wil...
      2017-07-27 19:00:10.854
      2017-07-27 18:49:21.000
      0
      0.00
      2017-07-28T15:54:04.436Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      fa173051c2b870516c71938b853d7cf38d6f2b35
      http://www.huffingtonpost.com/entry/district-l...
      District Leadership Lending a Helping Hand wit...
      2017-07-27 19:03:20.362
      2017-07-27 18:48:21.958
      0
      0.00
      2017-07-28T15:56:04.983Z
      0
      0
      0
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      7cdf9f9f724a8ae379cfd654756bbc5fb9b9bee6
      https://www.washingtonpost.com/business/vw-and...
      VW and regulators agree on fix for cars in che...
      2017-07-27 19:00:12.076
      2017-07-27 18:48:21.000
      0
      0.00
      2017-07-28T15:54:04.439Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      8728784bc322e1cc1fe2348c7f12fae2cf342b2c
      https://www.washingtonpost.com/sports/national...
      Rangers’ Adrian Beltre on the brink of 3,000th...
      2017-07-27 18:57:16.782
      2017-07-27 18:47:41.000
      0
      0.00
      2017-07-28T15:50:04.227Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      5de16111d2e5278ee91c9cc3b7cb7bd8d5a9d33c
      https://www.washingtonpost.com/business/europe...
      European Commission clears DuPont deals linked...
      2017-07-27 18:51:09.278
      2017-07-27 18:47:18.000
      0
      0.00
      2017-07-28T18:50:05.296Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      3c650009a2e2d71baaa078e1b233cc21077c1566
      http://www.bbc.co.uk/news/live/uk-england-leed...
      Yorkshire breaking news: Latest updates
      2017-07-07 10:39:04.327
      2017-07-07 10:35:10.000
      0
      0.00
      2017-07-08T00:18:02.259Z
      0
      0
      0
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      7970c4168ee7b88483317ccf03c036b5a502b5fe
      http://www.huffingtonpost.com/entry/curing-the...
      Curing the Disease of Self-Doubt
      2017-07-27 19:03:26.707
      2017-07-27 18:46:29.792
      1
      0.00
      2017-07-28T15:56:04.998Z
      0
      0
      1
      ...
      False
      nan
      NaN
      215
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      0471e67199b0dfc0452df00bb3ead5e8501762ab
      http://www.bbc.co.uk/sport/football/40531232
      Alex Gilliead: Bradford City re-sign Newcastle...
      2017-07-07 10:42:00.532
      2017-07-07 10:38:23.000
      1
      0.02
      2017-07-07T11:58:01.354Z
      0
      0
      1
      ...
      False
      nan
      NaN
      96
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      4fc062228cdb2b8550278042f8a35315d65d76af
      http://www.dailymail.co.uk/sport/football/arti...
      Wayne Rooney can inspire Everton, claims David...
      2017-07-14 11:06:22.435
      2017-07-14 11:03:12.000
      1
      0.08
      2017-07-14T11:20:05.731Z
      0
      0
      1
      ...
      False
      nan
      NaN
      158
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      72506d48f8176aa0b341b4770ea359719c8acbbd
      https://www.washingtonpost.com/opinions/no-tha...
      No, thanks, Mr. Stewart. Americans don’t need ...
      2017-07-19 22:33:06.575
      2017-07-19 22:21:50.000
      0
      0.00
      2017-07-20T21:30:03.112Z
      0
      0
      0
      ...
      False
      nan
      NaN
      191
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

144853 rows × 25 columns

```python
data["score_diff"] = data.promotion_score - data.response_score
```

```python
# promoted but low response
data.sort_values("score_diff", ascending=False).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      451e6f7b42d75a6fc2a7879ea9b7bcf55d598a8d
      https://www.buzzfeed.com/rubycramer/bernie-wor...
      Bernie World Waits For A 2020 Signal
      2017-07-26 22:51:19.070
      2017-07-25 07:53:28
      2
      0.17
      2017-07-26T23:04:02.990Z
      0
      0
      2
      ...
      2618395.00
      2017-07-27T17:44:00.000Z
      147
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      871c1545b86c7762881e3ef1fed937eafb8207a9
      https://www.buzzfeed.com/zahrahirji/lamar-smit...
      This Congressman Doesn’t Think Climate Change ...
      2017-07-14 09:15:32.780
      2017-07-13 01:11:52
      5
      0.17
      2017-07-14T09:28:05.055Z
      0
      1
      4
      ...
      2608106.00
      2017-07-14T15:44:00.000Z
      147
      1.00
      19.00
      12.00
      11.00
      42.00
      43.00
      41.00

      10b5f1191508061ce79faa2147664c96580073cf
      https://www.buzzfeed.com/ryanmac/donald-trump-...
      Donald Trump Has Your Full Attention. Can Anyo...
      2017-07-23 18:33:19.138
      2017-07-18 18:51:41
      2
      0.17
      2017-07-23T18:46:02.217Z
      0
      0
      2
      ...
      2616295.00
      2017-07-23T22:40:01.000Z
      147
      1.00
      19.00
      11.00
      11.00
      41.00
      42.00
      40.00

      0b8ef07807604d3edf607a7f078ae88f63b2d355
      https://www.buzzfeed.com/katiehasty/climate-do...
      Climate Documentaries Are Christian Movies, Too
      2017-07-19 16:35:08.344
      2017-07-15 02:21:54
      4
      0.08
      2017-07-19T16:48:05.539Z
      0
      0
      4
      ...
      2615834.00
      2017-07-22T17:40:00.000Z
      147
      1.00
      19.00
      10.00
      11.00
      40.00
      41.00
      39.00

      545b6b2c745a39e68d09a06fa423230f3e4815b5
      https://www.buzzfeed.com/buzzfeednews/2017-g20...
      Trump Travels To Poland Ahead Of The G-20 Summit
      2017-07-05 12:30:18.484
      2017-07-05 12:23:31
      31
      0.10
      2017-07-05T15:50:01.721Z
      2
      7
      22
      ...
      2597376.00
      2017-07-07T08:45:08.000Z
      147
      5.00
      19.00
      14.00
      11.00
      44.00
      49.00
      39.00

      ee2a235d394177f521ecfccc7846263893566dd0
      https://www.buzzfeed.com/tariniparti/newt-ging...
      Newt Gingrich Is Living His Best Life — Thanks...
      2017-07-06 20:33:17.293
      2017-07-06 19:27:49
      52
      0.13
      2017-07-07T14:20:01.885Z
      5
      34
      13
      ...
      2600834.00
      2017-07-09T20:30:00.000Z
      147
      6.00
      19.00
      14.00
      11.00
      44.00
      50.00
      38.00

      e2c2e154812923920015a9f66f0aa6d11edadd17
      https://www.buzzfeed.com/zahrahirji/iceberg-br...
      An Iceberg The Size Of Delaware Just Broke Off...
      2017-07-12 08:55:04.055
      2017-07-01 18:06:00
      2
      0.17
      2017-07-12T09:08:04.509Z
      0
      0
      2
      ...
      2604526.00
      2017-07-12T12:28:00.000Z
      147
      1.00
      19.00
      8.00
      11.00
      38.00
      39.00
      37.00

      cc5556ad7c545d2a931620bf9adba4728bdf1b82
      https://www.buzzfeed.com/nidhiprakash/agencies...
      Agencies Say The Travel Ban's Cut-Off Date For...
      2017-07-05 23:54:15.774
      2017-07-05 23:51:43
      71
      0.45
      2017-07-06T01:10:01.587Z
      8
      35
      28
      ...
      2595962.00
      2017-07-06T00:15:00.000Z
      147
      8.00
      19.00
      14.00
      11.00
      44.00
      52.00
      36.00

      d0e42c4fcbb4bf8e975596b4f13813ad4378bc6a
      https://www.buzzfeed.com/sylviaobell/evolving-...
      Inside The New Chapter Of "Being Mary Jane"
      2017-07-13 19:05:11.684
      2017-07-13 18:53:41
      55
      0.15
      2017-07-14T01:30:01.705Z
      8
      39
      8
      ...
      2610563.00
      2017-07-16T21:24:00.000Z
      147
      7.00
      19.00
      13.00
      11.00
      43.00
      50.00
      36.00

      4c959676517e0137597559f6b394bcddc2c559f4
      https://www.buzzfeed.com/danvergano/get-the-le...
      We Have Been Lead Poisoning Ourselves For 2000...
      2017-07-08 14:10:05.297
      2017-07-07 20:43:38
      79
      0.35
      2017-07-08T16:28:02.117Z
      0
      54
      25
      ...
      2600374.00
      2017-07-09T02:20:00.000Z
      147
      8.00
      19.00
      13.00
      11.00
      43.00
      51.00
      35.00

      604fd3b484997d45a44441e5e134f680be4e3910
      https://www.buzzfeed.com/adriancarrasquillo/se...
      Sean Spicer’s Time As A Mini-Trump Was Always ...
      2017-07-22 12:39:23.522
      2017-07-22 02:37:44
      73
      0.37
      2017-07-22T14:56:04.495Z
      1
      52
      20
      ...
      2616263.00
      2017-07-23T18:40:03.000Z
      147
      8.00
      19.00
      13.00
      11.00
      43.00
      51.00
      35.00

      2feffa494cc242b7bf53670acf2a4bb92dc9d1e0
      https://www.buzzfeed.com/scaachikoul/who-wants...
      MTV Isn't What It Used To Be
      2017-07-17 16:20:06.146
      2017-07-14 17:07:19
      22
      0.25
      2017-07-17T16:34:03.581Z
      4
      9
      9
      ...
      2616183.00
      2017-07-23T02:05:26.000Z
      147
      4.00
      19.00
      9.00
      11.00
      39.00
      43.00
      35.00

      a3bf3d1cade0c297cc30425343e228d62576db8f
      http://www.huffingtonpost.com/entry/military-f...
      Military Families Are Looking For Something --...
      2017-07-04 12:36:15.139
      2017-07-03 12:22:01
      2
      0.08
      2017-07-04T12:50:04.119Z
      0
      1
      1
      ...
      9597197.00
      2017-07-04T13:30:24.000Z
      215
      1.00
      17.00
      5.00
      13.00
      35.00
      36.00
      34.00

      a79d96fc2a2db3612568a561c2d437a0c1f793f2
      https://www.buzzfeed.com/blakemontgomery/the-a...
      The Alt-Right Has A Payment Processor Problem
      2017-08-02 12:03:16.152
      2017-07-29 01:39:50
      1
      0.08
      2017-08-02T12:16:02.067Z
      0
      0
      1
      ...
      2623799.00
      2017-08-03T05:44:00.000Z
      147
      0.00
      15.00
      7.00
      11.00
      33.00
      33.00
      33.00

      576d7843382ecea45c910ada2d21ad0d98cfec77
      https://www.buzzfeed.com/nidhiprakash/trump-wh...
      The White House Wants The Media To Talk About ...
      2017-07-01 18:48:24.407
      2017-07-01 16:32:02
      183
      0.87
      2017-07-01T23:10:02.630Z
      14
      109
      60
      ...
      2590932.00
      2017-07-01T22:20:00.000Z
      147
      12.00
      19.00
      15.00
      11.00
      45.00
      57.00
      33.00

      aeacf7959c35ecc4cc5950e7824705928439d02f
      https://www.buzzfeed.com/danvergano/nasa-mars-...
      The NASA Mars Program Is As Sad As It’s Ever Been
      2017-07-30 13:50:04.794
      2017-07-29 19:14:02
      105
      0.60
      2017-07-30T20:16:00.889Z
      14
      62
      29
      ...
      2620707.00
      2017-07-30T18:44:00.000Z
      147
      9.00
      19.00
      12.00
      11.00
      42.00
      51.00
      33.00

      b1529e3bab920909d03cd99af36a7ce821b32316
      https://www.buzzfeed.com/christianstork/spy-pl...
      This Shadowy Company Is Flying Spy Planes Over...
      2017-08-04 11:10:05.455
      2017-07-29 15:47:23
      1
      0.08
      2017-08-04T11:24:03.630Z
      0
      0
      1
      ...
      nan
      NaN
      147
      0.00
      19.00
      14.00
      0.00
      33.00
      33.00
      33.00

      ed404ba7797b1e0b5007f0057629fbfd883a72c2
      https://www.buzzfeed.com/coralewis/left-polici...
      The Bernie Sanders Left Wants More From Democr...
      2017-08-01 20:27:14.325
      2017-07-28 13:08:50
      0
      0.00
      2017-08-01T20:40:01.996Z
      0
      0
      0
      ...
      nan
      NaN
      147
      0.00
      19.00
      13.00
      0.00
      32.00
      32.00
      32.00

      53413225c8ace2e9a141cc41fc048cf3145f4fef
      http://www.dailymail.co.uk/news/article-468489...
      Mother banished husband to sofa and found him ...
      2017-07-11 10:00:17.071
      2017-07-11 09:56:59
      3
      91.04
      2017-07-11T13:52:02.406Z
      0
      0
      3
      ...
      11730607.00
      2017-07-11T23:30:05.000Z
      158
      1.00
      18.00
      1.00
      14.00
      33.00
      34.00
      32.00

      24c4ccdd2f2573c9773f07bc89b86a92fc614198
      https://www.buzzfeed.com/meghara/isis-wants-to...
      ISIS Wants To Exploit Tensions In The Philippi...
      2017-07-09 08:24:09.504
      2017-07-07 15:53:43
      1
      0.08
      2017-07-09T08:38:02.191Z
      0
      0
      1
      ...
      nan
      NaN
      147
      0.00
      19.00
      13.00
      0.00
      32.00
      32.00
      32.00

      f9c9a844d6e7d8ac5b29cd87aa5bc2587124d8d8
      https://www.buzzfeed.com/amberjamieson/iraqs-p...
      Iraq's Prime Minister Has Declared The "Libera...
      2017-07-09 12:54:13.996
      2017-07-09 12:51:36
      157
      0.69
      2017-07-09T15:12:02.146Z
      5
      107
      45
      ...
      2600522.00
      2017-07-09T14:30:00.000Z
      147
      11.00
      19.00
      13.00
      11.00
      43.00
      54.00
      32.00

      7efd43a0d7f9b4e51c923735f83495cbe188ad6e
      https://www.buzzfeed.com/alannabennett/raunchy...
      Hollywood’s Tug-Of-War With Raucous, Unapologe...
      2017-07-13 20:15:15.177
      2017-07-12 12:03:22
      2
      0.00
      2017-07-13T20:28:04.999Z
      0
      0
      2
      ...
      nan
      NaN
      147
      1.00
      19.00
      13.00
      0.00
      32.00
      33.00
      31.00

      d886bbadf26a92af0a9573abf3177cc446cebd01
      https://www.buzzfeed.com/katherinemiller/lana-...
      Lana Del Rey Knows Exactly What To Do With Thi...
      2017-07-14 18:57:28.669
      2017-07-08 14:16:59
      4
      0.33
      2017-07-14T19:10:04.303Z
      0
      0
      4
      ...
      nan
      NaN
      147
      1.00
      19.00
      13.00
      0.00
      32.00
      33.00
      31.00

      f0c7fbc6c5415b6363c1d87a5ee7a58f4958b569
      https://www.buzzfeed.com/tylerkingkade/heres-w...
      Here's Why Schools Might Not Change Their Appr...
      2017-07-14 18:45:31.143
      2017-07-14 18:41:21
      16
      0.25
      2017-07-14T18:58:05.395Z
      0
      1
      15
      ...
      2610244.00
      2017-07-16T15:04:00.000Z
      147
      3.00
      9.00
      14.00
      11.00
      34.00
      37.00
      31.00

      05e1c884ca9a0e6e9d8c376d9964cd3322e16d8a
      https://www.buzzfeed.com/yuririaavila/this-uni...
      This University Professor Who Sexually Harasse...
      2017-07-13 17:06:19.791
      2017-07-11 16:55:02
      3
      0.25
      2017-07-13T17:20:04.766Z
      0
      1
      2
      ...
      nan
      NaN
      147
      1.00
      19.00
      13.00
      0.00
      32.00
      33.00
      31.00

25 rows × 26 columns

```python
# high response but not promoted
data.sort_values("score_diff", ascending=True).head(25)
```

      url
      headline
      discovered
      published
      fb_engagements
      fb_max_engagements_per_min
      fb_max_engagements_per_min_time
      fb_comments
      fb_reactions
      fb_shares
      ...
      fb_brand_page_likes
      fb_brand_page_time
      alexa_rank
      response_score
      lead_score
      front_score
      facebook_promotion_score
      promotion_score
      attention_index
      score_diff

      id

      954b7b6c74b2d7a8c911068c32b9e772eaabceee
      http://www.dailymail.co.uk/news/article-470472...
      Abandoned Girl Scouts camp in Ohio is up for a...
      2017-07-17 19:33:18.992
      2017-07-17 19:29:05.000
      102582
      61.42
      2017-07-20T01:50:01.248Z
      40018
      52624
      9940
      ...
      nan
      NaN
      158
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      4c81915ac1f2107bbc4d877e14cc6a9e92e829de
      http://www.huffingtonpost.com/entry/dont-forge...
      Don’t Forget to Check on Your Strong Friend
      2017-07-20 19:03:25.756
      2017-07-20 18:49:31.388
      179558
      136.77
      2017-07-22T15:50:02.068Z
      14077
      139857
      25624
      ...
      nan
      NaN
      215
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      b88dd3da90f725c21ab981ddfa8a2992bc5309cf
      https://www.washingtonpost.com/national/nprs-d...
      NPR’s Declaration of Independence tweetstorm c...
      2017-07-05 15:42:15.882
      2017-07-05 15:39:15.000
      210514
      1553.91
      2017-07-05T19:14:37.484Z
      45745
      140012
      24757
      ...
      nan
      NaN
      191
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      a89d0ceabac6464910780b5c28efd4f6bd1ba4e4
      http://www.huffingtonpost.com/entry/progress-c...
      Progress! Canada’s New Food Guide Will Favor P...
      2017-07-13 04:03:18.567
      2017-07-13 03:57:39.848
      137703
      60.92
      2017-07-14T01:22:01.315Z
      18694
      99844
      19165
      ...
      nan
      NaN
      215
      50.00
      0.00
      0.00
      0.00
      0.00
      50.00
      -50.00

      66fb936c46b9d758e2cd331e0623d37fc087b137
      https://www.nytimes.com/2017/07/20/health/thes...
      These Americans Hated the Health Law. Until th...
      2017-07-21 02:24:00.221
      2017-07-21 02:23:38.000
      88282
      115.16
      2017-07-21T13:08:02.797Z
      13336
      61337
      13609
      ...
      nan
      NaN
      120
      49.00
      0.00
      0.00
      0.00
      0.00
      49.00
      -49.00

      94859097f8ca70e132ebb4220a095f365ea14b8e
      http://www.huffingtonpost.co.uk/entry/bill-bro...
      Bill Browder's Senate Judiciary Committee Hear...
      2017-07-31 18:40:03.479
      2017-07-31 09:31:46.000
      448776
      328.71
      2017-08-01T01:18:00.260Z
      75824
      259365
      113587
      ...
      nan
      NaN
      215
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      ec893228b6221e09a2eb1b0bd11c2b0394dd64da
      http://www.mirror.co.uk/tv/tv-news/ant-mcpartl...
      Ant McPartlin set for sensational return to TV...
      2017-07-29 22:03:21.552
      2017-07-29 22:00:00.000
      141782
      383.62
      2017-07-30T08:42:00.117Z
      5511
      132605
      3666
      ...
      nan
      NaN
      706
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      ac32fcb0c75b2c204d52766b5799b09d8114c04a
      https://www.washingtonpost.com/news/post-polit...
      Sen. John McCain diagnosed with brain cancer
      2017-07-20 00:15:08.239
      2017-07-20 00:11:00.000
      316796
      5908.62
      2017-07-20T03:16:01.313Z
      52362
      238849
      25585
      ...
      nan
      NaN
      191
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      687b79edfbf3c3048b783f44de85d28cebb94717
      http://www.dailymail.co.uk/news/article-469026...
      Pictured: Eight service Marines killed in plan...
      2017-07-12 19:27:21.295
      2017-07-12 19:23:28.000
      178936
      161.86
      2017-07-13T02:00:02.797Z
      18754
      140570
      19612
      ...
      nan
      NaN
      158
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      3c01e1256ae9c1d2f33aa22996262c8079ed1fb7
      https://www.washingtonpost.com/news/post-natio...
      Trump announces ban on transgender people in U...
      2017-07-26 13:15:05.723
      2017-07-26 13:12:00.000
      111846
      13764.37
      2017-07-26T16:16:02.331Z
      40423
      62085
      9338
      ...
      nan
      NaN
      191
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      f11abf9d350269befc402d44f55ee3abf91af520
      http://www.independent.co.uk/arts-entertainmen...
      Swedish music festival to be male-only 'until ...
      2017-07-05 09:30:07.612
      2017-07-05 09:28:40.000
      118727
      179.93
      2017-07-05T19:08:05.142Z
      46585
      65392
      6750
      ...
      nan
      NaN
      386
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      56888f31d32a5fd901e5653d336a989f0da0c737
      http://www.foxnews.com/politics/2017/07/12/kid...
      Kid Rock confirms Michigan Senate run
      2017-07-12 22:50:08.532
      2017-07-12 21:02:00.000
      108543
      100.83
      2017-07-13T02:54:03.293Z
      34418
      62570
      11555
      ...
      nan
      NaN
      285
      50.00
      0.00
      1.00
      0.00
      1.00
      51.00
      -49.00

      1540d31c02f0c8276a330b59fdfbd5d060e43160
      http://www.dailymail.co.uk/femail/food/article...
      Aldi’s £10 gin is judged as one of the best in...
      2017-07-27 09:00:21.055
      2017-07-27 08:56:00.000
      98662
      150.25
      2017-07-27T12:44:03.462Z
      45999
      41113
      11550
      ...
      nan
      NaN
      158
      49.00
      0.00
      1.00
      0.00
      1.00
      50.00
      -48.00

      5c9f1cd1dd81840990b8821f6269d223be578ca2
      https://www.washingtonpost.com/news/acts-of-fa...
      The death of reading is threatening the soul
      2017-07-21 11:09:11.907
      2017-07-21 11:00:22.000
      79496
      17.27
      2017-07-24T02:34:02.215Z
      14412
      51439
      13645
      ...
      nan
      NaN
      191
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      ebdbdfd3c28dceeb41f1017d5860d174f02c6614
      http://www.huffingtonpost.com/entry/michelle-o...
      Michelle Obama's Black ESPYs Dress Is Instantl...
      2017-07-13 12:33:17.609
      2017-07-13 12:20:35.648
      82266
      682.98
      2017-07-13T17:36:02.767Z
      6072
      72003
      4191
      ...
      nan
      NaN
      215
      48.00
      0.00
      0.00
      0.00
      0.00
      48.00
      -48.00

      f7767ab339e6b9621b29d7c5cce99b62f29dcb67
      http://www.foxnews.com/opinion/2017/07/18/teen...
      Teen Vogue defends teaching kids how to engage...
      2017-07-18 21:05:03.380
      2017-07-18 20:33:00.000
      154801
      221.58
      2017-07-20T12:36:02.873Z
      30172
      106290
      18339
      ...
      nan
      NaN
      285
      50.00
      0.00
      2.00
      0.00
      2.00
      52.00
      -48.00

      46a15915cb7ed0a2779b068913912ea82f14b85c
      http://www.independent.co.uk/arts-entertainmen...
      Chester Bennington dead: Linkin Park singer 'd...
      2017-07-20 18:50:05.935
      2017-07-20 18:19:44.000
      77267
      592.73
      2017-07-20T19:16:03.534Z
      13636
      53011
      10620
      ...
      nan
      NaN
      386
      48.00
      0.00
      1.00
      0.00
      1.00
      49.00
      -47.00

      840c28eed294fd40928f664ee464ec992eecafb0
      https://www.buzzfeed.com/bradesposito/first-ma...
      Andy Murray Had The Best Response To A Journal...
      2017-07-13 03:06:21.983
      2017-07-13 03:04:45.000
      70434
      69.08
      2017-07-13T19:06:01.868Z
      3080
      63451
      3903
      ...
      nan
      NaN
      147
      47.00
      0.00
      0.00
      0.00
      0.00
      47.00
      -47.00

      4c7162a85740562763630c0b09bb6d321c8dc41c
      http://www.foxbusiness.com/features/2017/07/24...
      Wisconsin company to implant microchips in its...
      2017-07-24 15:20:04.536
      2017-07-24 14:10:00.000
      89054
      73.75
      2017-07-24T22:52:03.564Z
      41925
      34154
      12975
      ...
      nan
      NaN
      285
      49.00
      0.00
      2.00
      0.00
      2.00
      51.00
      -47.00

      80100e34326c76660aa20b435aea08dbda228f55
      http://www.independent.co.uk/news/uk/home-news...
      Thousands march in London to demand Theresa Ma...
      2017-07-01 14:00:18.218
      2017-07-01 13:25:07.000
      52671
      108.33
      2017-07-01T16:16:04.265Z
      5598
      42352
      4721
      ...
      nan
      NaN
      386
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      70225a16b434ea03740598f4b353c5488ca9ba3e
      http://www.dailymail.co.uk/news/article-471066...
      OJ Simpson caught masturbating in cell could i...
      2017-07-19 16:09:20.156
      2017-07-19 15:15:14.000
      62694
      244.83
      2017-07-19T17:22:03.596Z
      31686
      25837
      5171
      ...
      nan
      NaN
      158
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      0de8d1509eab1935f9220fa59d77c500f0103ebe
      http://www.huffingtonpost.com/entry/betsy-devo...
      Betsy DeVos Is Under Fire For Scaling Back Cam...
      2017-07-06 21:42:10.688
      2017-07-06 10:38:36.000
      55083
      37.30
      2017-07-11T01:38:02.933Z
      7788
      41455
      5840
      ...
      nan
      NaN
      215
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      4d6d8f3ca4893301b2a21b9994a6f805de679fc1
      http://www.independent.co.uk/news/world/americ...
      US planning to create 'Space Corps' as sixth b...
      2017-07-01 13:00:17.514
      2017-07-01 10:17:05.000
      58681
      138.89
      2017-07-02T01:48:02.549Z
      14637
      38038
      6006
      ...
      nan
      NaN
      386
      46.00
      0.00
      0.00
      0.00
      0.00
      46.00
      -46.00

      651f1a663c5f1d7ab4db22c509c4efaca3ebeec4
      http://www.nbcnews.com/news/us-news/trump-tell...
      Trump Tells French First Lady She's 'In Such G...
      2017-07-13 19:18:04.624
      2017-07-13 19:17:14.000
      61775
      197.27
      2017-07-13T21:10:02.015Z
      26261
      32116
      3398
      ...
      nan
      NaN
      826
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

      b5214aa403db36975c90a3921749c6ecbb67a76a
      http://www.dailymail.co.uk/tvshowbiz/article-4...
      Love Island's Amber Davies 'fears she might be...
      2017-07-08 00:03:23.755
      2017-07-08 00:01:56.000
      67612
      139.36
      2017-07-08T09:54:03.167Z
      29474
      22932
      15206
      ...
      nan
      NaN
      158
      47.00
      0.00
      1.00
      0.00
      1.00
      48.00
      -46.00

25 rows × 26 columns

Write that data to a file. Note that the scores here are provisional for two reasons:
1. they should be using a rolling-month based on the article publication date to calculate medians/min/max etc, whereas in this workbook we as just using values for the month of May
2. for analysis, we've rounded the numbers; we don't expect to do that for the actual scores

```python
data.to_csv("articles_with_provisional_scores_2017-07-01_2017-07-31.csv")
```

# Summary

The attention index of an article is comprised of four components:

- *lead score* (max 20 points) based on how long an article was the lead story on the publisher's home page, scaled by the traffic to that publisher
- *front score* (max 15 points) based on how long an article was present on the publisher's home page, scaled by traffic to that publisher
- *Facebook promotion score* (max 15 points) based on whether the article was promoted to the publisher's Facebook brand page, scaled by the reach of that brand page
- *response score* (max 50 points) based on the number of Facebook engagements the article received, relative to other articles

Or, in other words:

\begin{align}
attentionIndex_a &= leadScore_a + frontScore_a + facebookPromotionScore_a + responseScore_a \\
leadScore_a &= 20 \cdot \left(\frac{\min(minsAsLead_a, 60)}{alexaRank_a}\right) \cdot \left( \frac{\min(alexaRank)}{60} \right) \\
frontScore_a &=
15 \cdot
\left(\frac{\min(minsOnFront_a, 1440)}{alexaRank_a \cdot numArticlesOnFront_a}\right) \cdot
\left( \frac{\min(alexaRank \cdot numArticlesOnFront)}{1440} \right) \\
facebookPromotion_a &=
\begin{cases}
 0 \text{ if not shared on brand page }\\
   15 \cdot \frac{\log(brandPageLikes_a) - \log(\min(brandPageLikes))}{\log(\max(brandPageLikes)) - \log(\min(brandPageLikes))}  \text{ otherwise }
\end{cases}
\\
responseScore_a &=
\begin{cases}
   0 \text{ if } engagements_a = 0 \\
   50 \cdot
   \frac{\log(\min(engagements_a,limit) + median(engagements)) - \log(1 + median(engagements))}
   {\log(limit + median(engagements)) - \log(1 + median(engagements))} \text{ if } engagements_a > 0
\end{cases} \\
\end{align}

```python

```