```python
%matplotlib inline
```

================================================================
Morphing source estimates: Moving data from one brain to another
================================================================

Morphing refers to the operation of transferring
`source estimates <sphx_glr_auto_tutorials_plot_object_source_estimate.py>`
from one anatomy to another. It is commonly referred as realignment in fMRI
literature. This operation is necessary for group studies as one needs
then data in a common space.

In this tutorial we will morph different kinds of source estimation results
between individual subject spaces using :class:`mne.SourceMorph` object.

We will use precomputed data and morph surface and volume source estimates to a
reference anatomy. The common space of choice will be FreeSurfer's 'fsaverage'
See `sphx_glr_auto_tutorials_plot_background_freesurfer.py` for more
information. Method used for cortical surface data in based
on spherical registration [1]_ and Symmetric Diffeomorphic Registration (SDR)
for volumic data [2]_.

Furthermore we will convert our volume source estimate into a NIfTI image using
:meth:`morph.apply(..., output='nifti1') <mne.SourceMorph.apply>`.

In order to morph :class:`labels <mne.Label>` between subjects allowing the
definition of labels in a one brain and transforming them to anatomically
analogous labels in another use :func:`mne.Label.morph`.

Why morphing?
=============

Modern neuroimaging techniques, such as source reconstruction or fMRI analyses,
make use of advanced mathematical models and hardware to map brain activity
patterns into a subject specific anatomical brain space.

This enables the study of spatio-temporal brain activity. The representation of
spatio-temporal brain data is often mapped onto the anatomical brain structure
to relate functional and anatomical maps. Thereby activity patterns are
overlaid with anatomical locations that supposedly produced the activity.
Anatomical MR images are often used as such or are transformed into an inflated
surface representations to serve as  "canvas" for the visualization.

In order to compute group level statistics, data representations across
subjects must be morphed to a common frame, such that anatomically and
functional similar structures are represented at the same spatial location for
*all subjects equally*.

Since brains vary, morphing comes into play to tell us how the data
produced by subject A, would be represented on the brain of subject B.

See also this `tutorial on surface source estimation
<sphx_glr_auto_tutorials_plot_mne_solutions.py>`
or this `example on volumetric source estimation
<sphx_glr_auto_examples_inverse_plot_compute_mne_inverse_volume.py>`.

Morphing **volume** source estimates
====================================

A volumetric source estimate represents functional data in a volumetric 3D
space. The difference between a volumetric representation and a "mesh" (
commonly referred to as "3D-model"), is that the volume is "filled" while the
mesh is "empty". Thus it is not only necessary to morph the points of the
outer hull, but also the "content" of the volume.

In MNE-Python, volumetric source estimates are represented as
:class:`mne.VolSourceEstimate`. The morph was successful if functional data of
Subject A overlaps with anatomical data of Subject B, in the same way it does
for Subject A.

Setting up :class:`mne.SourceMorph` for :class:`mne.VolSourceEstimate`
----------------------------------------------------------------------

Morphing volumetric data from subject A to subject B requires a non-linear
registration step between the anatomical T1 image of subject A to
the anatomical T1 image of subject B.

MNE-Python uses the Symmetric Diffeomorphic Registration [2]_ as implemented
in dipy_ [3]_ (See
`tutorial <http://nipy.org/dipy/examples_built/syn_registration_3d.html>`_
from dipy_ for more details).

:class:`mne.SourceMorph` uses segmented anatomical MR images computed
using `FreeSurfer <sphx_glr_auto_tutorials_plot_background_freesurfer.py>`
to compute the transformations. In order tell SourceMorph which MRIs to use,
``subject_from`` and ``subject_to`` need to be defined as the name of the
respective folder in FreeSurfer's home directory.

See `sphx_glr_auto_examples_inverse_plot_morph_volume_stc.py`
usage and for more details on:

    - How to create a SourceMorph object for volumetric data

    - Apply it to VolSourceEstimate

    - Get the output is NIfTI format

    - Save a SourceMorph object to disk

Morphing **surface** source estimates
=====================================

A surface source estimate represents data relative to a 3-dimensional mesh of
the cortical surface computed using FreeSurfer. This mesh is defined by
its vertices. If we want to morph our data from one brain to another, then
this translates to finding the correct transformation to transform each
vertex from Subject A into a corresponding vertex of Subject B. Under the hood
`FreeSurfer <sphx_glr_auto_tutorials_plot_background_freesurfer.py>`
uses spherical representations to compute the morph, as relies on so
called *morphing maps*.

The morphing maps
-----------------

The MNE software accomplishes morphing with help of morphing
maps which can be either computed on demand or precomputed.
The morphing is performed with help
of the registered spherical surfaces (``lh.sphere.reg`` and ``rh.sphere.reg`` )
which must be produced in FreeSurfer.
A morphing map is a linear mapping from cortical surface values
in subject A ($x^{(A)}$) to those in another
subject B ($x^{(B)}$)

\begin{align}x^{(B)} = M^{(AB)} x^{(A)}\ ,\end{align}

where $M^{(AB)}$ is a sparse matrix
with at most three nonzero elements on each row. These elements
are determined as follows. First, using the aligned spherical surfaces,
for each vertex $x_j^{(B)}$, find the triangle $T_j^{(A)}$ on the
spherical surface of subject A which contains the location $x_j^{(B)}$.
Next, find the numbers of the vertices of this triangle and set
the corresponding elements on the *j* th row of $M^{(AB)}$ so that
$x_j^{(B)}$ will be a linear interpolation between the triangle vertex
values reflecting the location $x_j^{(B)}$ within the triangle
$T_j^{(A)}$.

It follows from the above definition that in general

\begin{align}M^{(AB)} \neq (M^{(BA)})^{-1}\ ,\end{align}

*i.e.*,

\begin{align}x_{(A)} \neq M^{(BA)} M^{(AB)} x^{(A)}\ ,\end{align}

even if

\begin{align}x^{(A)} \approx M^{(BA)} M^{(AB)} x^{(A)}\ ,\end{align}

*i.e.*, the mapping is *almost* a bijection.

Morphing maps can be computed on the fly or read with
:func:`mne.read_morph_map`. Precomputed maps are
located in ``$SUBJECTS_DIR/morph-maps``.

The names of the files in ``$SUBJECTS_DIR/morph-maps`` are
of the form:

 <*A*> - <*B*> -``morph.fif`` ,

where <*A*> and <*B*> are names of subjects. These files contain the maps
for both hemispheres, and in both directions, *i.e.*, both $M^{(AB)}$
and $M^{(BA)}$, as defined above. Thus the files
<*A*> - <*B*> -``morph.fif`` or <*B*> - <*A*> -``morph.fif`` are
functionally equivalent. The name of the file produced depends on the role
of <*A*> and <*B*> in the analysis.

About smoothing
---------------

The current estimates are normally defined only in a decimated
grid which is a sparse subset of the vertices in the triangular
tessellation of the cortical surface. Therefore, any sparse set
of values is distributed to neighboring vertices to make the visualized
results easily understandable. This procedure has been traditionally
called smoothing but a more appropriate name
might be smudging or blurring in
accordance with similar operations in image processing programs.

In MNE software terms, smoothing of the vertex data is an
iterative procedure, which produces a blurred image $x^{(N)}$ from
the original sparse image $x^{(0)}$ by applying
in each iteration step a sparse blurring matrix:

\begin{align}x^{(p)} = S^{(p)} x^{(p - 1)}\ .\end{align}

On each row $j$ of the matrix $S^{(p)}$ there
are $N_j^{(p - 1)}$ nonzero entries whose values
equal $1/N_j^{(p - 1)}$. Here $N_j^{(p - 1)}$ is
the number of immediate neighbors of vertex $j$ which
had non-zero values at iteration step $p - 1$.
Matrix $S^{(p)}$ thus assigns the average
of the non-zero neighbors as the new value for vertex $j$.
One important feature of this procedure is that it tends to preserve
the amplitudes while blurring the surface image.

Once the indices non-zero vertices in $x^{(0)}$ and
the topology of the triangulation are fixed the matrices $S^{(p)}$ are
fixed and independent of the data. Therefore, it would be in principle
possible to construct a composite blurring matrix

\begin{align}S^{(N)} = \prod_{p = 1}^N {S^{(p)}}\ .\end{align}

However, it turns out to be computationally more effective
to do blurring with an iteration. The above formula for $S^{(N)}$ also
shows that the smudging (smoothing) operation is linear.

From theory to practice
-----------------------

In MNE-Python, surface source estimates are represented as
:class:`mne.SourceEstimate` or :class:`mne.VectorSourceEstimate`. Those can
be used together with :class:`mne.SourceSpaces` or without.

The morph was successful if functional data of Subject A overlaps with
anatomical surface data of Subject B, in the same way it does for Subject A.

See `sphx_glr_auto_examples_inverse_plot_morph_surface_stc.py`
usage and for more details:

    - How to create a :class:`mne.SourceMorph` object using
      :func:`mne.compute_source_morph` for surface data

    - Apply it to :class:`mne.SourceEstimate` or
      :class:`mne.VectorSourceEstimate`

    - Save a :class:`mne.SourceMorph` object to disk

Please see also Gramfort *et al.* (2013) [4]_.

References
==========
.. [1] Greve D. N., Van der Haegen L., Cai Q., Stufflebeam S., Sabuncu M.
       R., Fischl B., Brysbaert M.
       A Surface-based Analysis of Language Lateralization and Cortical
       Asymmetry. Journal of Cognitive Neuroscience 25(9), 1477-1492, 2013.
.. [2] Avants, B. B., Epstein, C. L., Grossman, M., & Gee, J. C. (2009).
       Symmetric Diffeomorphic Image Registration with Cross- Correlation:
       Evaluating Automated Labeling of Elderly and Neurodegenerative
       Brain, 12(1), 26-41.
.. [3] Garyfallidis E, Brett M, Amirbekian B, Rokem A, van der Walt S,
       Descoteaux M, Nimmo-Smith I and Dipy Contributors (2014). DIPY, a
       library for the analysis of diffusion MRI data. Frontiers in
       Neuroinformatics, vol.8, no.8.
.. [4] Gramfort A., Luessi M., Larson E., Engemann D. A., Strohmeier D.,
       Brodbeck C., Goj R., Jas. M., Brooks T., Parkkonen L. & Hämäläinen, M.
       (2013). MEG and EEG data analysis with MNE-Python. Frontiers in
       neuroscience, 7, 267.


```python
%matplotlib inline
```

```python
from IPython.core.display import HTML
css_file = './inet.css'
HTML(open(css_file, "r").read())
```

```python
import matplotlib.pyplot as plt
from matplotlib import collections  as mc
plt.xkcd()

import numpy as np
import pandas as pd
import seaborn as sn
import statsmodels.formula.api as smf

from IPython.html import widgets
from IPython.display import display

import pypwt
```

    :0: FutureWarning: IPython widgets are experimental and may change in the future.

 (Linear) Regression

Q: What is regression?

A: Drawing a "line" through a scatter of data points in a principled way...

Q: What is linear regression?

A: Drawing a straight "line" through a scatter of data points in a principled way...

Linear regression in practice

 Step 1: Grab some data...

```python
# grab the entire Penn World Tables data from the web...
pwt = pypwt.load_pwt_data()
```

```python
#...this gives us a panel (i.e., two dimensional) data set
pwt
```

    <class 'pandas.core.panel.Panel'>
    Dimensions: 38 (items) x 167 (major_axis) x 62 (minor_axis)
    Items axis: country to delta_k
    Major_axis axis: AGO to ZWE
    Minor_axis axis: 1950-01-01 00:00:00 to 2011-01-01 00:00:00

```python
def labor_supply(data, year="1950-01-01"):
    """
    Labor supply in a given year is the product of number of employed
    persons, 'emp', and the average number of hours worked, 'avh'.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    year : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".

    Returns
    -------
    L : pandas.Series
        Effective labor supply in units of employed person-years.

    """
    L = data.minor_xs(year)["emp"] * data.minor_xs(year)["avh"]
    return L

def real_gdp_per_unit_labor(data, year="1950-01-01"):
    """
    Real gross domestic product (GDP) per unit labor is the ratio of
    some measure of real output and some measure of labor supply.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    year : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".

    Returns
    -------
    rgdppul : pandas.Series
        Real gdp per unit labor supply.

    """
    rgdppul = data.minor_xs(year)["rgdpo"] / labor_supply(data, year)
    return rgdppul

def growth_rate_real_gdp_per_unit_labor(data, start="1950-01-01", end="2011-01-01"):
    """
    Plot the growth rate of real GDP per unit labor over some time period
    against the level of real GDP per unit labor at the start of the time
    period.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    start : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    end : str (default="2011-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".

    """
    gr = (np.log(real_gdp_per_unit_labor(data, end)) -
          np.log(real_gdp_per_unit_labor(data, start)))
    return gr

def some_interesting_plot(data, start="1950-01-01", end="2011-01-01"):
    """
    Plot the growth rate of real GDP per unit labor over some time period
    against the level of real GDP per unit labor at the start of the time
    period.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    start : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    end : str (default="2011-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".

    """
    # create the scatter plot
    fig, ax = plt.subplots(1, 1, figsize=(12,9))
    xs = np.log(real_gdp_per_unit_labor(data, start))
    ys = growth_rate_real_gdp_per_unit_labor(data, start, end)
    ax.scatter(xs, ys, color='k')

    # axis labels, title, etc
    ax.set_xlabel('Log income (per unit labor) in 1960', fontsize=25)
    ax.set_xlim(0.95 * xs.min(), 1.05 * xs.max())
    ax.set_ylabel('Income (per unit labor) growth\n({}-{})'.format(start[:4], end[:4]),
                  fontsize=25)
    ax.set_ylim(1.05 * ys.min(), 1.05 * ys.max())
    ax.set_title('Do poor countries grow faster than rich countries?',
                 fontsize=25, family="serif")
```

Let's focus on a relation between...

1. (log) level of real income
2. subsequent growth rate of income

...note that income is "per unit labor supply"

Step 2: plot your data...

```python
some_interesting_plot(data=pwt, start="1960-01-01", end="2010-01-01")
```

Step 3: draw a (straight!) line through your data...

```python
def another_interesting_plot(data, start, end, intercept=2.5, slope=-0.5):
    """
    Plot the growth rate of real GDP per unit labor over some time period
    against the level of real GDP per unit labor at the start of the time
    period. Then add a regression line associated with given values for the
    intercept and slope.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    start : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    end : str (default="2011-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    intercept: float (defalut=2.5)
        Intercept for the regression line.
    slope : float (default=-0.5)
        Slope for the regression line.

    """
    # create the scatter plot
    fig, ax = plt.subplots(1, 1, figsize=(12,9))
    xs = np.log(real_gdp_per_unit_labor(data, start))
    ys = growth_rate_real_gdp_per_unit_labor(data, start, end)
    ax.scatter(xs, ys, color='k')

    # compute the regression line given params
    grid = np.linspace(0.95 * xs.min(), 1.05 * xs.max(), 1000)
    predicted = lambda x: intercept + slope * x
    yhat, = ax.plot(grid, predicted(grid) , color='b',
                    label=r"$\hat{y}_i=%.2f + %.2fx_i$" %(intercept, slope))

    # axis labels, title, etc
    ax.set_xlabel('Log income (per unit labor) in 1960', fontsize=25)
    ax.set_xlim(0.95 * xs.min(), 1.05 * xs.max())
    ax.set_ylabel('Income (per unit labor) growth\n({}-{})'.format(start[:4], end[:4]),
                  fontsize=25)
    ax.set_ylim(1.05 * ys.min(), 1.05 * ys.max())
    ax.legend(bbox_to_anchor=(1.0, 0.95), prop={'size': 25})

```

Since we are drawing straight lines...

$$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

where

\begin{align}
    y_i=& \text{income growth from 1960-2010} \\
    x_i=& \ln (\text{income in 1960})
\end{align}

and $i=\text{AGO},\dots,\text{ZWE}$.

```python
another_interesting_plot(pwt, start="1960-01-01", end="2010-01-01")
```

Step 4: choose some measure of deviation between model and data...

Given our choice of model...

$$ y_i = \beta_0 + \beta_1 x_i + \epsilon_i,\ i=1,\dots,N $$

...focus on the sum of squared errors (SSE):

$$ SSE \equiv \sum_{i=1}^N  \epsilon_i^2 = \sum_{i=1}^N \big(y_i - (\beta_0 + \beta_1x_i)\big)^2 $$

Step 5: choose params to minimize deviation between model and data...

```python
def another_static_plot(data, start, end, intercept=2.5, slope=-0.5):
    """
    Plot the growth rate of real GDP per unit labor over some time period
    against the level of real GDP per unit labor at the start of the time
    period. Then add a regression line associated with given values for the
    intercept and slope.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    start : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    end : str (default="2011-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    intercept: float (default=2.5)
        Intercept for the regression line.
    slope : float (default=-0.5)
        Slope for the regression line.

    """
    # create the scatter plot
    fig, ax = plt.subplots(1, 1, figsize=(12,9))
    xs = np.log(real_gdp_per_unit_labor(data, start))
    ys = growth_rate_real_gdp_per_unit_labor(data, start, end)
    ax.scatter(xs, ys, color='k')

    # compute and plot the regression line given current params
    predicted = lambda x: intercept + slope * x
    grid = np.linspace(0.95 * xs.min(), 1.05 * xs.max(), 1000)
    yhat, = ax.plot(grid, predicted(grid) , color='b')
    ssr = np.sum((predicted(xs) - ys)**2)

    # add the residuals to the plot
    lines = zip(zip(xs, ys), zip(xs, predicted(xs)))
    lc = mc.LineCollection(lines, colors='r', linewidths=2)
    ax.add_collection(lc)

    # axis labels, title, etc
    ax.set_xlabel('Log income (per unit labor) in 1960', fontsize=25)
    ax.set_xlim(0.95 * xs.min(), 1.05 * xs.max())
    ax.set_ylabel('Income (per unit labor) growth\n({}-{})'.format(start[:4], end[:4]),
                  fontsize=25)
    fig.suptitle(r'Sum of squared errors (SSE) $\equiv \sum_{i=1}^N (y_i - (\beta_0 + \beta_1x_i))^2$',
                 x=0.5, y=1.025, fontsize=25, family="serif")
    fig.legend([yhat, lc], [r"$\hat{y}_i=\hat{\beta}_0 + \hat{\beta}_1x_i$", r"$SSE={0:.4f}$".format(ssr)],
               bbox_to_anchor=(0.8, 0.85), prop={'size': 25})

# create the interactive widget
intercept_widget = widgets.FloatSlider(value=2.5e0, min=0.0, max=5.0, step=5e-2, description=r"$\hat{\beta}_0$")
slope_widget = widgets.FloatSlider(value=-0.5, min=-1.0, max=1.0, step=5e-2, description=r"$\hat{\beta}_1$")
some_interactive_plot = widgets.interactive(another_static_plot,
                                            data=widgets.fixed(pwt),
                                            start=widgets.fixed("1960-01-01"),
                                            end=widgets.fixed("2010-01-01"),
                                            intercept=intercept_widget,
                                            slope=slope_widget,
                                           )
```

Informal parameter estimation...

```python
display(some_interactive_plot)
```

Formal parameter estimation...

Our estimates $\hat{\beta}_0$ and $\hat{\beta}_1$ solve the following optimization problem...

$$ \min_{\beta_0, \beta_1} \sum_{i=1}^N (y_i - (\beta_0 + \beta_1x_i))^2 $$

Taking of derivatives yields a system of first-order conditions...

\begin{align}
    0 =& -2\sum_{i=1}^N (y_i - (\beta_0 + \beta_1x_i)) \\
    0 =& -2\sum_{i=1}^N (y_i - (\beta_0 + \beta_1x_i))x_i
\end{align}

...which after some algebra yields...

\begin{align}
    \hat{\beta}_0 =& \bar{y} - \hat{\beta}_1 \bar{x} \\
    \hat{\beta}_1 =& \frac{\frac{1}{N}\sum_{i=1}^N(x_i - \bar{x})(y_i - \bar{y})}{\frac{1}{N}\sum_{i=1}^N(x_i - \bar{x})^2}
\end{align}

Intuition for $\hat{\beta}_1$...

\begin{align}
    \hat{\beta}_1 =& \frac{Cov(x, y)}{Var(x)} \\
    \equiv& \frac{\text{How much do $x$ and $y$ vary together?}}{\text{How much does $x$ vary in general?}}
\end{align}

```python
# compute the optimal parameter estimates "by hand"
estimated_slope = xs.cov(ys) / xs.var()
estimated_intercept = ys.mean() - estimated_slope * xs.mean()
```

```python
print "Estimated intercept: {}".format(estimated_intercept)
print "Estimated slope: {}".format(estimated_slope)
```

    Estimated intercept: 2.60212926608
    Estimated slope: -0.610771408878

```python
# define the variables
xs = np.log(real_gdp_per_unit_labor(pwt, '1960-01-01'))
ys = growth_rate_real_gdp_per_unit_labor(pwt, '1960-01-01', '2010-01-01')

# compute and plot the regression line given current params
data = pd.DataFrame.from_dict({'x': xs, 'y': ys})
lm = smf.ols(formula="y ~ x", data=data).fit()
```

```python
lm.summary()
```

OLS Regression Results

  Dep. Variable:	            y	          R-squared:         	    0.438

  Model:	                   OLS	         Adj. R-squared:    	    0.420

  Method:	             Least Squares	    F-statistic:       	    24.19

  Date:	             Thu, 21 May 2015	   Prob (F-statistic):	 2.71e-05

  Time:	                 10:50:38	       Log-Likelihood:    	  -21.613

  No. Observations:	          33	        AIC:               	    47.23

  Df Residuals:	              31	        BIC:               	    50.22

  Df Model:	                   1

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  Intercept	     2.6021	     0.276	     9.442	  0.000	     2.040     3.164

  x	            -0.6108	     0.124	    -4.919	  0.000	    -0.864    -0.358

  Omnibus:	        6.668	   Durbin-Watson:     	    1.627

  Prob(Omnibus):	  0.036	   Jarque-Bera (JB):  	    5.773

  Skew:	          -1.021	   Prob(JB):          	   0.0558

  Kurtosis:	       3.178	   Cond. No.          	     8.68

```python
def plot_fitted_model(data, start="1950-01-01", end="2011-01-01"):
    """
    Plot the growth rate of real GDP per unit labor over some time period
    against the level of real GDP per unit labor at the start of the time
    period. Then add a regression line associated with given values for the
    intercept and slope.

    Parameters
    ----------
    data : pandas.Panel
        The Penn World Tables (PWT) data set.
    start : str (default="1950-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".
    end : str (default="2011-01-01")
        Some year between 1950 and 2011. Format should be "YYYY-01-01".

    """
    # create the scatter plot
    fig, ax = plt.subplots(1, 1, figsize=(12,9))
    xs = np.log(real_gdp_per_unit_labor(data, start))
    ys = (np.log(real_gdp_per_unit_labor(data, end)) -
          np.log(real_gdp_per_unit_labor(data, start)))
    ax.scatter(xs, ys, color='k')

    # compute and plot the regression line given current params
    data = pd.DataFrame.from_dict({'x': xs, 'y': ys})
    lm = smf.ols(formula="y ~ x", data=data).fit()
    predicted = lambda x: lm.params[0] + lm.params[1] * x

    grid = np.linspace(0.95 * xs.min(), 1.05 * xs.max(), 1000)
    yhat, = ax.plot(grid, predicted(grid) , color='b')
    ssr = np.sum((predicted(xs) - ys)**2)

    # add the residuals to the plot
    lines = zip(zip(xs, ys), zip(xs, predicted(xs)))
    lc = mc.LineCollection(lines, colors='r', linewidths=2)
    ax.add_collection(lc)

    # axis labels, title, etc
    ax.set_xlabel('Log income (per unit labor) in 1960', fontsize=25)
    ax.set_xlim(0.95 * xs.min(), 1.05 * xs.max())
    ax.set_ylabel('Income (per unit labor) growth\n({}-{})'.format(start[:4], end[:4]),
                  fontsize=25)
    ax.set_ylim(1.05 * ys.min(), 1.05 * ys.max())
    fig.legend([yhat, lc], [r"$\hat{y}_i=%.4f + %.4f x_i$" % (lm.params[0], lm.params[1]),
                            r"$SSR={0:.4f}$".format(ssr)],
               bbox_to_anchor=(0.85, 0.85), prop={'size': 25})

```

Plotting the final result...

```python
plot_fitted_model(pwt, '1960-01-01', '2010-01-01')
```

Have we found any evidence for convergence?

Yes...

* slope coefficient has the correct sign

...but...

What about...

* Omitted variable bias!
* Sample selection issues!
* Measurement error?
* etc.


# Building a Spam Filter with Naive Bayes

In this project, we're going to build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. Our goal is to write a program that classifies new messages with an accuracy greater than 80% — so we expect that more than 80% of the new messages will be classified correctly as spam or ham (non-spam).

To train the algorithm, we'll use a dataset of 5,572 SMS messages that are already classified by humans. The dataset was put together by Tiago A. Almeida and José María Gómez Hidalgo, and it can be downloaded from the [The UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). The data collection process is described in more details on [this page](http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/#composition), where you can also find some of the papers authored by Tiago A. Almeida and José María Gómez Hidalgo.

## Exploring the Dataset

We'll now start by reading in the dataset.

```python
import pandas as pd

sms_spam = pd.read_csv('SMSSpamCollection', sep='\t', header=None, names=['Label', 'SMS'])

print(sms_spam.shape)
sms_spam.head()
```

    (5572, 2)

      Label
      SMS

      0
      ham
      Go until jurong point, crazy.. Available only ...

      1
      ham
      Ok lar... Joking wif u oni...

      2
      spam
      Free entry in 2 a wkly comp to win FA Cup fina...

      3
      ham
      U dun say so early hor... U c already then say...

      4
      ham
      Nah I don't think he goes to usf, he lives aro...

Below, we see that about 87% of the messages are ham, and the remaining 13% are spam. This sample looks representative, since in practice most messages that people receive are ham.

```python
sms_spam['Label'].value_counts(normalize=True)
```

    ham     0.865937
    spam    0.134063
    Name: Label, dtype: float64

## Training and Test Set

We're now going to split our dataset into a training and a test set, where the training set accounts for 80% of the data, and the test set for the remaining 20%.

```python
# Randomize the dataset
data_randomized = sms_spam.sample(frac=1, random_state=1)

# Calculate index for split
training_test_index = round(len(data_randomized) * 0.8)

# Training/Test split
training_set = data_randomized[:training_test_index].reset_index(drop=True)
test_set = data_randomized[training_test_index:].reset_index(drop=True)

print(training_set.shape)
print(test_set.shape)
```

    (4458, 2)
    (1114, 2)

We'll now analyze the percentage of spam and ham messages in the training and test sets. We expect the percentages to be close to what we have in the full dataset, where about 87% of the messages are ham, and the remaining 13% are spam.

```python
training_set['Label'].value_counts(normalize=True)
```

    ham     0.86541
    spam    0.13459
    Name: Label, dtype: float64

```python
test_set['Label'].value_counts(normalize=True)
```

    ham     0.868043
    spam    0.131957
    Name: Label, dtype: float64

The results look good! We'll now move on to cleaning the dataset.

## Data Cleaning

To calculate all the probabilities required by the algorithm, we'll first need to perform a bit of data cleaning to bring the data in a format that will allow us to extract easily all the information we need.

Essentially, we want to bring data to this format:

### Letter Case and Punctuation

We'll begin with removing all the punctuation and bringing every letter to lower case.

```python
# Before cleaning
training_set.head()
```

      Label
      SMS

      0
      ham
      Yep, by the pretty sculpture

      1
      ham
      Yes, princess. Are you going to make me moan?

      2
      ham
      Welp apparently he retired

      3
      ham
      Havent.

      4
      ham
      I forgot 2 ask ü all smth.. There's a card on ...

```python
# After cleaning
training_set['SMS'] = training_set['SMS'].str.replace('\W', ' ')
training_set['SMS'] = training_set['SMS'].str.lower()
training_set.head()
```

      Label
      SMS

      0
      ham
      yep  by the pretty sculpture

      1
      ham
      yes  princess  are you going to make me moan

      2
      ham
      welp apparently he retired

      3
      ham
      havent

      4
      ham
      i forgot 2 ask ü all smth   there s a card on ...

### Creating the Vocabulary

Let's now move to creating the vocabulary, which in this context means a list with all the unique words in our training set.

```python
training_set['SMS'] = training_set['SMS'].str.split()

vocabulary = []
for sms in training_set['SMS']:
    for word in sms:
        vocabulary.append(word)

vocabulary = list(set(vocabulary))
```

It looks like there are 7,783 unique words in all the messages of our training set.

```python
len(vocabulary)
```

    7783

### The Final Training Set

We're now going to use the vocabulary we just created to make the data transformation we want.

```python
word_counts_per_sms = {unique_word: [0] * len(training_set['SMS']) for unique_word in vocabulary}

for index, sms in enumerate(training_set['SMS']):
    for word in sms:
        word_counts_per_sms[word][index] += 1
```

```python
word_counts = pd.DataFrame(word_counts_per_sms)
word_counts.head()
```

      user
      repeating
      mob
      exmpel
      145
      basically
      wrkin
      as
      dieting
      contacts
      ...
      simpler
      nalli
      yest
      barkleys
      could
      services
      waaaat
      store
      aha
      okors

      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      4
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 7783 columns

```python
training_set_clean = pd.concat([training_set, word_counts], axis=1)
training_set_clean.head()
```

      Label
      SMS
      ticket
      kappa
      too
      abdomen
      unhappy
      hoody
      start
      die
      ...
      09058095201
      chase
      thru
      ru
      xclusive
      fellow
      red
      entitled
      auto
      bothering

      0
      ham
      [yep, by, the, pretty, sculpture]
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      ham
      [yes, princess, are, you, going, to, make, me,...
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      ham
      [welp, apparently, he, retired]
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3
      ham
      [havent]
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      4
      ham
      [i, forgot, 2, ask, ü, all, smth, there, s, a,...
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 7785 columns

## Calculating Constants First

We're now done with cleaning the training set, and we can begin creating the spam filter. The Naive Bayes algorithm will need to answer these two probability questions to be able to classify new messages:

\begin{equation}
P(Spam | w_1,w_2, ..., w_n) \propto P(Spam) \cdot \prod_{i=1}^{n}P(w_i|Spam)
\end{equation}

\begin{equation}
P(Ham | w_1,w_2, ..., w_n) \propto P(Ham) \cdot \prod_{i=1}^{n}P(w_i|Ham)
\end{equation}

Also, to calculate P(wi|Spam) and P(wi|Ham) inside the formulas above, we'll need to use these equations:

\begin{equation}
P(w_i|Spam) = \frac{N_{w_i|Spam} + \alpha}{N_{Spam} + \alpha \cdot N_{Vocabulary}}
\end{equation}

\begin{equation}
P(w_i|Ham) = \frac{N_{w_i|Ham} + \alpha}{N_{Ham} + \alpha \cdot N_{Vocabulary}}
\end{equation}

Some of the terms in the four equations above will have the same value for every new message. We can calculate the value of these terms once and avoid doing the computations again when a new messages comes in. Below, we'll use our training set to calculate:

- P(Spam) and P(Ham)
- NSpam, NHam, NVocabulary

We'll also use Laplace smoothing and set $\alpha = 1$.

```python
# Isolating spam and ham messages first
spam_messages = training_set_clean[training_set_clean['Label'] == 'spam']
ham_messages = training_set_clean[training_set_clean['Label'] == 'ham']

# P(Spam) and P(Ham)
p_spam = len(spam_messages) / len(training_set_clean)
p_ham = len(ham_messages) / len(training_set_clean)

# N_Spam
n_words_per_spam_message = spam_messages['SMS'].apply(len)
n_spam = n_words_per_spam_message.sum()

# N_Ham
n_words_per_ham_message = ham_messages['SMS'].apply(len)
n_ham = n_words_per_ham_message.sum()

# N_Vocabulary
n_vocabulary = len(vocabulary)

# Laplace smoothing
alpha = 1
```

## Calculating Parameters

Now that we have the constant terms calculated above, we can move on with calculating the parameters $P(w_i|Spam)$ and $P(w_i|Ham)$. Each parameter will thus be a conditional probability value associated with each word in the vocabulary.

The parameters are calculated using the formulas:

\begin{equation}
P(w_i|Spam) = \frac{N_{w_i|Spam} + \alpha}{N_{Spam} + \alpha \cdot N_{Vocabulary}}
\end{equation}

\begin{equation}
P(w_i|Ham) = \frac{N_{w_i|Ham} + \alpha}{N_{Ham} + \alpha \cdot N_{Vocabulary}}
\end{equation}

```python
# Initiate parameters
parameters_spam = {unique_word:0 for unique_word in vocabulary}
parameters_ham = {unique_word:0 for unique_word in vocabulary}

# Calculate parameters
for word in vocabulary:
    n_word_given_spam = spam_messages[word].sum()   # spam_messages already defined in a cell above
    p_word_given_spam = (n_word_given_spam + alpha) / (n_spam + alpha*n_vocabulary)
    parameters_spam[word] = p_word_given_spam

    n_word_given_ham = ham_messages[word].sum()   # ham_messages already defined in a cell above
    p_word_given_ham = (n_word_given_ham + alpha) / (n_ham + alpha*n_vocabulary)
    parameters_ham[word] = p_word_given_ham
```

## Classifying A New Message

Now that we have all our parameters calculated, we can start creating the spam filter. The spam filter can be understood as a function that:

- Takes in as input a new message (w1, w2, ..., wn).
- Calculates P(Spam|w1, w2, ..., wn) and P(Ham|w1, w2, ..., wn).
- Compares the values of P(Spam|w1, w2, ..., wn) and P(Ham|w1, w2, ..., wn), and:
    - If P(Ham|w1, w2, ..., wn) > P(Spam|w1, w2, ..., wn), then the message is classified as ham.
    - If P(Ham|w1, w2, ..., wn) < P(Spam|w1, w2, ..., wn), then the message is classified as spam.
    -  If P(Ham|w1, w2, ..., wn) = P(Spam|w1, w2, ..., wn), then the algorithm may request human help.

```python
import re

def classify(message):
    '''
    message: a string
    '''

    message = re.sub('\W', ' ', message)
    message = message.lower().split()

    p_spam_given_message = p_spam
    p_ham_given_message = p_ham

    for word in message:
        if word in parameters_spam:
            p_spam_given_message *= parameters_spam[word]

        if word in parameters_ham:
            p_ham_given_message *= parameters_ham[word]

    print('P(Spam|message):', p_spam_given_message)
    print('P(Ham|message):', p_ham_given_message)

    if p_ham_given_message > p_spam_given_message:
        print('Label: Ham')
    elif p_ham_given_message < p_spam_given_message:
        print('Label: Spam')
    else:
        print('Equal proabilities, have a human classify this!')
```

```python
classify('WINNER!! This is the secret code to unlock the money: C3421.')
```

    P(Spam|message): 1.3481290211300841e-25
    P(Ham|message): 1.9368049028589875e-27
    Label: Spam

```python
classify("Sounds good, Tom, then see u there")
```

    P(Spam|message): 2.4372375665888117e-25
    P(Ham|message): 3.687530435009238e-21
    Label: Ham

## Measuring the Spam Filter's Accuracy

The two results above look promising, but let's see how well the filter does on our test set, which has 1,114 messages.

We'll start by writing a function that returns classification labels instead of printing them.

```python
def classify_test_set(message):
    '''
    message: a string
    '''

    message = re.sub('\W', ' ', message)
    message = message.lower().split()

    p_spam_given_message = p_spam
    p_ham_given_message = p_ham

    for word in message:
        if word in parameters_spam:
            p_spam_given_message *= parameters_spam[word]

        if word in parameters_ham:
            p_ham_given_message *= parameters_ham[word]

    if p_ham_given_message > p_spam_given_message:
        return 'ham'
    elif p_spam_given_message > p_ham_given_message:
        return 'spam'
    else:
        return 'needs human classification'
```

Now that we have a function that returns labels instead of printing them, we can use it to create a new column in our test set.

```python
test_set['predicted'] = test_set['SMS'].apply(classify_test_set)
test_set.head()
```

      Label
      SMS
      predicted

      0
      ham
      Later i guess. I needa do mcat study too.
      ham

      1
      ham
      But i haf enuff space got like 4 mb...
      ham

      2
      spam
      Had your mobile 10 mths? Update to latest Oran...
      spam

      3
      ham
      All sounds good. Fingers . Makes it difficult ...
      ham

      4
      ham
      All done, all handed in. Don't know if mega sh...
      ham

Now, we'll write a function to measure the accuracy of our spam filter to find out how well our spam filter does.

```python
correct = 0
total = test_set.shape[0]

for row in test_set.iterrows():
    row = row[1]
    if row['Label'] == row['predicted']:
        correct += 1

print('Correct:', correct)
print('Incorrect:', total - correct)
print('Accuracy:', correct/total)
```

    Correct: 1100
    Incorrect: 14
    Accuracy: 0.9874326750448833

The accuracy is close to 98.74%, which is really good. Our spam filter looked at 1,114 messages that it hasn't seen in training, and classified 1,100 correctly.

## Next Steps

In this project, we managed to build a spam filter for SMS messages using the multinomial Naive Bayes algorithm. The filter had an accuracy of 98.74% on the test set we used, which is a pretty good result. Our initial goal was an accuracy of over 80%, and we managed to do way better than that.

Next steps include:

- Analyze the 14 messages that were classified incorrectly and try to figure out why the algorithm classified them incorrectly
- Make the filtering process more complex by making the algorithm sensitive to letter case