# How does the SFF method work?

Vanderburg and Johnson (2014) introduced a method for "Self Flat Fielding" by tracking how the lightcurve changes with motion of the spacecraft:

[A Technique for Extracting Highly Precise Photometry for the Two-Wheeled Kepler Mission](http://adsabs.harvard.edu/abs/2014PASP..126..948V)

In this notebook we replicate the K2SFF method following the same example source, #60021426, as that in the publication.  We aim to demystify the technique, which is extremely popular within the K2 community.  We have focused on reproducibility, so that we achieve the same result at the publication.

The Vanderburg & Johnson 2014 paper uses data from the Kepler two-wheel "Concept Engineering Test", predating campaign 0, and sometimes called campaign *"eng"* or abbreviated CET.  This vestigal "campaign" lacks some of the standardization of later K2 campaigns--- it was much shorter, only about 9 days long, it lacks some of the standard quality flags, targets have non-traditional EPIC IDs, and other quirks.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
import pandas as pd
```

## Retrieve the K2SFF data for ENG test source `60021426`

First we will retrieve data and inspect the mask used in the paper.

```python
path = 'http://archive.stsci.edu/hlsps/k2sff/cet/060000000/21426/hlsp_k2sff_k2_lightcurve_060021426-cet_kepler_v1_llc.fits'
vdb_fits = fits.open(path)
```

The `BESTAPER` keyword explains which aperture was chosen as the "best" by Vanderburg & Johnson 2014.  The FITS header for that slice contains the metadata needed to reproduce the mask.

```python
keys = ['MASKTYPE', 'MASKINDE', 'NPIXSAP']
_ = [print(key, ' : ', vdb_fits['BESTAPER'].header[key]) for key in keys]
```

    MASKTYPE  :  PRF FIT
    MASKINDE  :  4
    NPIXSAP  :  29.0

We want the *exact same* mask as Vanderburg & Johnson 2014, but the publication version and MAST version differ!

Publication version:

MAST Version:

Aperture 7 should yield a bigger mask, more similar to what was used in the paper.

```python
VDB_J_mask = vdb_fits['PRF_APER_TBL'].data[7,:, :] == True
VDB_J_mask.sum()
```

    53

Save the mask for easy use in our next notebook.

```python
np.save('VDB_J_2014_mask.npy', VDB_J_mask)
```

## Manually reproduce with the Vanderburg-provided diagnostic data
Retrieve the Vanderburg-provided diagnostic data for the Kepler ENG testing.
Uncomment the line below to retrieve the data programmatically, or manually get the linked file in a browser and save it to this directory.

```python
#! wget https://www.cfa.harvard.edu/~avanderb/k2/ep60021426alldiagnostics.csv
```

```python
df = pd.read_csv('ep60021426alldiagnostics.csv',index_col=False)
df.head()
```

      BJD - 2454833
      Raw Flux
      Corrected Flux
      X-centroid
      Y-centroid
      arclength
      Correction
      Thrusters On

      0
      1862.502368
      0.995119
      0.995985
      25.135097
      24.661074
      2.327480
      0.999130
      1.0

      1
      1862.522801
      0.997313
      0.996767
      25.289752
      24.418689
      1.175322
      1.000548
      1.0

      2
      1862.543235
      0.996713
      0.996136
      25.288052
      24.429406
      1.214627
      1.000580
      0.0

      3
      1862.563668
      0.996930
      0.996277
      25.275216
      24.448405
      1.306617
      1.000656
      0.0

      4
      1862.584102
      0.996862
      0.996228
      25.253864
      24.480184
      1.460259
      1.000636
      0.0

We can mean-subtract the provided $x-y$ centroids, assigning them column and row identifiers, then rotate the coordinates into their major and minor axes.

```python
col = df[' X-centroid'].values
col = col - np.mean(col)
row = df[' Y-centroid'].values
row = row - np.mean(row)
```

```python
def _get_eigen_vectors(centroid_col, centroid_row):
    '''get the eigenvalues and eigenvectors given centroid x, y positions'''
    centroids = np.array([centroid_col, centroid_row])
    eig_val, eig_vec = np.linalg.eigh(np.cov(centroids))
    return eig_val, eig_vec
```

```python
def _rotate(eig_vec, centroid_col, centroid_row):
    '''rotate the centroids into their predominant linear axis'''
    centroids = np.array([centroid_col, centroid_row])
    return np.dot(eig_vec, centroids)
```

```python
eig_val, eig_vec = _get_eigen_vectors(col, row)
v1, v2 = eig_vec
```

The major axis is the latter.

```python
platescale = 4.0 # The Kepler plate scale; has units of arcseconds / pixel
```

```python
plt.figure(figsize=(5, 6))
plt.plot(col * platescale, row * platescale, 'ko', ms=4)
plt.plot(col * platescale, row * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel('X position [arcseconds]')
plt.ylabel('Y position [arcseconds]')
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, v1[0]], [0, v1[1]], color='blue', lw=3)
plt.plot([0, v2[0]], [0, v2[1]], color='blue', lw=3);
```

Following the form of **Figure 2** of Vanderburg & Johsnon 2014.

```python
rot_colp, rot_rowp = _rotate(eig_vec, col, row) #units in pixels
```

You can rotate into the new reference frame.

```python
plt.figure(figsize=(5, 6))
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ko', ms=4)
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel("X' position [arcseconds]")
plt.ylabel("Y' position [arcseconds]")
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, 1], [0, 0], color='blue')
plt.plot([0, 0], [0, 1], color='blue');
```

We need to calculate the arclength using:
\begin{equation}s= \int_{x'_0}^{x'_1}\sqrt{1+\left( \frac{dy'_p}{dx'}\right)^2} dx'\end{equation}

where $x^\prime_0$ is the transformed $x$ coordinate of the point with the smallest $x^\prime$ position, and $y^\prime_p$ is the best--fit polynomial function.

Fit a $5^{th}$ order polynomial to the rotated coordinates.

```python
z = np.polyfit(rot_rowp, rot_colp, 5)
p5 = np.poly1d(z)
p5_deriv = p5.deriv()
```

```python
x0_prime = np.min(rot_rowp)
xmax_prime = np.max(rot_rowp)
x_dense = np.linspace(x0_prime, xmax_prime, 2000)
```

```python
plt.plot(rot_rowp, rot_colp, '.')
plt.plot(x_dense, p5(x_dense))
plt.ylabel('Position along minor axis (pixels)')
plt.xlabel('Position along major axis (pixels)')
plt.title('Performance of polynomial regression')
plt.ylim(-0.1, 0.1);
```

We see evidence for a [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff), suggesting some modest opportunity for improvement.

```python
@np.vectorize
def arclength(x):
    '''Input x1_prime, get out arclength'''
    gi = x_dense <x
    s_integrand = np.sqrt(1 + p5_deriv(x_dense[gi]) ** 2)
    s = np.trapz(s_integrand, x=x_dense[gi])
    return s
```

Let's double check that we compute the same arclength as the published paper.

```python
aspect_ratio = plt.figaspect(1)
plt.figure(figsize=aspect_ratio)
plt.plot(df[' arclength'], arclength(rot_rowp)*4.0, '.')
plt.xlabel('$s$  (Vanderburg & Johnson 2014)')
plt.ylabel('$s$  (This work)')
plt.plot([0, 4], [0, 4], 'k--');
```

Yes, we compute arclength correctly.

Now we apply a **high-pass filter** to the raw lightcurve data.  We follow the original paper by using *BSplines* with 1.5 day breakpoints.  You can also apply data exclusion at this stage.

```python
from scipy.interpolate import BSpline
from scipy import interpolate
```

```python
times, raw_fluxes = df['BJD - 2454833'].values, df[' Raw Flux'].values
```

We find the weighted least square spline for a given set of knots, $t$.  We supply interior knots as knots on the ends are added automatically, as stated in the `interpolate.splrep()` docstring.

```python
interior_knots = np.arange(times[0]+1.5, times[0]+6, 1.5)
t,c,k = interpolate.splrep(times, raw_fluxes, s=0, task=-1, t=interior_knots)
bspl = BSpline(t,c,k)
```

```python
plt.plot(times, raw_fluxes, '.')
plt.plot(times, bspl(times))
plt.xlabel('$t$ (days)')
plt.ylabel('Raw Flux');
```

The Spline fit looks good, so we can normalize the flux by the long-term trend.
Plot the normalized flux versus arclength to see the position-dependent flux.

```python
fluxes = raw_fluxes/bspl(times)
```

Mask the data by keeping only the good samples.

```python
bi = df[' Thrusters On'].values == 1.0
gi = df[' Thrusters On'].values == 0.0
clean_fluxes = fluxes[gi]
al = arclength(rot_rowp[gi]) * platescale
```

```python
sorted_inds = np.argsort(al)
```

We will follow the paper by interpolating **flux versus arclength position** in 15 bins of means, which is a *piecewise linear fit*.

```python
knots = np.array([np.min(al)]+
                 [np.median(splt) for splt in np.array_split(al[sorted_inds], 15)]+
                 [np.max(al)])
```

```python
bin_means = np.array([clean_fluxes[sorted_inds][0]]+
                     [np.mean(splt) for splt in np.array_split(clean_fluxes[sorted_inds], 15)]+
                     [clean_fluxes[sorted_inds][-1]])
```

```python
zz = np.polyfit(al, clean_fluxes,6)
sff = np.poly1d(zz)
al_dense = np.linspace(0, 4, 1000)
interp_func = interpolate.interp1d(knots, bin_means)
```

```python
plt.figure(figsize=(5, 6))
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'ko', ms=4)
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'o', color='#3498db', ms=3)
plt.plot(arclength(rot_rowp[bi])*4.0, fluxes[bi], 'o', color='r', ms=3)
plt.plot(np.sort(al), interp_func(np.sort(al)), '-', color='#e67e22')

plt.xticks([0, 1,2, 3, 4])
plt.minorticks_on()
plt.xlabel('Arclength [arcseconds]')
plt.ylabel('Relative Brightness')
plt.title('EPIC 60021426, Kp =10.3')
plt.xlim(0,4)
plt.ylim(0.997, 1.002);
```

Following **Figure 4** of Vanderburg & Johnson 2014.

Apply the Self Flat Field (SFF) correction:

```python
corr_fluxes = clean_fluxes / interp_func(al)
```

```python
plt.figure(figsize=(10,6))

dy = 0.004
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'ko', ms=4)
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'o', color='#3498db', ms=3)
plt.plot(df['BJD - 2454833'][bi], df[' Raw Flux'][bi]+dy, 'o', color='r', ms=3)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='k', ms = 4)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='#e67e22', ms = 3)

plt.xlabel('BJD - 2454833')
plt.ylabel('Relative Brightness')

plt.xlim(1862, 1870)
plt.ylim(0.994, 1.008);
```

Following **Figure 5** of Vanderburg & Johnson 2015.

*The end.*


### TODO:

* use sympy to generate the file to show
* show how to do the line plot using VTK
* make sure the slider works

```
from cStringIO import StringIO

def figure2png(fig):
        buffer = StringIO()
        fig.savefig(buffer, format='png')
        buffer.reset()
        data = buffer.read().encode('base64')
        return "data:image/png;base64,{0}".format(data.replace('\n', ''))

class Slider(object):

    def __init__(self, python_var, callback, min=1, max=100, step=1):
        self._min = min
        self._max = max
        self._step = step
        self._value_init = (min+max)/2
        self._id = "slider" + str(id(self))
        self._python_var = python_var
        self._callback = callback

    def update(self, value):
        return self._callback(value)

    def _repr_html_(self):
        javascript = """ \

""" % {"id": self._id, "python_var": self._python_var}
        html = """ \
%(min)s <input id="%(id)s" type="range" min="%(min)s" max="%(max)s" step="%(step)s" value="%(value_init)s"
    onchange="%(id)s_updated(this.value, false)" onmouseup="%(id)s_updated(this.value, true)" /> %(max)s</div>
<div id="chosen%(id)s">%(python_var)s = %(value_init)s</div>
Result: <div id="%(id)s_result_output"></div>
<br>
""" % {"id": self._id, "min": self._min, "max": self._max, "step": self._step, "value_init": self._value_init,
        "python_var": self._python_var}
        return javascript + html
```

```
from vtk import (vtkSphereSource, vtkPolyDataMapper, vtkActor, vtkRenderer,
        vtkRenderWindow, vtkWindowToImageFilter, vtkPNGWriter, vtkStructuredGridReader, vtkPlane,
        vtkCutter, vtkDataSetMapper, vtkContourFilter, vtkCamera, vtkOutlineSource,
        vtkRectilinearGridReader, vtkLineSource, vtkProbeFilter, vtkDataObjectToTable)
from vtk.util.numpy_support import vtk_to_numpy
from numpy import array, linspace, sqrt
from pylab import plot

def vtk_show(renderer, w=100, h=100):
    """
    Takes vtkRenderer instance and returns an IPython Image with the rendering.
    """
    renderWindow = vtkRenderWindow()
    renderWindow.SetOffScreenRendering(1)
    renderWindow.AddRenderer(renderer)
    renderWindow.SetSize(w, h)
    renderWindow.Render()

    windowToImageFilter = vtkWindowToImageFilter()
    windowToImageFilter.SetInput(renderWindow)
    windowToImageFilter.Update()

    writer = vtkPNGWriter()
    writer.SetWriteToMemory(1)
    writer.SetInputConnection(windowToImageFilter.GetOutputPort())
    writer.Write()
    data = str(buffer(writer.GetResult()))

    from IPython.display import Image
    return Image(data)

def get_plane(r, n1, n2, n3):
    plane = vtkPlane()
    plane.SetOrigin(r.GetOutput().GetCenter())
    plane.SetNormal(n1, n2, n3)

    c = vtkCutter()
    c.SetInputConnection(r.GetOutputPort())
    c.SetCutFunction(plane)

    m = vtkPolyDataMapper()
    m.SetInputConnection(c.GetOutputPort())
    m.SetScalarRange(r.GetOutput().GetScalarRange())

    a = vtkActor()
    a.SetMapper(m)
    return a

def get_outline(r):
    s = vtkOutlineSource()
    s.SetBounds(r.GetOutput().GetBounds())
    m = vtkDataSetMapper()
    m.SetInputConnection(s.GetOutputPort())
    a = vtkActor()
    a.SetMapper(m)
    a.GetProperty().SetColor(0, 0, 0)
    return a

def get_contour(r):
    iso = vtkContourFilter()
    iso.SetInputConnection(r.GetOutputPort())
    rmin, rmax = r.GetOutput().GetScalarRange()
    iso.SetValue(0, (rmin+rmax)/2)
    #iso.SetValue(1, (rmin+rmax)/4)

    m = vtkPolyDataMapper()
    m.SetInputConnection(iso.GetOutputPort())
    m.SetScalarRange(r.GetOutput().GetScalarRange())
    a = vtkActor()
    a.SetMapper(m)
    return a

def get_renderer(l):
    renderer = vtkRenderer()
    for a in l:
        renderer.AddActor(a)
    renderer.SetBackground(1, 1, 1)
    return renderer

def line_values(r, a, b, n):
    a = array(a)
    b = array(b)
    line = vtkLineSource()
    line.SetPoint1(a)
    line.SetPoint2(b)
    line.SetResolution(n)
    probe = vtkProbeFilter()
    probe.SetInputConnection(line.GetOutputPort())
    probe.SetSourceData(r.GetOutput())
    dot = vtkDataObjectToTable()
    dot.SetInputConnection(probe.GetOutputPort())
    dot.Update()
    table = dot.GetOutput()
    x = linspace(0, sqrt(sum((a-b)**2)), n+1)
    y = vtk_to_numpy(table.GetColumn(0))
    return x, y
```

```
r = vtkRectilinearGridReader()
r.SetFileName("data/Venq.vtk")
r.Update()

renderer = get_renderer([
    get_plane(r, 1, 0, 0),
    get_plane(r, 0, 1, 0),
    get_plane(r, 0, 0, 1),
    get_outline(r),
    #get_contour(r),
    ])

camera = renderer.GetActiveCamera()
renderer.ResetCamera()
camera.Elevation(30)
camera.Azimuth(30)

vtk_show(renderer, 500, 500)
```

```
def update(value):
    camera = vtkCamera()
    camera.Elevation(value)
    camera.Azimuth(30)
    renderer.SetActiveCamera(camera)
    renderer.ResetCamera()
    i = vtk_show(renderer, 500, 500)
    data = i.data.encode('base64')
    return "data:image/png;base64,{0}".format(data.replace('\n', ''))

s = Slider("s", update)
s
```

 1 <input id="slider72952400" type="range" min="1" max="100" step="1" value="50"
    onchange="slider72952400_updated(this.value, false)" onmouseup="slider72952400_updated(this.value, true)" /> 100

s = 50

Result:

```
from sympy import var, factor
var("x")
def update(value):
    return factor(x**value-1)

t = Slider("t", update)
t
```

 1 <input id="slider59049296" type="range" min="1" max="100" step="1" value="50"
    onchange="slider59049296_updated(this.value, false)" onmouseup="slider59049296_updated(this.value, true)" /> 100

t = 50

Result:

```

```


# Introduction
> Introduction to Radio Astronomy, Spectrometers and DESHIMA

- toc: true
- badges: false
- comments: true
- permalink: /introduction
- image: images/introduction.jpg
- sectionnr: 2

The ultimate weapon in the arsenal of a historian just might be a time machine. While historians can infer a lot from written texts and archaeological digs, no substitute can outperform being in the past and observing. Unfortunately for historians, a time machine has yet to be invented. Astronomers, however, are more fortunate in this field.

Around the turn of the century, new physics was being invented to explain previous observations and conclusions. As a result, it was understood that the speed of light was finite and moreover a constant, independent of the speed of the observer, which Albert Einstein postulated in 1905{% cite einstein_1905 %}. It logically follows that the arriving light from distant stars and galaxies is reaching us from the past.

Since light is a wave traveling through the expanding universe, it undergoes Doppler shift and therefore cosmological objects that move away from the observer undergo a shift in the frequency of light. This is known as redshift, because the expanding wavelength means that visible light would shift towards the red end of the spectrum. Just 12 years after Einstein's discovery, American Astronomer Vesto Slipher {% cite slipher_1917 %} discovered the redshift of distant galaxies and with it cosmological spectroscopy was born.

***A part of the light spectrum. Visible light is expanded to show the different colors. Note how red light in the visible spectrum has a longer wavelength, giving redshift its name. Image taken from {% cite lemke_2020 %}***

Combining these effects gives us a sort of astronomical time machine: looking far out into the cosmos means looking into the past where the universe was much younger. However, just pointing a telescope at the night sky poses a few problems.

## Cosmological formation

Understanding how stars and galaxies form out of a uniformly gaseous universe is one of the largest areas of research in modern physical cosmology. The physical processes by which these happen might be well understood, but the amount of possible initial values of these processes are so vast that they can have wildly different outcomes{% cite blain_2002 %}. Looking at these events occurring in the distant universe (and therefore distant past) is therefore important to further the understanding of the forming of the universe.

Unfortunately, much of the formation of these cosmological bodies happens inside thick dust clouds which are opaque to visible light and are therefore investigated using what is called sub-millimeter astronomy: astronomy ranging from $100\:\mathrm{\mu m}$ to about $1\:\mathrm{mm}$. In frequency terms, following

$$\begin{equation}
f = \frac{c}{\lambda}
\end{equation}$$

this corresponds with about $150\:\mathrm{GHz}$ to $1500\:\mathrm{GHz}$.

***The Messer-16 nebula, dubbed "The Eagle", captured by the Hubble Space Telescope is an example of a dust cloud that is opaque to visible light. Image taken from {% cite garner_2017 %}***

## Redshift

As mentioned before, the redshift of light gives us the opportunity to calculate the relative speed and therefore distance and age of astronomical sources. By the term "redshift", I mean the following formal definition {% cite endo_session_2 %}

$$\begin{equation}
1+z = \frac{f_\mathrm{emit}}{f_\mathrm{obs}}
\end{equation}$$

where $z$ denotes redshift and $f_\mathrm{emit}$ and $f_\mathrm{obs}$ denote the emitted and observed frequency of a signal respectively. In order to calculate redshift we therefore need to know both frequencies.

### Emitted frequency

Quantum mechanics tells us that the electron configuration in atoms exist in quantized states. That is to say: they can move between defined states, but not in the continuum inbetween. When an atom moves from one configuration to a higher state energy is absorbed, when it moves to a lower state energy is emitted. These quantized energy states give way to what is known as spectral lines, where a specific wavelength is known to be emitted by specific atoms.

***A few energy states of carbon. Note how higher energy gaps result in shorter wavelengths. Image taken from {% cite nave_hyperphysics %}***

The energy gaps shown are to wide to emit photons in the submillimeter wavelength range DESHIMA operates in, however *fine structure* spectral lines are in the correct wavelength gap. This is the splitting of two similar states due to spin and relativistic effects. Another mechanism by which molecules emit spectral lines is due to the shifting in quantized rotational states. The various states astronomical sources split between are well known and therefore can be tracked to measure redshift.

This becomes clear when we look at the spectral flux density $E_{e,\nu}$ of a galaxy, simulated with the Python module `galspec`{% cite galspec %}.

```python
#hide
!pip install -q galspec numpy pandas plotly "deshima-sensitivity==0.3.0"

import numpy as np
import pandas as pd
import plotly.express as px

import warnings
warnings.filterwarnings('ignore')

from thesis_plot import plot

# Page Specific
import galspec
from deshima_sensitivity.atmosphere import eta_atm_func
```

```python
#hide
h = 6.62607004 * 10**-34
k_b = 1.38064852 * 10**-23
e = 1.60217662 * 10**-19 # electron charge
c = 299792458.
```

```python
#hide_input
fLow = 30 #GHz
fHigh = 1000 #GHz
numFreqBins = 1000
linewidth = 600

z = [3,5,7,11]

gal_freq, gal_flux_density = galspec.spectrum(12, 1, fLow, fHigh, numFreqBins, linewidth, mollines = 'True')

df = pd.DataFrame({'freq':gal_freq, '1': gal_flux_density})

for zi in z:
    gal_freq, gal_flux_density = galspec.spectrum(12, zi, fLow, fHigh, numFreqBins, linewidth, mollines = 'True')
    df = df.join(pd.DataFrame({str(zi): gal_flux_density}))

df.set_index('freq', inplace=True)
plot(df,axis_labels=('$\\nu\:\mathrm{[GHz]}$','$E_{e,\\nu}\:\mathrm{[Jy]}$','$z$'),hover_labels=('nu [GHz]','Spectral flux density [Jy]','z'),e_notation=(False,True),plot_type='loglog')
```

***The spectral flux density of a simulated galaxy with different redshifts. Here the spectral flux is given in units of Jansky, which is equivalent to 10-26 WM-2Hz-1***

From the figure it is clear that with higher $z$ the peaks move further to the left, towards lower frequencies and therefore longer wavelengths. If one were to look for peaks in incoming signal and measure the wavelength at which these peaks occur, the redshift and thus age of a galaxy can be detected. Besides these clear spikes in the flux density, the galaxy also emits broad spectrum thermal noise, originating from the black-body emission of interstellar dust.

### Observed frequency

In order to know the wavelength of these spectral lines at our detection point, we need to measure the specific frequency of an incoming photon in a spectrometer. This can be done using a *Coherent Heterodyne receiver*, except that these devices work to a bandwidth of about $36\:\mathrm{GHz}${% cite baker_2007 %}, much lower than the spectral range we are interested in as shown above.

Traditional photon detectors, at their core, work by converting an incoming photon into a charge or current, which means that the specific momentum, energy or frequency of the photon is lost. A way around this is to move the incoming photons depending on their wavelength, such as what happens in an optical prism.

***When a photon enters the prism, its angle changes depending on its wavelength. Image taken from {% cite edmund_optics%}***

This is how a lot of spectrometers work, but due to the long wavelengths and wide redshift range $(1+z\approx1-10)${% cite endo_2019 %} of the signals involved in galaxy spectroscopy this approach is cumbersome. The long wavelengths and range means that the spectrometer needs to physically large and to reduce the noise they need to be cooled, making the entire system even larger. This means that these devices are inherently not scalable to multiple pixels.

## DESHIMA
DESHIMA, short for the Deep Spectroscopic High-Redshift Mapper, is a spectrometer developed by a joint team of researchers from various institutions including the TU Delft and SRON that takes another approach{% cite endo_2019 %}{% cite endo_2019_2 %}. DESHIMA uses a series of bandpass filters on a superconducting chip to direct incoming photons into bins, where they are detected by Microwave Kinetic Inductance Detectors (MKIDs). The advantage of this approach is that the spectrometer itself is really small, taking up just a few squared centimeters and is therefore easily scalable to $\gt 100$ spectrometer pixels{% cite endo_2019 %}.

***A schematic overview of the DESHIMA 2.0 chip. Image taken from {% cite huijten_2020%}***

DESHIMA is placed on ASTE, a ground-based sub-millimeter telescope near the ALMA site in the Atacama Desert.

***A panoramic shot from the ASTE site in the Atacama Desert. The building on the left is NANTEN-2 whereas the building on the left is ASTE. Image taken from {% cite ASTE_panorama %}***

## Ground-based submillimeter astronomy

While the ASTE site is high up in the Atacama desert, the telescope still needs to look through some atmosphere when doing observations. Since cosmological object of interest are often far away and therefore very dim, they are much dimmer than the effects from the atmosphere. The atmosphere is almost completely opaque at some frequencies DESHIMA operates in and this loads the antenna dish because it is much hotter and therefore brighter than the astronomical source.

Below the atmospheric loading at the ASTE site is calculated by the software package `deshima-sensitivity`{% cite deshima-sensitivity%} and plotted with a galaxy spectrum synthesized by `galspec`:

```python
#hide
pwv = 1

eta_A = 0.6 # aperture efficiency
telescope_diameter = 10
A_p = (telescope_diameter/2)**2 * np.pi # telescope physical area
A_e = eta_A * A_p  # telescope effective area

F, S_gal = galspec.spectrum(luminosity=13.7, redshift=4.43, fLow=fLow, fHigh=fHigh, numFreqBins=numFreqBins, linewidth=linewidth)

eta_atm = eta_atm_func(F=F, pwv=pwv, EL=90)
T_p_atm = 273.
S_sky = (1-eta_atm) * T_p_atm * 1e26 * 2 *k_b /A_e

S_gal_gnd = S_gal*eta_atm

```

```python
#hide_input
df = pd.DataFrame({'F':F, 'Atmosphere': S_sky, 'Galaxy above atmosphere': S_gal, 'Galaxy from ground': S_gal_gnd})
df = df.set_index('F')
plot(df,axis_labels=('$\\nu\:\mathrm{[GHz]}$','$E_{e,\\nu}\:\mathrm{[Jy]}$',''),hover_labels=('nu [GHz]','Spectral flux density [Jy]',''),e_notation=(False,False),plot_type='loglin',y_range=[-4,5])
```

***Comparing the spectral flux density of atmospheric loading to loading from an astronomical source. The former is over 3 orders of magnitude brighter. Note how the atmospheric transmission means that for some frequencies the spectrum is barely usable.***

The atmosphere is over 3 orders of magnitude brighter than the galaxy spectrum before the latter is attenuated. After the atmospheric attenuation the difference is even bigger. Therefore modeling the expected loading from the atmosphere and the entire chain from antenna to detector is important to be able to do measurements and to find the signal within the noise. `deshima-sensitivity`{% cite deshima-sensitivity%} was designed to do this, but takes some approximations that were deemed too reductive.

## This Project

The current version of `deshima-sensitivity` does not take filter profiles into consideration when calculating the power received on each channel. It samples the power spectral density calculated to be at the filter's center frequency and multiplies this with the filter bandwidth, creating a crude box filter. While this approach holds for flat power spectra, it doesn't quite hold for spectra like the one above.

In this project I will modify the existing `deshima-sensitivity` package to more accurately resemble the DESHIMA 2.0 filter profiles and therefore the spectrometer as a whole. In order to do so, I will first need to dive into the DESHIMA system in more depth and then give an overview of *photon statistics* and the associated photon noise. Finally I will model the system and compare it with previous versions of `deshima-sensitivity`.

Next Chapter

# Bibliography
{% bibliography --cited_in_order %}


```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
import pyross
```

In this notebook we consider a control protocol consisting of an initial lockdown, which is then partly released. For our numerical study we generate synthetic data using the stochastic SEAI5R model.

While we use the Denmark age structure and contact matrix, we emphasise that **the model parameters considered in this notebook have NOT been obtained from real data, but rather are chosen ad-hoc**.

**Outline of this notebook:**

1. We briefly summarize the SEAI5R model.
2. We load the age structure and contact matrix for Denmark. The contact matrix is generally given as
\begin{equation}
    C = C_H + C_W + C_S + C_O,
\end{equation}
where the four terms denote the number of contacts at home, work, school, and all other remaining contacts.
3. We define the other model parameters of the SEAI5R model **(again, these are not fitted to any real data, but rather chosen ad-hoc)**.
4. We define a "lockdown-protocol":
    1. After a fixed time, a lockdown is imposed. The contact matrix is reduced to
    \begin{equation}
        C = C_H + 0.1\cdot C_W + 0.4 \cdot C_O,
        \end{equation}
        i.e. the "home" contacts $C_H$, and 10% of the "work" as well as 40% of the "other" contacts are reainted. The latter two model that even in lockdown, people go to work, and that initially people might be lenient in following the lockdown advice.
    2. 5 Days after the initial lockdown, the 40% "social interaction" part of the contact matrix is removed, and the contact matrix becomes
        \begin{equation}
        C = C_H + 0.1\cdot C_W.
        \end{equation}
        This models that people take the lockdown seriously and seize to interact socially.
    3. 50 Days after the initial lockdown, 50% of the "school" contacts are added to the contact matrix,
        \begin{equation}
        C = C_H + 0.1\cdot C_W + 0.5 \cdot C_S.
        \end{equation}
        This models an opening of schools, with a social distancing protocol in place such that the number of contacts at school is halved.
5. We run 150 stochastic simulations of this lockdown protocol, and find that in Stage 2 (only home and 10% of work contacts), the total number of known active infectives (the sum of the stages symptomatic infective, hospitalised, and in ICU) shows a maximum and then decreases. After schools are opened, the total number of known active infectives shows a minimum and starts to decrease again. We plot the distribution of waiting time between change in the contact matrix and the extrema in the stochastic time series.
6. Finally, we consider the protocol from point 4 again, but this time with a prefactor 0.2 in front of $C_S$, signifying a more strict social distancing protocol. Running 150 stochastic simulations with this protocol, we find that this time the total number of active infectives keeps decreases even after schools have been opened.

## 1. Brief summary of the model

The SEAI5R model contains the following stages:

If a susceptible individual is infected, it enters the "exposed" class, where it does not show symptoms and is not yet infective. In the subsequent class, called "activated", the individual still does not show symptoms, but is infective. After the "activated" class, an individual is either a symptomatic or an asymptomatic infective. Those who are asymptomatic recover, those who show symptoms either recover, or need to be hospitalised. The condition of a hospitalised individual might improve, so that the individual recovers, or worsens, at which point the individual enters the "ICU" class. From this class the individual can again recover, or enter a final "mortality" class.

## 2. Load Denmark age structure and contact matrix

```python
my_data = np.genfromtxt('../data/age_structures/Denmark-2019.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

Ni0=aM+aF;

M=16  ## number of age classes

Ni = Ni0[:M]
N=np.sum(Ni)

print("Age groups are in brackets of 5 (i.e. 0-4, 5-9, 10-14, .. , 75-79).")
print("Number of individuals in each bracket:")
print(Ni.astype('int'))
print("Total number of individuals: {0}".format(np.sum(Ni.astype('int'))))
```

    Age groups are in brackets of 5 (i.e. 0-4, 5-9, 10-14, .. , 75-79).
    Number of individuals in each bracket:
    [302353 305513 338779 341219 379522 395469 342443 320132 366147 385944
     422585 381360 338039 319145 346572 220374]
    Total number of individuals: 5505596

```python
# Get individual contact matrices
CH, CW, CS, CO = pyross.contactMatrix.Denmark()

# By default, home, work, school, and others contribute to the contact matrix
C = CH + CW + CS + CO

# Illustrate the individual contact matrices:
fig,aCF =  plt.subplots(2,2,figsize=(10,10));
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
for i in range(2):
    for j in range(2):
        aCF[i][j].set_xlabel('Age group of individual')
        aCF[i][j].set_ylabel('Age group of contact')
```

The above contact matrices illustrate the interactions at home (upper left), work (upper right), school (lower left), and the other remaining contacts (lower right). x- and y- axes denote the age groups, a darker color indicates more interaction.

## 3. Define model parameters

**Note: These have not been fitted to real data.**

```python
beta  = 0.036692            # infection rate

gE    = 1/5.
gA    = 1/4.
gIa   = 1./3.                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.4                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/7
gIc   = 1/14

sa    = 0*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 0            # birth
sa[3] = 0          # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
hh[:M] = 0.01
hh[M:2*M] = 0.025
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));

Ia_0 = 10*np.ones((M)) # start with 10 asymptomatic infectives
R_0  = np.zeros((M))
E_0  = np.zeros((M))
A_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (E_0 + A_0 + Ia_0 + Is_0 + Ih_0 + Ic_0 +Im_0 + R_0)
```

## 4. Define events for protocol

```python
# Dummy event for initial (standard) contact matrix
events = [lambda t: 1]
contactMatrices = [C]

# After 20 days, start lockdown
lockdown_threshold_0 = 20
def event0(t,rp):
    return t - lockdown_threshold_0
events.append(event0)
contactMatrices.append( CH + 0.1*CW + 0.4*CO )

# After 25 days, decrease contacts even further
lockdown_threshold_1 = 25
def event1(t,rp):
    return t- lockdown_threshold_1
events.append(event1)
contactMatrices.append( CH + 0.1*CW)

# After 70 days, add 50% of school contacts to contact matrix
lockdown_threshold_2 = 70
def event2(t,rp):
    return t - lockdown_threshold_2
events.append(event2)
contactMatrices.append( CH + 0.1*CW + 0.5*CS ) # school opens, with social distancing
```

## 5. Simulate protocol and analyse results

#### Initialise pyross.control, run and plot a single test simulation

```python
# duration of simulation
Tf=150;  Nf=Tf+1;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa,
              'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.control.SEAI5R(parameters, M, Ni.copy())

# run model once
method='gillespie'
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
```

```python
# Plot result

t = data['t'];

# get total population in classes
# - symptomatic infective,
# - hospitalised,
# - in ICU
y_plot = np.sum ( data['X'][:,4*M:7*M], axis = -1)

lw=2
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.plot(t,y_plot,lw=lw,)
ax.axvline(data['events_occured'][0][0],
           color='crimson',lw=lw,
          label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=fs,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Run 150 simulations with protocol, then

* plot trajectories,
* plot mean trajectory + standard deviation,
* plot distribution of delay between interventions (lockdown, re-opening of schools) until subsequent maximum/minimum.

```python
# Run 150 simulations with this protocol
N_simulations = 150

data_results = np.zeros([N_simulations,Nf,M*9],
                        dtype=int)
for i in range(N_simulations):
    print('Running simulation {0} of {1}'.format(i+1,N_simulations),end='\r')
    data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
    data_results[i] = data['X']
```

    Running simulation 150 of 150

```python
# Plot result

t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
for i,e in enumerate(data_results):
    # get total population in classes
    # - symptomatic infective,
    # - hospitalised,
    # - in ICU
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    ax.plot(t,y_plot,lw=0.5,)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=fs,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Mean and standard deviation

```python
# For all simulations, plot total population in classes
# - symptomatic infective,
# - hospitalised,
# - in ICU

# mean and standard deviation of trajectories at every time
y_mean = np.mean( ( np.sum(data_results[:,:,4*M:7*M], axis = -1) ), axis=0)
y_std = np.std ( np.sum( data_results[:,:,4*M:7*M], axis = -1), axis =0)

t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.fill_between(t,y_mean - y_std,y_mean+y_std,
                label='Standard deviation',
           color='grey',alpha=0.4)
ax.plot(t,y_mean,
        label='Mean',
           color='black',alpha=1)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='upper left',fontsize=17,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Plot distributions for both

* time from beginning of lockdown to subsequent peak, and
* time from opening of schools to subsequent minimum.

```python
# calculate time from lockdown to local maximum for each trajectory
mask = (t > lockdown_threshold_0)*(t < lockdown_threshold_2)
t_mask = t[(mask)]
durations_from_lockdown_to_local_maximum = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmax( y_plot[(mask)])
    durations_from_lockdown_to_local_maximum[i] = t_mask[index] - lockdown_threshold_0
print('Mean time from beginning of lockdown to subsequent peak = {0:3.1f} days'.format(np.mean(durations_from_lockdown_to_local_maximum)))

hist, bin_edges = np.histogram(durations_from_lockdown_to_local_maximum,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_lockdown_to_local_maximum),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from beginning of lockdown until subsequent peak [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()

# calculate time from opening of schools to local minimum for each trajectory
# (recall lockdown_threshold_2 = time at which schools are opened)
mask = (t > lockdown_threshold_2)*(t < Tf)
t_mask = t[(mask)]
durations_from_school_opening_to_local_minimum = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmin( y_plot[(mask)])
    durations_from_school_opening_to_local_minimum[i] = t_mask[index] - lockdown_threshold_2
print('Mean time from opening of schools to subsequent minimum = {0:3.1f} days'.format(np.mean(durations_from_school_opening_to_local_minimum)))

hist, bin_edges = np.histogram(durations_from_school_opening_to_local_minimum,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_school_opening_to_local_minimum),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from school re-opening until subsequent minimum [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()
```

## 6. Redo simulations and analysis of point 5, but with more social distancing at school

```python
# we modify only the last contact matrix of the previous protocol:
contactMatrices[-1] =  CH + 0.1*CW + 0.2*CS
```

```python
# Run 150 simulations with this protocol
N_simulations = 150

data_results2 = np.zeros([N_simulations,Nf,M*9],
                        dtype=int)
for i in range(N_simulations):
    print('Running simulation {0} of {1}'.format(i+1,N_simulations),end='\r')
    data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
    data_results2[i] = data['X']
```

    Running simulation 150 of 150

```python
# Plot results

print("Simulated trajectories:")
t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
for i,e in enumerate(data_results2):
    # get total population in classes
    # - symptomatic infective,
    # - hospitalised,
    # - in ICU
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    ax.plot(t,y_plot,lw=0.5,)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=15,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()

print("Mean trajectory and standard deviation:")
# mean and standard deviation of trajectories at every time
y_mean = np.mean( ( np.sum(data_results2[:,:,4*M:7*M], axis = -1) ), axis=0)
y_std = np.std ( np.sum( data_results2[:,:,4*M:7*M], axis = -1), axis =0)
t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.fill_between(t,y_mean - y_std,y_mean+y_std,
                label='Standard deviation',
           color='grey',alpha=0.4)
ax.plot(t,y_mean,
        label='Mean',
           color='black',alpha=1)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='upper right',fontsize=15,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()

print("Time from lockdown to local maximum:")
# calculate time from lockdown to local maximum for each trajectory
mask = (t > lockdown_threshold_0)*(t < lockdown_threshold_2)
t_mask = t[(mask)]
durations_from_lockdown_to_local_maximum2 = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results2):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmax( y_plot[(mask)])
    durations_from_lockdown_to_local_maximum2[i] = t_mask[index] - lockdown_threshold_0
print('Mean time from beginning of lockdown to subsequent peak = {0:3.1f} days'.format(np.mean(durations_from_lockdown_to_local_maximum2)))

hist, bin_edges = np.histogram(durations_from_lockdown_to_local_maximum2,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_lockdown_to_local_maximum2),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from beginning of lockdown until subsequent peak [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()
```

```python

```

```python

```

examples/control/notebooks-to-be-updated/ex05 - SEAI5R - Denmark.ipynb


# Representational Similarity Analysis
This week's tutorial is about Representational Similarity Analysis (RSA)! We'll be looking at how to transform patterns into RDMs using various distance measures, how to test the relation between "feature RDMs" and "brain RDMs", and take a look at exploratory RDM visualization using multidimensional-scaling (MDS).

**What you'll learn**: At the end of this tutorial, you ...

* know you to preprocess patterns for RSA
* understand the concept of an RDM and how it can be computed
* can list different types of candidate RDMs and their differences
* are able to test the association between candidate and neural RDMs

**Estimated time needed to complete**: 8-12 hours

```python
# Some imports for the rest of the tutorial
import os
import numpy as np
import nibabel as nib
import seaborn as sns
import matplotlib.pyplot as plt
from glob import glob
from nilearn import image, datasets, plotting, masking
import pandas as pd
```

## Loading in the data
In this notebook, we are going to work with real data straightaway! Like in the previous decoding tutorial, we'll work only with the data from a single run, though. In addition, to not use too much RAM, we'll analyze only the data from an specific ROI. In contrast to last week, we are going to use a "functional ROI" based on the localizer data from our "flocBLOCKED" task. To derive a functional ROI, we already computed (for each subject) multiple contrasts and the associated whole-brain $z$-score maps in both subject "native" space (*T1w*) and standard space (*MNI152NLin2009cAsym*):

```python
import os
data_dir = os.path.join(os.path.expanduser('~'), 'NI-edu-data')

print("Downloading ROIs for all subjects (+- ... MB) ...")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/floc/*"
print("\nDone!")
```

```python
floc_dir = os.path.join(data_dir, 'derivatives', 'floc', 'sub-03', 'rois')
print("We have the following maps:\n-", '\n- '.join(sorted(os.listdir(floc_dir))))
```

These maps have been computed using a "condition > other conditions". We are, of course, going to use the "face > (place, character, body, object)" map for our ROI and use the data in MNI space:

```python
floc_roi = os.path.join(floc_dir, 'sub-03_task-flocBLOCKED_space-MNI152NLin2009cAsym_desc-face_zscore.nii.gz')

# Let's plot the unthresholded map as well
plotting.plot_stat_map(floc_roi, cut_coords=(40, -46, -20))
```

As you can see in the plot above, this subject shows a strong response to faces (relative to other conditions) in right temporal lobe, just where you'd expect to find the fusiform face area (FFA). But to derive an ROI from this map, we should somehow binarize this image. While this choice is somewhat arbitrary, let's threshold this map at $z > 3$:

```python
floc_roi_bin = image.math_img('(img > 3).astype(int)', img = floc_roi)
plotting.plot_roi(floc_roi_bin, cut_coords=(40, -46, -20));
```

The current ROI, however, contains *a lot* of voxels, also many far outside the location of where we'd expect the FFA. One "trick" we can use to further restrict the number of voxels is to constrain our functional ROI to a particular anatomical location. Here, we'll pick the right temporal ocipital fusiform (rTOF):

```python
ho_atlas = datasets.fetch_atlas_harvard_oxford('cort-maxprob-thr25-2mm', symmetric_split=True)
ho_map = ho_atlas['maps']
rTOF_idx = ho_atlas['labels'].index('Right Temporal Occipital Fusiform Cortex')
rTOF_roi = nib.Nifti1Image((ho_map.get_fdata() == rTOF_idx).astype(int), ho_map.affine)
plotting.plot_roi(rTOF_roi, cut_coords=(40, -46, -20));
```

Note that the anatomical ROI has a slightly higher spatial resolution (2 mm$^2$, instead of $2.7 \times 2.7 \times 2.97$). As such, we need to resample the anatomical mask to our functional resolution:

```python
rTOF_roi_resamp = image.resample_to_img(rTOF_roi, floc_roi_bin, interpolation='nearest')
```

Finally, we can intersect the two masks using the `intersect_masks` from the Nilearn `masking` module:

```python
# Setting the threshold to 1 means: only select voxels that are in *both* masks
ffa_mask = masking.intersect_masks((floc_roi_bin, rTOF_roi_resamp), threshold=1)
plotting.plot_roi(ffa_mask, cut_coords=(40, -46, -20));
```

Alright, now let's download the patterns from sub-03, run 1 (if not downloaded already):

```python
# Download the patterns (if not done already)
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/pattern_estimation/sub-03/ses-1/patterns/*"
```

... and load them in:

```python
patterns_dir = os.path.join(data_dir, 'derivatives', 'pattern_estimation', 'sub-03', 'ses-1', 'patterns')
betas_path = os.path.join(patterns_dir, 'sub-03_ses-1_task-face_run-1_space-MNI152NLin2009cAsym_desc-trial_beta.nii.gz')

# Load 4D array and immediately mask it
R = masking.apply_mask(betas_path, ffa_mask)
print("Shape of R:", R.shape)

plt.imshow(R, aspect='auto')
plt.xlabel("Voxels")
plt.ylabel("Samples (trials)")
plt.title(r"$\mathbf{R}$", fontsize=20)
plt.colorbar()
plt.show()
```

Lastly, we need some experimental variable(s) to relate to this brain pattern. For convenience, we included the events file in the same directory as the estimated (variance of the) patterns:

```python
import pandas as pd
events_file = os.path.join(patterns_dir, 'sub-03_ses-1_task-face_run-1_events.tsv')
events_df = pd.read_csv(events_file, sep='\t')

# Let's remove the rating/response events
# The .query method is great for filtering!
events_df = events_df.query("trial_type != 'rating' and trial_type != 'response'")
events_df
```

There are many different experimental features that we'd could use for our analysis, but for now, we'll stick with a single categorical (binary) one: face sex ("male" or "female").

```python
S = events_df['face_sex'].to_numpy()
print(S)
```

    ToDo (1 point): For our analysis, we'll need the labels ($S$) in numeric format. Convert the string labels ("male", "female") to a numeric format (male: 1, female: 0) and store the result in a variable called <tt>S_num</tt> (which should be a numpy array).

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_str2num
test_str2num(S, S_num)
```

## Preprocessing
In terms of preprocessing, there are a couple of things that you need to keep in mind when planning to use representational similarity analyses. First, while standardization (ensuring zero mean and unit standard deviation for each brain feature) is a common preprocessing step in decoding analyses, it is somewhat of a controversial for RSA (see e.g. [this article](https://www.frontiersin.org/articles/10.3389/fnins.2013.00174/full)). As such, we are not going to apply standardization to our data.

### Multivariate noise normalization
In week 1, we discussed univariate noise normalization, i.e., dividing each brain feature's activity estimate ($\hat{\beta}$) by the standard deviation of the noise ($\hat{\sigma}$), which allows you to "downweigh" noisy voxels. Specifically for RSA, some people use *multivariate* noise normalization, which additionally incorporates the noise *covariance* between voxels. Like the temporal "uncorrelation" method we discussed in week 1, multivariate noise normalization effectively uncorrelates the data, yet this time in the *spatial* dimension. One often-cited reason for multivariate noise normalization in representational similarity analyses is that some distance metrics (discussed later) assume that the brain feature (e.g., voxels) of the data are independent.

So, before going on, let's first load in the residuals from run 1:

```python
# First, let's download them!
print("Downloading residuals for ses-1, run-1, sub-03 (+- ... MB) ...")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/pattern_estimation/sub-03/ses-1/model/*task-face*run-1*residuals.nii.gz"
print("\nDone!")
```

```python
model_dir = patterns_dir.replace('patterns', 'model')
resids_path = os.path.join(model_dir, 'sub-03_ses-1_task-face_run-1_space-MNI152NLin2009cAsym_desc-model_residuals.nii.gz')
# We immediately apply the FFA mask to the residuals
resids = masking.apply_mask(resids_path, ffa_mask)
print("Shape of masked residuals (TxK):", resids.shape)
```

In week 1, we computed the voxelwise noise standard deviation by simply called the numpy `std` function (or method) on our time axis:

```python
noise_std = np.std(resids, axis=0)

plt.figure(figsize=(12, 3))
plt.plot(noise_std)
plt.xlabel("Voxel", fontsize=15)
plt.ylabel(r'$\hat{\sigma}$', fontsize=20)
plt.xlim(0, noise_std.size)
sns.despine()
plt.show()
```

However, we can also get the voxelwise noise standard deviation by computing the noise variance-covariance matrix and extracting the (square root of the) diagonal, because the diagonal represents the variance of the voxels (the "covariance with itself", so to say).

Let's first compute the covariance matrix:

```python
# We need to transpose (.T) the residuals,
# otherwise we'd get a TxT covariance matrix
noise_cov = np.cov(resids.T, bias=True)
plt.imshow(noise_cov)
plt.title("Noise covariance matrix", fontsize=18)
plt.xlabel('Voxels', fontsize=15)
plt.ylabel('Voxels', fontsize=15)
plt.colorbar()
plt.show()
```

Just to convince you that the (square root of the) diagonal is the same as the standard deviation we computed earlier:

```python
noise_std_from_cov = np.sqrt(np.diag(noise_cov))
fig, axes = plt.subplots(ncols=2, figsize=(15, 4))
axes[0].plot(noise_std)
axes[1].plot(noise_std_from_cov, c='tab:orange')
axes[0].set_xlabel("Voxel", fontsize=15)
axes[1].set_xlabel("Voxel", fontsize=15)
axes[0].set_ylabel(r'$\hat{\sigma}$', fontsize=20)
fig.tight_layout()
sns.despine()
fig.show()
```

The reason we need the variance-covariance matrix is that for *multivariate* noise normalization, we use the *full* matrix, i.e., including the off-diagonal elements (the covariance between voxels). To do so, we first need to compute the *whitening* matrix, which is often denoted by $D$ and is computed by taking the square root of the inverse of the estimated variance-covariance matrix ($\hat{\Sigma}$):

\begin{align}
D = \hat{\Sigma}^{-\frac{1}{2}}
\end{align}

Whitening using this particular whitening matrix ($\hat{\Sigma}^{-\frac{1}{2}}$) is also called [ZCA or Mahalanobis whitening](https://en.wikipedia.org/wiki/Whitening_transformation). To apply this whitening matrix to our patterns ($\mathbf{R}$), we simply take the dot product between the patterns and the whitening matrix:

\begin{align}
R_{\mathrm{mnn}} = RD
\end{align}

where $R_{\mathrm{mnn}}$ is the multivariate noise normalized pattern matrix. Note that this operation is *very* similar to the temporal uncorrelation method, but instead of uncorrelating the trials (i.e., the rows of the pattern matrix), it uncorrelates the brain features (i.e., the columns of the pattern matrix).

Let's first compute the whitening matrix. We can use the matrix square root function `sqrtm` from the `scipy.linalg` package:

```python
# square root of inv = ^(-1/2)
from scipy.linalg import sqrtm
D = sqrtm(np.linalg.inv(noise_cov))
```

And to get the multivariate noise normalized patterns, we compute the dot product:

```python
R_mnn = R @ D
```

Let's visualize the unnormalized, univariate noise normalized, and multivariate noise normalized patterns:

```python
R_unn = R / noise_std

fig, axes = plt.subplots(ncols=3, figsize=(13, 3))
axes[0].imshow(R)
axes[0].set_title("No normalization", fontsize=15)
axes[1].imshow(R_unn)
axes[1].set_title("UNN", fontsize=15)
axes[2].imshow(R_mnn)
axes[2].set_title("MNN", fontsize=15)
for i in range(3):
    axes[i].set_xlabel("Voxels", fontsize=15)
axes[0].set_ylabel("Trials", fontsize=15)
fig.tight_layout()
fig.show()
```

Here, our patterns consist of only 64 voxels. Often, however, you might want to use patterns with (many) more voxels. For multivariate noise normalization, which uses the full $K \times K$ variance-covariance matrix, using more brain features ($K$) than samples ($N$) often lead to a very unstable variance-covariance matrix (or in technical terms, a matrix that is not "positive semidefinite"). One trick that makes the variance-covariance matrix estimation more stable is to apply regularization (sometimes called "shrinkage"). This regularization will "shrink" the matrix more towards the identity matrix ($I$, i.e., a matrix with all zeros except the diagonal, which contains ones) when the ratio between brain features and samples becomes larger.

One such shrinkage method is the "Ledoit-Wolf" covariance estimator. Below, we import this function:

```python
from sklearn.covariance import ledoit_wolf
```

    ToDo (1 point): Read through the documentation of the ledoit_wolf function and then use it to compute the regularized covariance matrix of our pattern matrix and subsequently use this to multivariate noise normalize our pattern matrix. Store the result in a new variable called <tt>R_mnn_reg</tt>.

```python
''' Implement your ToDo here. '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the ToDo above. '''
from niedu.tests.nipa.week_3 import test_ledoit_wolf
test_ledoit_wolf(R, resids, R_mnn_reg)
```

## Neural RDMs
The first step in a representational similarity analysis is to create a "neural representational dissimilarity matrix" (RDM). This matrix is a symmetric $N \times N$ matrix which represents how "dissimilar" patterns of different samples (i.e., rows in your pattern matrix, $\mathbf{R}$). Note that the samples could be trials (e.g., "face 1", "face 2", "face 3", etc.) but could also be conditions (e.g., "faces", "houses", "objects", etc.). In our case, we're focusing on trials.

For example, suppose we have only data from four trials (i.e., four rows in our pattern matrix $\mathbf{R}$). A corresponding $4\times 4$ RDM could like the following:

```python
# some made up RDM
example_rdm = np.array([
    [0, 7.2, 1.2, 5.2],
    [7.2, 0, 3.1, 4.7],
    [1.2, 3.1, 0, 6.5],
    [5.2, 4.7, 6.5, 0]
])

plt.imshow(example_rdm)
plt.yticks(np.arange(4))
plt.xlabel("Trials", fontsize=15)
plt.ylabel("Trials", fontsize=15)
plt.title("Example RDM", fontsize=20)
cbar = plt.colorbar()
cbar.ax.set_ylabel('Dissimilarity', fontsize=15)
plt.show()
```

In this example RDM, each cell represents a particular distance between two patterns. For example, the cell in the bottom left corner represents the dissimilarity between trial 1 and trial 4. Note that the RDM is symmetric, because the dissimilarity between trial 1 and 4 is the same as trial 4 and 1; also, the cells on the diagonal are all zero, as they represent the distance between a particular pattern and itself, which is zero!  Note that these two properties (symmetricity, zero diagonal) are only true when you compute your RDM on trials within the same run (unlike the between-run pattern distances, discussed previously).

For now, we'll stick with within-run RDMs (as they're a little easier to compute).

In a way, you can think about RDMs as "inverse" correlation matrices in which cells do not represent correlations (a kind of *similarity* metric) but distances.

By now, you might ask youself: "but how do you actually compute these dissimilarities?" Well, this depends on what *distance metric* you use! There are many different functions you can use to quantify the dissimilarity between two vectors (i.e., two rows in our pattern matrix). Actually, reflecting the intuition that an RDM is basically the inverse of a correlation matrix, one metric that is sometimes used is the $1-r$ distance\*. This distance simply quantifies distance as the 1 minus the correlation between two patterns. For example, the $1-r$ distance between "pattern A" and "pattern B" is $1-\mathrm{corr(pattern\ A, pattern\ B})$.

---
\* In the RSA literature, some people use the *cosine distance*, which is the angle between two vectors; when the patterns are mean centered (i.e., the rows in $\mathbf{R}$ have a mean of 0), this is exactly the same as the $1-r$ distance!

    ToDo (1 point): For our data (the variable <tt>R</tt>), compute the RDM using the $1 -r$ distance and store it in a variable named <tt>rdm_1minr</tt>. Note: no need for a for-loop! (You might want to check out the <tt>np.corrcoef</tt> function.)

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_1minr
test_1minr(R, rdm_1minr)
```

Another often-used distance metric used for RDMs, and perhaps the most intuitive one, is the Euclidean distance. This distance is computed as the square root of the sum of squared distances between two patterns (e.g., $p$ and $q$) consisting of $K$ elements:

\begin{align}
\delta_{euclidean} = \sqrt{\sum_{j=1}^{K}{(p_{j} - q_{j})^{2}}}
\end{align}

Below, we define two example patterns with four features and compute the Euclidean distance between them:

```python
p = np.array([1, 3, 8, -4])
q = np.array([5, -3, 2, 1])

# No need for a for loop!
euc_dist = np.sqrt(np.sum((p - q)**2))
print("Euclidean distance between p and q:", euc_dist)
```

    ToDo (2 points): While all pairwise distances were easy to calculate using the $1-r$ distance, it takes a little more code to do this using the Euclidean distance. Compute the RDM based on the Euclidean distance for our data (the variable <tt>R</tt>) and store this in a variable named <tt>rdm_euc</tt>. Also visualize the RDM using the pyplot <tt>imshow</tt> function. Unless you're a linear algebra wizard, you need to use for loops to compute the RDM. Do not use any external functions (beyond numpy). Hint: pre-allocate your RDM first and then fill it cell by cell in a nested for loop.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from sklearn.metrics import euclidean_distances
np.testing.assert_array_almost_equal(euclidean_distances(R), rdm_euc, decimal=3)
print("Well done!")
```

As you might have seen in the test cell, scikit-learn actually provides several functions to quickly compute distance matrices (RDMs) using various distance metrics. We recommend using the generic `pairwise_distances` function:

```python
from sklearn.metrics import pairwise_distances
```

To compute an $N\times N$ distance matrix from a $N\times K$ pattern array ($\mathbf{R}$), you can use it as follows:
```python
rdm = pairwise_distances(R, metric='name_of_metric')
```

For example, to compute an RDM based on the "cosine" distance (which is similar to the $1-r$ distance), you can run:

```python
rdm_cosine = pairwise_distances(R, metric='cosine')

plt.imshow(rdm_cosine)
plt.xlabel("Trials", fontsize=15)
plt.ylabel("Trials", fontsize=15)
plt.title("Cosine-based RDM", fontsize=20)
cbar = plt.colorbar()
cbar.ax.set_ylabel('Cosine distance', fontsize=15)
plt.show()
```

Note that there is no agreed-upon "best" distance metric! If you want to know more about the different (dis)similarity metrics for pattern analyses, check out [this article](https://link.springer.com/article/10.1007/s42113-019-00068-5).

For the next couple of sections, we'll use the Euclidean distance-based RDM:

```python
rdm_R = pairwise_distances(R, metric='euclidean')
```

    ToThink (1 point): Euclidean and $1-r$/cosine distances differ in one major aspect related to the "type" of information they can encode/pick up. What do you think this is?

YOUR ANSWER HERE

## Exploratory analysis using MDS
Most applications of RSA involve relating *neural* RDMs with RDMs based on experimental features (which we'll discuss in the next section). However, you can also do exploratory analyses on your neural RDM only! This is usually done by investigating the (dis)similarity structure (or "representational geometry" in RSA terms), usually in a 2D or 3D space.

As pattern analyses are often applied to very high-dimensional data (i.e., patterns with many brain features, $K$), people often project the data (i.e., the patterns) into a lower dimensional space. We already encountered one such method in week 1: PCA! However, when interested in the (dis)similarity structure of your data, *multidimensional scaling* (MDS) is more appropriate. Just like PCA, this technique aims to create combinations of features into a lower-dimensional subset of components, such that the high-dimensional distances are presented as much as possible in the lower-dimensional space. For example, if the distance between A and B is 436 in high-dimensional space (e.g., $K=500$), MDS tries to create a lower-dimensional space (usually 2 components) in which the distance between A and B is as close as possible to 436 (as well as all other distances between patterns).

Of course, scikit-learn contains an implementation of MDS that uses the familiar `fit`/`transform` methods. Importantly, it can take in a $N\times K$ matrix (like our pattern matrix $\mathbf{R}$) and compute the high-dimensional distance structure (i.e., the RDM) internally or you can give it your precomputed RDM. In the latter case, you need to initialize it with `dissimilarity='prepcomputed'`, which is what we're going to do:

```python
from sklearn.manifold import MDS
mds = MDS(dissimilarity='precomputed', n_components=2)
```

Then, we call the `fit_transform` method to compute lower-dimensional ($K=2$) representation of the data (which is, again, an $N \times K$ array, but this time, $K=2$!):

```python
mds_R = mds.fit_transform(rdm_R)
print("Shape of mds_R:", mds_R.shape)
plt.figure(figsize=(8, 5))
plt.grid()
plt.scatter(mds_R[:, 0], mds_R[:, 1])
plt.xlabel('MDS component 1', fontsize=20)
plt.ylabel('MDS component 2', fontsize=20)
plt.show()
```

    ToDo (1 point): Below, re-plot the MDS scatterplot, but this time, color the datapoints (i.e., the trial patterns) according to their trial onset (you can set the color of the points with <tt>c</tt> parameter in the <tt>scatter</tt> function), which you can extract from the <tt>events_df</tt> dataframe. This way, datapoints (i.e., patterns) with a similar onset have a similar hue.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

    ToThink (1 point): Which phenomenon (that we discussed before) is clearly visible in this low-dimensional embedding of the data?

YOUR ANSWER HERE

Instead of coloring the datapoints according to some experimental property/feature (such as onset), an even more potent way to visualize MDS-embeddings is to plot the actual images that correspond with the trial patterns (here: the faces shown to the subject)! This way, you may find patterns in the data that you might not have thought of!

However, in our opinion, the true strength of RSA lies in their ability to test hypotheses about complex representational structures using experimental features, which is discussed next.

## Categorical RDMs
In most pattern analyses, we'd like to evaluate the (possibly) association between experimental features and brain patterns. In RSA, this is done by comparing neural RDMs (discussed in the previous sections) with RDMs based on experimental features. These experimental feature RDMs (let's call them "feature RDMs") are constructed in largely the same way as neural RDMs: by computing the pairwise distance between samples!

The experimental features ($P$) that you use for your feature RDM of course depend on your hypothesis! Importantly, unlike decoding models, RSA naturally handles high-dimensional feature spaces very well. Like neural RDMs, no matter how many features you use, you'll always analyze the resulting $N\times N$ RDM!

After constructing your feature RDM, you can test whether the "geometry" of your hypothesized feature space matches the geometry of your brain patterns. In other words, you test whether the pattern of distances is similar in your brain data and your experimental features. Technically, you can use *any* (set of) feature(s) that you believe match the geometry of the corresponding brain patterns. How to actually test this will be discussed in section 5.

In this section, we'll focus on the most straightforward type of feature RDM: the categorical RDM. This RDM, basically, investigates whether patterns belonging to the same condition are more similar than patterns belonging to a different condition (note the similarity to decoding models). For example, we could hypothesize that the FFA represents face gender, which should accordingly lead to relatively small neural distances between images of the same face gender and relatively large neural distances between images of a different face gender. Before delving into how we should construct a corresponding feature RDM, let's first define our experimental feature: face gender. In the first section, we already extracted this from the events file. Now, let's convert it to a numeric format:

```python
from sklearn.preprocessing import LabelEncoder
face_gen = LabelEncoder().fit_transform(S)
print("Face gender, numeric:", face_gen)
```

Now, to capture this feature into an RDM, we can construct an RDM with zeros in cells corresponding to trials with the same condition (both male or both female faces) and ones everywhere else, which capture the hypothesis that trials should have a smaller distance when they are of the same condition (0) than when they are of a different condition (1).

    ToDo (1 point): Create this face-gender RDM and store it in a variable named <tt>rdm_fg</tt>, an $N \times N$ numpy array. Do not use any external functions for this.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_rdm_fg
test_rdm_fg(face_gen, rdm_fg)
```

Technically, to create this categorical RDM (with only two levels), you can also use the `pairwise_distances` function with the "manhattan" metric (the sum of *absolute* distances):

```python
# Note the np.newaxis, which is needed because pairwise_distances
# assumes that the input is 2D
rdm_fg_pd = pairwise_distances(face_gen[:, np.newaxis], metric='manhattan')
plt.imshow(rdm_fg_pd)
plt.colorbar()
plt.title("Categorical RDM")
plt.show()
```

## Continuous/computational RDMs
The categorical RDMs discussed in the previous section are the most simple implementation of feature RDMs, based on only a single categorical feature. RSA really shines, though, when relating more complex feature sets (sometimes called "feature spaces") consisting of multiple (continuous) variables.

For example, for our data, we have subject-specific ratings of dominance, trustworthiness, and attractiveness for all of the faces shown to (the same) subjects. If we'd want to investigate whether a particular brain region represents these face properties (you might call them "[social judgements](https://www.sciencedirect.com/science/article/abs/pii/S0959438813000147)), we could create a feature RDM base on these three features; in fact, we'll do that below:

```python
sj = events_df.loc[:, ['subject_dominance', 'subject_trustworthiness', 'subject_attractiveness']].to_numpy()
rdm_sj = pairwise_distances(sj, metric='euclidean')

plt.imshow(rdm_sj)
plt.title("Social judgements RDM", fontsize=15)
plt.show()
```

Note though, that all information about the individual features (attractiveness, trustworthiness, and dominance) is lost in this RDM! The feature RDM should represent the geometry of the entire feature space, not the effects of the individual features on brain activity (although we discuss a technique that allows for this type of inference in the next section).

    ToDo (2 point): Note that as long as you can formalize your hypothesis into a feature matrix ($N \times P$), you can create an RDM from it (and test it with RSA)! Suppose that I believe that social judgements are not represented in a bipolar fashion (from very unattractive to very attractive, very untrustworthy to very trustworthy, etc.; like we assumed before) but in an unipolar fashion relative from the mean rating. For example, this would mean that a face with an attractiveness/trusworthiness/dominance rating of -4 would be represented in the same way as a face with a rating of 4 (assuming a mean rating of 0) Moreover, suppose I believe that this representation is quadratic, not linear. For example, a face with an attractiveness rating of 2 is four times as attractive as a face with an attractiveness rating of 1. Using the social judgement data (i.e., the <tt>sj</tt> variable), create an RDM that represents this hypothesis.

```python
''' Implement the ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_rdm_sj2
test_rdm_sj2(sj, rdm_sj2)
```

RSA also lends itself very well for testing computational models, i.e., models that yield a (set of) feature(s) that were directly computed from the data. For example, in vision science, there are many computational models that yield a set of (visual) feature that are computed from the image directly (i.e., from the pixels). Again, as long as you can specify a set of features that embody your hypothesis, you can create an RDM from it!

In the next ToDo, you're going to practice a bit to get into this "computational mindset" by applying a very simple (and theoretically meaningless) computational model to the face stimuli. In the current directory there is a subfolder `stim`, which contains all the stimuli from the first run:

```python
imgs = sorted(glob(os.path.join('stims', '*.jpg')))
print(imgs)
```

To load an image as a numpy array, we can use the `imageio` library. Note that the function returns a 3D numpy array, where the first two dimensions represent width and height, and the third dimension represents the three color channels (red, green, and blue).

```python
import imageio
example_stim = imageio.imread(imgs[0])
print("Shape of image data:", example_stim.shape)
```

    ToDo (2 points): Suppose that I have a very naive theory about color representation in the brain. Specifically, suppose that I think that a particular brain area simply represents stimuli according to their average redness, greenness, and blueness. In other words, I believe that stimuli with similar average red/green/blue values have similar brain patterns. For the forty stimuli from run 1, construct an RDM that represents this hypothesis and store it in a variable named <tt>rdm_rgb</tt> (a $40\times 40$ array). Use a Euclidean distance metric. Hint: before constructing your RDM, your feature matrix should be of shape $40 \times 3$.

```python
''' Implement your ToDo here. '''

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_rdm_rgb
test_rdm_rgb(imgs, rdm_rgb)
```

Just by eye, it's difficult to judge whether the neural RDM contains a similar representational geometry as the feature RDMs we created earlier. Fortunately, we have statistics!

## Testing RDMs
In this section, we'll discuss how to evaluate the "fit" of feature RDMs.

### Correlation-based tests
Alright, so now we got two RDMs: the feature RDM and the neural RDM. To evaluate to what extent the two RDMs share the same representational geometry, we can simply correlate them! Before doing so, we have to do one more thing: extract the lower (or upper) triangle of the RDM. This is because RDMs are symmetric: the values above and below the diagonal are exactly the same. If we used the entire (flattened) RDM, we'd "artifically" create twice as many datapoints (i.e. the pairwise dissimilarities) than there really are, which will inflate the significance of the correlation between the RDMs because of increased sample size. So, instead of using all $N\cdot N$ pairwise differences from the RDM, we need to extract only the flattened $N\times (N-1)/2$ pairwise dissimilarity values, the "representational dissimilarity vector" (RDV) if you will. This means that we do not include the diagonal!

Fortunately, there is a function that easily extracts the lower triangle of a square distance matrix: `squareform` (from the `scipy.spatial.distance` module):

```python
from scipy.spatial.distance import squareform

# Let's extract the RDV from our neural RDM
rdv_R = squareform(rdm_R.round(5))
print("Shape rdv_R:", rdv_R.shape)
```

    Tip: Sometimes, an RDM might not be exactly symmetric due to floating point inaccuracies, which will give an error my trying to extract the lower triangle using <tt>squareform</tt>. To circumvent this, you can round the RDM values to, e.g., 5 decimals using the <tt>.round(decimals)</tt> array method.

As you can see, the shape of the `rdv_R` is as expected: $40 \times (40-1) / 2 = 780$. Let's do the same for the face-gender RDM we created earlier (`rdm_fg_pd`):

```python
rdv_fg = squareform(rdm_fg_pd)
print("Shape rdv_fg:", rdv_fg.shape)
```

Importantly, the correlation between feature and neural RDMs is often evaluated using a rank-based correlation metric. For continuous feature RDMs, this is usually the Spearman correlation, but for categorical feature RDMs (such as our face-gender RDM), often the "Kendall Tau $\alpha$" correlation is used, as it deals properly with tied ranks. Implementations of both correlations are available from the `scipy.stats` module. Here, we'll use Kendall's Tau $\alpha$, because our face-gender RDM is categorical:

```python
from scipy.stats import kendalltau
rdm_corr, pval = kendalltau(rdv_fg, rdv_R)
print("Correlation between RDMs (p-value): %.3f (%.3f)" % (rdm_corr, pval))
```

Given the slighly positive correlation between our face-gender RDM and the FFA RDM, this means that patterns related to trials with the same face gender are slightly less dissimilar than patterns related to trials with a different face gender!

### Reweighting RDVs
One more advanced RSA technique is "reweighting". This technique allows you to use multiple feature RDVs to explain your neural RDV. Essentially, you assume that the neural RDV can be approximated as a linear weighted sum of different feature RDVs. For example, for two feature RDVs ($\mathrm{RDV}_{S_{1}}$ and $\mathrm{RDV}_{S_{2}}$):

\begin{align}
\mathrm{RDV}_{R} = \beta_{0} + \mathrm{RDV}_{S_{1}}\beta_{1} + \mathrm{RDV}_{S_{2}}\beta_{2} + \epsilon
\end{align}

You might recognize this formulation as a linear model (GLM) with the neural RDV as dependent variable and the feature RDVs as independent variables. Here, the parameters ($\beta$) represent the "reweighting" factors. This technique is very useful to disentangle the contributions of different (possibly correlated) feature spaces. Note that, often, a variant of ordinary least squares (OLS) is used to determine the parameters: non-negative least squares (NNLS), which forces the parameters to be positive (for details about why NNLS should be used, see [this article](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003915), and more information about reweighting in general, see [this article](https://www.sciencedirect.com/science/article/pii/S0028393215301998#bib5)).

After the reweighted RDV ($\mathrm{RDV}_{S}\hat{\beta}$) is computed, it can again be evaluated using a (rank-based) correlation: $r(\mathrm{RDV}_{S}\hat{\beta}, \mathrm{RDV}_{R})$.

    ToDo (2 points): Previously, we defined two different feature RDMs: one based on face-gender (<tt>rdm_fg</tt>) and another one based on social judgements (<tt>rdm_sj</tt>). While theoretically meaningless, perform a reweighting analysis with the two feature RDVs as independent variables (in addition to an intercept!) and the neural RDV (based on <tt>rdm_R</tt>) as dependent variable. You can use the <tt>nnls</tt> implementation from <tt>scipy.optimize</tt> to perform NNLS. Note that the <tt>nnls</tt> function returns two things — the first object is the array with parameters, and it takes two arguments: the design matrix (independent variables) and the dependent variable. Compute the Spearman correlation between the reweighted RDV and the neural RDV and store the result in a variable named <tt>corr_reweighted_analysis</tt>.

```python
from scipy.stats import spearmanr
from scipy.optimize import nnls

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nipa.week_3 import test_reweighting_todo
test_reweighting_todo(rdm_fg, rdm_sj, rdm_R, corr_reweighted_analysis)
```

    ToThink (2 points): Suppose that you have a very high-dimensional experimental feature space (e.g., $P=10,000$) and you decide, instead of using a single feature RDM, to use an individual RDM for each feature and analyze the neural RDM in with a reweighted analysis. Explain why, while this would guarantee amazing results, this is probably not a good idea and propose a practical solution.

YOUR ANSWER HERE

### Group-level analyses
So far, we only analyzed a single subject. Often, you might want to analyze multiple subjects and perform inference on the group level. Just like we did with single-subject decoding model performance scores, we can test whether results are significant by testing them against chance-level (i.e., 0 when using correlations). For example, suppose I have the following subject-specific RDV correlations:

```python
rdv_corrs = np.array([0.03, 0.01, 0.02, 0.00, 0.05, 0.07, 0.05, 0.12, 0.08, -0.07, -0.02, -0.01])
```

Just like we evaluated single-subject RDV correlations using a non-parameteric rank-based correlation (Spearman or Kendall's Tau), it is common to use a non-parametric test for group-level analyses: the Wilcoxon signed-rank test:

```python
from scipy.stats import wilcoxon
stat, pval = wilcoxon(rdv_corrs)
print("P-value:", pval)
```

    ToDo (4 points): In this ToDo, you'll implement an RSA analysis from start to end (on 10 subjects) in which you investigate whether facial expression (smiling vs. neutral) is represented in the right temporal occipital fusiform. In the end, you should have 10 RDV correlations (stored in the pre-allocated <tt>rdv_corrs</tt> variable). We have written the start of the for-loop for you already. Within the loop, do the following:

* Load in the events file as a DataFrame, extract the facial expression information (column: "expression"), convert it to a numeric format, and compute the corresponding feature RDM;
* Load in the patterns and apply the previously defined rTOF mask (the variable <tt>rTOF_roi_resamp</tt>);
* Compute the Kendall Tau correlation between the feature RDV and neural RDV and store this in the <tt>rdv_corrs</tt> array;
* After the loop, compute the associated group-level p-value using the Wilcoxon signed-rank test and store this in a variable named <tt>pval</tt>.

```python
print("Downloading patterns for all subjects (+- 820 MB) ...")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/pattern_estimation/sub*/ses-1/*task-face*run-1*space-MNI*"
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/pattern_estimation/sub*/ses-1/*task-face*run-1*.tsv"
print("\nDone!")
```

```python
''' Implement the ToDo here. '''

# We'll only analyze the first 10 subjects
subs = sorted([os.path.basename(d) for d in
               glob(os.path.join(data_dir, 'derivatives', 'pattern_estimation', 'sub-*'))])
subs = subs[:10]

# Pre-allocate the correlation vector
rdv_corrs = np.zeros(len(subs))

pattern_est_dir = os.path.join(data_dir, 'derivatives', 'pattern_estimation')

# Iterate over subjects (i = 0:9)
for i, sub in enumerate(subs):
    print("Processing %s ..." % sub)
    events_file = os.path.join(pattern_est_dir, sub, 'ses-1', 'patterns', sub + '_ses-1_task-face_run-1_events.tsv')
    patterns = os.path.join(pattern_est_dir, sub, 'ses-1', 'patterns', sub + '_ses-1_task-face_run-1_space-MNI152NLin2009cAsym_desc-trial_beta.nii.gz')
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
''' Tests the above ToDo (part 1). '''
np.testing.assert_array_almost_equal(
    np.array([ 0.01328101, -0.01517032, -0.01950736,  0.01904201, -0.0036204 ,
              -0.01989825,  0.03360738,  0.02309053, -0.00340634,  0.01325309]),
    rdv_corrs,
    decimal=4
)
print("Well done!")
```

```python
''' Tests the above ToDo (part 1). '''
np.testing.assert_almost_equal(pval, 0.6953125, decimal=4)
print("Well done!")
```

This notebook discussed the basics of representational similarity analysis. There are many more topics that we didn't discuss (such as cross-validated distance metrics, noise ceilings, and comparing different feature RDMs).

For a more in-depth technical overview of RSA, we recommend the following article:
* [Nili, H., Wingfield, C., Walther, A., Su, L., Marslen-Wilson, W., & Kriegeskorte, N. (2014). A toolbox for representational similarity analysis. PLoS computational biology, 10(4).](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003553)

(Note that the paper introduces a MATLAB toolbox for RSA; while very much in development, the authors recently rewrote the toolbox as a [Python package](https://github.com/rsagroup/pyrsa)).