# Data Analysis with Jupyter Notebooks.

# Tutorial 5

Benjamin J. Morgan, University of Bath.

# Contents

- [Data analysis and statistics with numpy](#data_analysis)
- [Linear regression](#linear_regression)

# Data analysis and statistics with numpy<a id='data_analysis'></a>

`numpy` contains a lot of powerful functions for performing simple statistical analysis on our data. For example, consider the set of numbers 1 to 50:

>```python
import numpy as np
a = np.arange(1,51)
a
```


```python

```

To find the minimum and maximum values we can use `np.min()` and `np.max()`

>```python
np.min(a)
```


```python

```

>```python
np.max(a)
```


```python

```

To find the **sum** of all these numbers, we can use `np.sum()`

>```python
np.sum(a)
```


```python

```

The **mean** of a set of numbers is defined as 

\begin{equation}
\frac{\sum_i^N x_i}{N}
\end{equation}

which we could calculate with

>```python
np.sum(a) / len(a)
# len(a) returns the length of the array `a`
```


```python

```

or with `np.mean()`

>```python
np.mean( a )
```


```python

```

The **standard deviation**, $\sigma$ quantifies how much the numbers in our set deviate from the mean.

\begin{equation}
\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^N(x_i-\mu)^2}
\end{equation}

where $\mu$ is the mean.

Again, we could write this out in code:

>```python
import math
sigma = math.sqrt( np.sum( ( a - np.mean(a))**2 ) / len(a) )
sigma
```


```python

```

Or use the `np.std()` function

>```python
np.std(a)
```


```python

```

### Linear Regression<a id='linear_regression'></a>

Another commonly used data analysis technique is **linear regression**. This is used to calculate the relationship between two data sets, $X$ and $Y$, assuming that this relationship can be described by a straight line

\begin{equation}
y_i = m x_i + c.
\end{equation}

For any real data set, the data points are unlikely to all fall exactly on the same line. Linear regression is the process of calculating the line that &ldquo;best fits&rdquo; the given data.

<div class="alert alert-success">
Look at the following snippet, and try to work out what the result will be.<br/>
<span style='font-family:monospace'>np.random.rand(10)</span> creates an array of 10 random numbers between 0 and 1.
</div>

>```python
import matplotlib.pyplot as plt
x = np.arange(1,11)
offset = 2.0
y = x + ( np.random.rand(10) - 0.5 ) * offset
plt.plot( x, y, 'o' )
plt.show()
```


```python

```

You can see this approximately gives the straight line relationship between $y=x$. We can use linear regression to calculate the &ldquo;best&rdquo; straight line that describes these data.

There are a number of different ways in Python to calculate a line of best-fit. One of the simplest is to use another module, [`scipy.stats`](https://docs.scipy.org/doc/scipy-0.18.1/reference/stats.html). As you might suspect from the name, `scipy.stats` contains an large set of statistical analysis tools. We want [`linregress()`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.stats.linregress.html#scipy.stats.linregress):

>```python
from scipy.stats import linregress
linregress( x, y )
```


```python

```

You can see the output is complicated, but includes a list of values that includes the slope and the intercept. In fact you can treat the output like a [list](lists), and use indexing to select a specific result.

>```python
linregress( x, y )[0] # use indexing to get the slope
```


```python

```

Another option is to collect all five of the output values at once

>```python
slope, intercept, rvalue, pvalue, stderr = linregress( x, y )
print( "slope =", slope )
print( "intercept =", intercept )
```


```python

```

To plot the best-fit line against the original data, we generate a new data set according to $y=mx+c$, where $m$ and $c$ are set to the `slope` and `intercept`, calculated from `linregress`.

>```python
y_fit = slope * x + intercept 
plt.plot( x, y, 'o' )
plt.plot( x, y_fit, '-' )
plt.xlabel( 'x' )
plt.ylabel( 'y' )
plt.title( 'y =', slope, 'x +', intercept )
plt.show()
```


```python

```

bbfa199b0b620eeb2e054ecb90fba99429cd85fe
ipynb
Jupyter Notebook
Tutorial_5.ipynb
pythoninchemistry/chem_data_analysis_jupyter
4af545f1a8acdded28d96508bb5adc8929da92cc
CC-BY-4.0
2019-05-05T00:21:55.000Z
2021-09-16T14:15:15.000Z
Tutorial_5.ipynb
pythoninchemistry/chem_data_analysis_jupyter
4af545f1a8acdded28d96508bb5adc8929da92cc
CC-BY-4.0
Tutorial_5.ipynb
pythoninchemistry/chem_data_analysis_jupyter
4af545f1a8acdded28d96508bb5adc8929da92cc
CC-BY-4.0
Qwen/Qwen-72B
1. YES
2. YES