# Python Data Science Guide

This notebook provides an introduction to the tools and libraries that we use in this GWAS lecture. These are: 

1. [NumPy](http://www.numpy.org/)
2. [h5py](https://www.h5py.org/)
3. [pandas](https://pandas.pydata.org/)
4. [matplotlib](https://matplotlib.org/)

## NumPy

NumPy stands for ‘Numerical Python’ or ‘Numeric Python’. It is an open source module of Python which provides fast mathematical computation on arrays and matrices.

Numpy can be imported into the notebook using


```python
%matplotlib inline
import numpy as np
```

### Creating NumPy array 

NumPy’s main object is the homogeneous multidimensional array. It is a table with same type elements, i.e, integers or string or characters (homogeneous), usually integers. In NumPy, dimensions are called axes. The number of axes is called the rank.

There are several ways to create an array in NumPy like np.array, np.zeros, no.ones, etc. Each of them provides some flexibility.

| Numpy function | What it does ?                                                                                                        |
|----------------|-----------------------------------------------------------------------------------------------------------------------|
| np.array       | Convert array like python object to NumPy array                                                                       |
| np.arange      | Create arrays with regularly incrementing values.                                                                     |
| np.ones        | Create an array filled with 1 values                                                                                  |
| np.zeros       | Create an array filled with 0 values                                                                                  |
| np.full        | Create an array filled with the specified value                                                                       |
| np.random.rand | Generate arrays of random values in the specified shape                                                               |
| np.linspace    | Create arrays with a specified number of elements, and spaced equally between the specified beginning and end values. |


```python

```


```python
# Convert array like python object to NumPy array
np.array([2,3,4])
```




    array([2, 3, 4])




```python
# Create arrays with regularly incrementing values.
np.arange(2, 3, 0.1)  # Number between 2, 3 with gap of 0.1
```




    array([2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9])




```python
# Create an array filled with 0 values
np.zeros((2, 3))
```




    array([[0., 0., 0.],
           [0., 0., 0.]])




```python
# Create an array filled with 1 values
np.ones((2, 3))  # 
```




    array([[1., 1., 1.],
           [1., 1., 1.]])




```python
# Create an array filled with the specified value
np.full( (3,4), 0.11 )  
```




    array([[0.11, 0.11, 0.11, 0.11],
           [0.11, 0.11, 0.11, 0.11],
           [0.11, 0.11, 0.11, 0.11]])




```python
# Generate arrays of random values in the specified shape
np.random.rand(2,3)
```




    array([[0.75061919, 0.22210586, 0.39882487],
           [0.16872907, 0.32425882, 0.44510628]])




```python
# Create arrays with a specified number of elements, and spaced equally between the specified beginning and end values
np.linspace(0, 5/3, 6)
```




    array([0. , 0.2, 0.4, 0.6, 0.8, 1. ])



Some of the important attributes of a NumPy object are:

- **ndim**: displays the dimension of the array
- **shape**: returns a tuple of integers indicating the size of the array
- **size**: returns the total number of elements in the NumPy array
- **dtype**: returns the type of elements in the array, i.e., int64, character
- **itemsize**: returns the size in bytes of each item
- **reshape**: Reshapes the NumPy array


```python
arr = np.ones((2,3))
print arr
print("ndim: %s" % arr.ndim)
print("shape: (%s,%s)" % arr.shape)
print("size: %s" % arr.size)
print("dtype: %s" % arr.dtype)
print("itemsize: %s" % arr.itemsize)
arr.reshape((3,2))
```

    [[1. 1. 1.]
     [1. 1. 1.]]
    ndim: 2
    shape: (2,3)
    size: 6
    dtype: float64
    itemsize: 8





    array([[1., 1.],
           [1., 1.],
           [1., 1.]])



By default NumPy numeric arrays will be created using *float64* type. 
This can be overridden by providing an explicit dtype.
For example to create a numpy array with 32bit integer elements use:


```python
np.ones((2,3), dtype=np.int32)
```




    array([[1, 1, 1],
           [1, 1, 1]], dtype=int32)



### Accessing NumPy array

NumPy array elements can be accessed using indexing. Below are some of the useful examples:

- arr[2:5] will print items 2 to 4. Index in NumPy arrays starts from 0
- arr[2::2] will print items 2 to end skipping 2 items
- arr[::-1] will print the array in the reverse order
- arr[1:] will print from row 1 to end

Accessing a multi-dimensional NumPy array can be done using following form:

```Array[row_start_index:row_end_index, column_start_index: column_end_index]```


```python
arr = a = np.arange(25).reshape(5, 5)
arr
```




    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19],
           [20, 21, 22, 23, 24]])




```python
# access rows 2 to 4 and all columns
arr[2:4]
```




    array([[10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])




```python
# access colum 2 to 4 and all rows
arr[:,2:4]
```




    array([[ 2,  3],
           [ 7,  8],
           [12, 13],
           [17, 18],
           [22, 23]])




```python
# access colum 2 to 4 and row 2 to 4
arr[2:4,2:4]
```




    array([[12, 13],
           [17, 18]])




```python
# access last row 
arr[-1]
```




    array([20, 21, 22, 23, 24])




```python
# access last column 
arr[:,-1]
```




    array([ 4,  9, 14, 19, 24])




```python
# access second row and third column
arr[2,3]
```




    13



NumPy arrays can also be accessed using boolean indexing.


```python
a = np.arange(12).reshape(3, 4)
a
```




    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])




```python
rows_on = np.array([True, False, True])
rows_on
```




    array([ True, False,  True])




```python
a[rows_on,:]
```




    array([[ 0,  1,  2,  3],
           [ 8,  9, 10, 11]])



### NumPy array manipulation 

NumPy arrays are capable of performing all basic operations such as addition, subtraction, element-wise product, matrix dot product, element-wise division, element-wise modulo, element-wise exponents and conditional operations.

An important feature with NumPy arrays is broadcasting:  



```python
# Without broadcasting
a = np.array([0, 1, 2])
b = np.array([5, 5, 5])
a + b
```




    array([5, 6, 7])




```python
# With broadcasting
a + 5
```




    array([5, 6, 7])



NumPy provides basic mathematical and statistical functions like mean, min, max, sum, prod, std, var, summation across different axes, transposing of a matrix, etc.


```python
c = a + b
print("mean: %s" % c.mean())
print("std: %s" % c.std())
print("sum: %s" % c.sum())
print("var: %s" % c.var())
print("min: %s" % c.min())
print("max: %s" % c.max())
```

    mean: 6.0
    std: 0.816496580927726
    sum: 18
    var: 0.6666666666666666
    min: 5
    max: 7


A particular NumPy feature of interest is solving a system of linear equations. NumPy has a function to solve linear equations. 
For example these linear equations 

\begin{align}
2x + 6y & =  6 \\
5x + 3y & =  -9 
\end{align}

can be solved by:


```python
coeffs  = np.array([[2, 6], [5, 3]])
depvars = np.array([6, -9])
solution = np.linalg.solve(coeffs, depvars)
solution
```




    array([-3.,  2.])



---
## HDF5 & h5py

The h5py package is a Pythonic interface to the HDF5 binary data format.

It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want.


```python
# First we download the data
import sys
sys.path.append('./..')
import data as tutorial_data
file_name = tutorial_data.get_file('BYxRM')
```

    file not found, downloading from https://www.ebi.ac.uk/~stegle/limix/tutorials/data/BYxRM/BYxRM.hdf5


An HDF5 file is a container for two kinds of objects: datasets, which are array-like collections of data, and groups, which are folder-like containers that hold datasets and other groups. The most fundamental thing to remember when using h5py is:

**Groups work like dictionaries, and datasets work like NumPy arrays**

Open an existing HDF5 file:


```python
import h5py
f = h5py.File(file_name,'r')
```

**HDF** stands for *Hierarchical Data Format*.   
Every object in an HDF5 file has a name, and they’re arranged in a POSIX-style hierarchy with `/`-separators.  
The “folders” in this system are called groups. The **File** object we created is itself a *group*, in this case the *root* group, named `/`  



What is stored in this file? 
Remember **h5py.File** acts like a Python dictionary, thus we can check the keys,


```python
list(f.keys())
```




    [u'genotype', u'phenotype']



The root object contains two children (*genotype* and *phenotype*).  
To find out what type (dataset or group) of object the *phenotype* is we can run: 


```python
f['phenotype']
```




    <HDF5 group "/phenotype" (3 members)>



It looks like the *phenotype* object itself is a group with 3 children


```python
list(f['phenotype'].keys())
```




    [u'col_header', u'matrix', u'row_header']



An easy way to iterate over all elements of a HDF5 file is to use the visit method:


```python
def printname(name):
    print("%s: %s " % (name, f[name]))
f.visit(printname)
```

    genotype: <HDF5 group "/genotype" (3 members)> 
    genotype/col_header: <HDF5 group "/genotype/col_header" (3 members)> 
    genotype/col_header/alleles: <HDF5 dataset "alleles": shape (11623, 2), type "|S37"> 
    genotype/col_header/chrom: <HDF5 dataset "chrom": shape (11623,), type "<i8"> 
    genotype/col_header/pos: <HDF5 dataset "pos": shape (11623,), type "<i8"> 
    genotype/matrix: <HDF5 dataset "matrix": shape (1008, 11623), type "|u1"> 
    genotype/row_header: <HDF5 group "/genotype/row_header" (1 members)> 
    genotype/row_header/sample_ID: <HDF5 dataset "sample_ID": shape (1008,), type "|S99"> 
    phenotype: <HDF5 group "/phenotype" (3 members)> 
    phenotype/col_header: <HDF5 group "/phenotype/col_header" (1 members)> 
    phenotype/col_header/phenotype_ID: <HDF5 dataset "phenotype_ID": shape (46,), type "|S22"> 
    phenotype/matrix: <HDF5 dataset "matrix": shape (1008, 46), type "<f8"> 
    phenotype/row_header: <HDF5 group "/phenotype/row_header" (1 members)> 
    phenotype/row_header/sample_ID: <HDF5 dataset "sample_ID": shape (1008,), type "|S22"> 


Every object in an HDF5 file can have custom attributes which can be checked like any other dictionary:


```python
list(f['phenotype']['matrix'].attrs.keys())
```




    []



Let us examine the *phenotype/matrix* dataset as a `Dataset` object.   
The object we obtained isn’t an array, but an *HDF5* dataset. Like *NumPy* arrays, datasets have both a *shape* and a *data type*:


```python
dset = f['phenotype/matrix']
print('shape: (%s,%s)' % dset.shape)
print('dtype: %s' % dset.dtype)
```

    shape: (1008,46)
    dtype: float64


They also support array-style slicing. This is how you read and write data from a dataset in the file. 


```python
# retrieve all rows between 10 and 20 for the the second column
dset[10:20, 2]
```




    array([ 2.24597261,  2.6897812 , -0.46048117,  0.03086018, -0.29291799,
           -1.09117648,  0.70117087, -0.86986989, -0.13682744, -0.28441332])



When filtering/slicing/indexing rows and columns of a dataset, not the entire dataset is loaded into the main memory. This is very useful if you need to retrieve a subset of the data from a terabyte big HDF5 file.  
To read the entire dataset into the memory run:


```python
dset[:]
```




    array([[-7.32351971e+00,  2.79992827e-01,  3.13118166e-01, ...,
             8.90841948e-01,  4.11837231e+00,  8.59281836e+00],
           [-8.09823582e+00, -2.06326076e-01, -5.34843783e-01, ...,
             6.06164117e-03,  6.65150029e-02, -4.22047646e+00],
           [ 7.60571968e+00, -1.27959825e-01, -3.11102424e-01, ...,
             1.72315729e+00,  5.71408803e+00, -6.50651895e+00],
           ...,
           [            nan,  3.21844466e-01,             nan, ...,
             1.80142187e+00,             nan, -4.99069190e-01],
           [            nan, -1.00150686e+00,             nan, ...,
            -1.74752575e+00,             nan, -7.43596147e+00],
           [            nan,             nan,             nan, ...,
                        nan,             nan,             nan]])




```python
arr = dset[:]
print(type(arr))
arr
```

    <type 'numpy.ndarray'>





    array([[-7.32351971e+00,  2.79992827e-01,  3.13118166e-01, ...,
             8.90841948e-01,  4.11837231e+00,  8.59281836e+00],
           [-8.09823582e+00, -2.06326076e-01, -5.34843783e-01, ...,
             6.06164117e-03,  6.65150029e-02, -4.22047646e+00],
           [ 7.60571968e+00, -1.27959825e-01, -3.11102424e-01, ...,
             1.72315729e+00,  5.71408803e+00, -6.50651895e+00],
           ...,
           [            nan,  3.21844466e-01,             nan, ...,
             1.80142187e+00,             nan, -4.99069190e-01],
           [            nan, -1.00150686e+00,             nan, ...,
            -1.74752575e+00,             nan, -7.43596147e+00],
           [            nan,             nan,             nan, ...,
                        nan,             nan,             nan]])



---
## Pandas

Similar to *NumPy*, *Pandas* is one of the most widely used python libraries in data science. It provides high-performance, easy to use structures and data analysis tools. Unlike *NumPy* library which provides objects for multi-dimensional arrays, *Pandas* provides in-memory 2d table object called *Dataframe*. It is like a spreadsheet with column names and row label.  

Some commonly used data structures in pandas are:

- *Series* objects: 1D array, similar to a column in a spreadsheet
- *DataFrame* objects: 2D table, similar to a spreadsheet
- *Panel* objects: Dictionary of DataFrames, similar to sheet in MS Excel

Pandas can be imported into Python using:


```python
import pandas as pd
```

### Pandas Series

Pandas Series object is created using *pd.Series* function. Each row is provided with an index and by defaults is assigned numerical values starting from *0*. Like *NumPy*, Pandas also provide the basic mathematical functionalities like addition, subtraction and conditional operations and broadcasting.


```python
weight = pd.Series([68, 83, 112],index=["alice", "bob", "charles"])
birthyear = pd.Series([1984, 1985, 1992], index=["bob", "alice", "charles"], name="year")
children = pd.Series([0, 3], index=["charles", "bob"])
hobby = pd.Series(["Biking", "Dancing"], index=["alice", "bob"])
print(weight)
print('---')
print(weight.values)
```

    alice       68
    bob         83
    charles    112
    dtype: int64
    ---
    [ 68  83 112]


For a lot of applications, a plain old *Series* is probably not a lot of use, but it is the core component of the Pandas workhorse, the *DataFrame*, so it’s useful to know about.   

The *DataFrame* represents tabular data, a bit like a spreadsheet. *DataFrames* are organised into colums (each of which is a *Series*), and each column can store a single data-type, such as floating point numbers, strings, boolean values etc. *DataFrames* can be indexed by either their row or column names. (They are similar in many ways to R’s `data.frame`.).  

We can create a *DataFrame* in *Pandas* from a Python dictionary, or by loading in a text file containing tabular data. First we are going to look at how to create one from a dictionary.



```python
df = pd.DataFrame({'weight': weight, 'birthyear': birthyear, 'children': children, 'hobby': hobby})
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>birthyear</th>
      <th>children</th>
      <th>hobby</th>
      <th>weight</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>alice</th>
      <td>1985</td>
      <td>NaN</td>
      <td>Biking</td>
      <td>68</td>
    </tr>
    <tr>
      <th>bob</th>
      <td>1984</td>
      <td>3.0</td>
      <td>Dancing</td>
      <td>83</td>
    </tr>
    <tr>
      <th>charles</th>
      <td>1992</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>112</td>
    </tr>
  </tbody>
</table>
</div>



An alternative way to create a *DataFrame* is to load it from existing files. There are various loading function to load data from different file sources which can be found [here](https://pandas.pydata.org/pandas-docs/stable/api.html#input-output). The most important ones are:
- *pd.read_pickle*: Load pickled pandas object (or any object) from file
- *pd.read_csv*: Read CSV (comma-separated) file into DataFrame
- *pd.read_json*: Convert a JSON string to pandas object
- *pd.read_excel*: Read an Excel table into a pandas DataFrame
- *pd.read_hdf*: Read from the HDF5 file (PyTasbles format)

Let's return now to our HDF5 file that we previously examined.
The *phenotype* group contains 3 datasets:


```python
print(list(f['phenotype'].keys()))
print(f['phenotype/col_header/phenotype_ID'])
print(f['phenotype/row_header/sample_ID'])
```

    [u'col_header', u'matrix', u'row_header']
    <HDF5 dataset "phenotype_ID": shape (46,), type "|S22">
    <HDF5 dataset "sample_ID": shape (1008,), type "|S22">


This seems to be a phenotype matrix where the *matrix* dataset contains the 2d array with the phenotype values and the *col_header/phenotype_ID* contains the phenotype names and the *row_header/sample_ID* contains the accession ids.  
We can construct a Panda *dataframe* in the following way:


```python
pheno_group = f['phenotype']
df = pd.DataFrame(pheno_group['matrix'][:], 
                        columns=pheno_group['col_header']['phenotype_ID'][:],
                        index=pheno_group['row_header']['sample_ID'][:])

```

**Dataframes** have some essential methods to access the data:
- **head()**: returns the top 5 rows in the dataframe object
- **tail()**: returns the bottom 5 rows in the dataframe
- **info()**: prints the summary of the dataframe
- **describe()**: gives a nice overview of the main aggregated values over each column


```python
# retrieve first 10 rows
df.head(10)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Cadmium_Chloride</th>
      <th>Caffeine</th>
      <th>Calcium_Chloride</th>
      <th>Cisplatin</th>
      <th>Cobalt_Chloride</th>
      <th>Congo_red</th>
      <th>Copper</th>
      <th>Cycloheximide</th>
      <th>Diamide</th>
      <th>E6_Berbamine</th>
      <th>...</th>
      <th>x6-Azauracil</th>
      <th>Xylose</th>
      <th>YNB</th>
      <th>YNB:ph3</th>
      <th>YNB:ph8</th>
      <th>YPD</th>
      <th>YPD:15C</th>
      <th>YPD:37C</th>
      <th>YPD:4C</th>
      <th>Zeocin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>A01_01</th>
      <td>-7.323520</td>
      <td>0.279993</td>
      <td>0.313118</td>
      <td>1.658179</td>
      <td>-1.604442</td>
      <td>5.841617</td>
      <td>-4.130950</td>
      <td>0.821226</td>
      <td>3.622602</td>
      <td>-0.378747</td>
      <td>...</td>
      <td>1.199054</td>
      <td>-0.309149</td>
      <td>17.470822</td>
      <td>0.055225</td>
      <td>-0.184268</td>
      <td>24.548971</td>
      <td>0.712171</td>
      <td>0.890842</td>
      <td>4.118372</td>
      <td>8.592818</td>
    </tr>
    <tr>
      <th>A01_02</th>
      <td>-8.098236</td>
      <td>-0.206326</td>
      <td>-0.534844</td>
      <td>-0.918012</td>
      <td>0.892198</td>
      <td>-1.618172</td>
      <td>1.131947</td>
      <td>-0.764736</td>
      <td>-2.946279</td>
      <td>-2.475193</td>
      <td>...</td>
      <td>0.443182</td>
      <td>0.697908</td>
      <td>18.052925</td>
      <td>0.283462</td>
      <td>1.662201</td>
      <td>26.808476</td>
      <td>-1.522498</td>
      <td>0.006062</td>
      <td>0.066515</td>
      <td>-4.220476</td>
    </tr>
    <tr>
      <th>A01_03</th>
      <td>7.605720</td>
      <td>-0.127960</td>
      <td>-0.311102</td>
      <td>-2.712088</td>
      <td>3.301709</td>
      <td>-6.680571</td>
      <td>-1.138056</td>
      <td>-3.382532</td>
      <td>-3.157866</td>
      <td>-4.178616</td>
      <td>...</td>
      <td>1.472629</td>
      <td>1.474329</td>
      <td>18.163782</td>
      <td>-0.951379</td>
      <td>0.369565</td>
      <td>26.183975</td>
      <td>1.082289</td>
      <td>1.723157</td>
      <td>5.714088</td>
      <td>-6.506519</td>
    </tr>
    <tr>
      <th>A01_04</th>
      <td>-6.147649</td>
      <td>0.878392</td>
      <td>-0.563331</td>
      <td>-0.827358</td>
      <td>-1.291270</td>
      <td>2.753851</td>
      <td>-2.268121</td>
      <td>0.867456</td>
      <td>-6.832079</td>
      <td>-1.970630</td>
      <td>...</td>
      <td>0.054694</td>
      <td>-0.739040</td>
      <td>15.575856</td>
      <td>-1.690867</td>
      <td>-0.840344</td>
      <td>20.893646</td>
      <td>-0.636280</td>
      <td>0.053258</td>
      <td>-3.196936</td>
      <td>0.294462</td>
    </tr>
    <tr>
      <th>A01_05</th>
      <td>9.379060</td>
      <td>-1.353169</td>
      <td>0.405204</td>
      <td>-1.137234</td>
      <td>4.260286</td>
      <td>-7.391216</td>
      <td>-3.222742</td>
      <td>1.586376</td>
      <td>1.048391</td>
      <td>-3.971730</td>
      <td>...</td>
      <td>0.267463</td>
      <td>0.959138</td>
      <td>17.808444</td>
      <td>0.127621</td>
      <td>1.967267</td>
      <td>28.621507</td>
      <td>-3.720999</td>
      <td>-0.768724</td>
      <td>-4.002786</td>
      <td>7.045575</td>
    </tr>
    <tr>
      <th>A01_06</th>
      <td>9.394562</td>
      <td>1.655436</td>
      <td>0.531654</td>
      <td>-2.044903</td>
      <td>1.577325</td>
      <td>-5.769939</td>
      <td>1.355905</td>
      <td>2.336068</td>
      <td>4.038331</td>
      <td>-2.917101</td>
      <td>...</td>
      <td>0.591233</td>
      <td>-0.134019</td>
      <td>16.085601</td>
      <td>-0.100294</td>
      <td>0.710547</td>
      <td>23.549116</td>
      <td>-2.680231</td>
      <td>2.025716</td>
      <td>-0.889829</td>
      <td>-4.108786</td>
    </tr>
    <tr>
      <th>A01_07</th>
      <td>-7.229972</td>
      <td>-1.423605</td>
      <td>-0.007546</td>
      <td>-1.714243</td>
      <td>0.954948</td>
      <td>2.111495</td>
      <td>0.590067</td>
      <td>1.415252</td>
      <td>-4.861309</td>
      <td>-4.082216</td>
      <td>...</td>
      <td>-0.330839</td>
      <td>0.294128</td>
      <td>16.472448</td>
      <td>-0.612281</td>
      <td>-0.242591</td>
      <td>23.901629</td>
      <td>-1.834152</td>
      <td>-3.487390</td>
      <td>3.618048</td>
      <td>-0.035142</td>
    </tr>
    <tr>
      <th>A01_08</th>
      <td>-7.585476</td>
      <td>1.750342</td>
      <td>0.941181</td>
      <td>0.306306</td>
      <td>-2.997501</td>
      <td>0.063007</td>
      <td>-4.352137</td>
      <td>2.634605</td>
      <td>3.147146</td>
      <td>8.292883</td>
      <td>...</td>
      <td>0.043294</td>
      <td>-0.826648</td>
      <td>16.210856</td>
      <td>NaN</td>
      <td>-0.538390</td>
      <td>23.595423</td>
      <td>-1.449024</td>
      <td>1.679693</td>
      <td>-2.227455</td>
      <td>-5.773160</td>
    </tr>
    <tr>
      <th>A01_09</th>
      <td>-7.890982</td>
      <td>-0.559696</td>
      <td>-1.396937</td>
      <td>-0.879233</td>
      <td>-1.044798</td>
      <td>-6.214873</td>
      <td>0.211404</td>
      <td>-0.006420</td>
      <td>-9.470146</td>
      <td>-4.480685</td>
      <td>...</td>
      <td>-0.048497</td>
      <td>-0.483113</td>
      <td>15.161760</td>
      <td>-0.035535</td>
      <td>-0.398174</td>
      <td>22.693341</td>
      <td>-1.580512</td>
      <td>-1.320567</td>
      <td>-4.863302</td>
      <td>9.024782</td>
    </tr>
    <tr>
      <th>A01_10</th>
      <td>-8.144131</td>
      <td>-2.787328</td>
      <td>-1.111711</td>
      <td>-0.239611</td>
      <td>-2.933533</td>
      <td>1.231604</td>
      <td>1.397043</td>
      <td>2.405056</td>
      <td>8.532335</td>
      <td>14.355684</td>
      <td>...</td>
      <td>0.685319</td>
      <td>-0.594544</td>
      <td>16.388451</td>
      <td>-0.742849</td>
      <td>-0.427930</td>
      <td>23.697733</td>
      <td>-2.967864</td>
      <td>-1.553708</td>
      <td>-5.314155</td>
      <td>-1.229565</td>
    </tr>
  </tbody>
</table>
<p>10 rows × 46 columns</p>
</div>




```python
# retrieve last 5 rows
df.tail()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Cadmium_Chloride</th>
      <th>Caffeine</th>
      <th>Calcium_Chloride</th>
      <th>Cisplatin</th>
      <th>Cobalt_Chloride</th>
      <th>Congo_red</th>
      <th>Copper</th>
      <th>Cycloheximide</th>
      <th>Diamide</th>
      <th>E6_Berbamine</th>
      <th>...</th>
      <th>x6-Azauracil</th>
      <th>Xylose</th>
      <th>YNB</th>
      <th>YNB:ph3</th>
      <th>YNB:ph8</th>
      <th>YPD</th>
      <th>YPD:15C</th>
      <th>YPD:37C</th>
      <th>YPD:4C</th>
      <th>Zeocin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>A11_92</th>
      <td>-4.262720</td>
      <td>-0.893290</td>
      <td>-0.329504</td>
      <td>0.257061</td>
      <td>-3.120957</td>
      <td>2.556543</td>
      <td>2.567092</td>
      <td>2.054232</td>
      <td>1.672347</td>
      <td>5.465978</td>
      <td>...</td>
      <td>0.232614</td>
      <td>1.130568</td>
      <td>16.629005</td>
      <td>0.531583</td>
      <td>NaN</td>
      <td>23.520842</td>
      <td>0.159946</td>
      <td>-5.158120</td>
      <td>-0.090273</td>
      <td>-7.030154</td>
    </tr>
    <tr>
      <th>A11_93</th>
      <td>-10.147285</td>
      <td>1.145573</td>
      <td>0.048948</td>
      <td>-1.089350</td>
      <td>0.539833</td>
      <td>2.720952</td>
      <td>-3.445357</td>
      <td>0.891066</td>
      <td>3.003684</td>
      <td>-1.667592</td>
      <td>...</td>
      <td>-0.317916</td>
      <td>1.977915</td>
      <td>15.687951</td>
      <td>-0.098451</td>
      <td>NaN</td>
      <td>21.553930</td>
      <td>2.977722</td>
      <td>-1.913176</td>
      <td>4.279626</td>
      <td>-4.650492</td>
    </tr>
    <tr>
      <th>A11_94</th>
      <td>NaN</td>
      <td>0.321844</td>
      <td>NaN</td>
      <td>1.016712</td>
      <td>1.801923</td>
      <td>-4.157083</td>
      <td>-0.097598</td>
      <td>-1.954080</td>
      <td>-1.739499</td>
      <td>-6.520551</td>
      <td>...</td>
      <td>-1.698334</td>
      <td>1.363267</td>
      <td>17.357844</td>
      <td>0.390511</td>
      <td>NaN</td>
      <td>23.977236</td>
      <td>0.078992</td>
      <td>1.801422</td>
      <td>NaN</td>
      <td>-0.499069</td>
    </tr>
    <tr>
      <th>A11_95</th>
      <td>NaN</td>
      <td>-1.001507</td>
      <td>NaN</td>
      <td>0.004191</td>
      <td>4.675717</td>
      <td>-10.240998</td>
      <td>1.244410</td>
      <td>-0.810857</td>
      <td>2.305318</td>
      <td>9.321186</td>
      <td>...</td>
      <td>0.259829</td>
      <td>0.942454</td>
      <td>17.098753</td>
      <td>-0.432459</td>
      <td>NaN</td>
      <td>22.927686</td>
      <td>-0.254905</td>
      <td>-1.747526</td>
      <td>NaN</td>
      <td>-7.435961</td>
    </tr>
    <tr>
      <th>A11_96</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>-0.155556</td>
      <td>NaN</td>
      <td>2.603438</td>
      <td>-0.026752</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>...</td>
      <td>NaN</td>
      <td>1.776737</td>
      <td>17.924399</td>
      <td>-0.165314</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.553372</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 46 columns</p>
</div>




```python
# Summary of dataframe
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1008 entries, A01_01 to A11_96
    Data columns (total 46 columns):
    Cadmium_Chloride          799 non-null float64
    Caffeine                  1004 non-null float64
    Calcium_Chloride          949 non-null float64
    Cisplatin                 990 non-null float64
    Cobalt_Chloride           1007 non-null float64
    Congo_red                 979 non-null float64
    Copper                    972 non-null float64
    Cycloheximide             1004 non-null float64
    Diamide                   1003 non-null float64
    E6_Berbamine              1005 non-null float64
    Ethanol                   967 non-null float64
    Formamide                 999 non-null float64
    Galactose                 962 non-null float64
    Hydrogen_Peroxide         769 non-null float64
    Hydroquinone              960 non-null float64
    Hydroxyurea               944 non-null float64
    Indoleacetic_Acid         994 non-null float64
    Lactate                   973 non-null float64
    Lactose                   1004 non-null float64
    Lithium_Chloride          1002 non-null float64
    Magnesium_Chloride        964 non-null float64
    Magnesium_Sulfate         1006 non-null float64
    Maltose                   1005 non-null float64
    Mannose                   957 non-null float64
    Menadione                 1005 non-null float64
    Neomycin                  1005 non-null float64
    Paraquat                  1004 non-null float64
    Raffinose                 662 non-null float64
    SDS                       873 non-null float64
    Sorbitol                  599 non-null float64
    Trehalose                 1003 non-null float64
    Tunicamycin               1006 non-null float64
    x4-Hydroxybenzaldehyde    964 non-null float64
    x4NQO                     1005 non-null float64
    x5-Fluorocytosine         957 non-null float64
    x5-Fluorouracil           992 non-null float64
    x6-Azauracil              1001 non-null float64
    Xylose                    1003 non-null float64
    YNB                       1006 non-null float64
    YNB:ph3                   979 non-null float64
    YNB:ph8                   887 non-null float64
    YPD                       1006 non-null float64
    YPD:15C                   1005 non-null float64
    YPD:37C                   1003 non-null float64
    YPD:4C                    806 non-null float64
    Zeocin                    957 non-null float64
    dtypes: float64(46)
    memory usage: 370.1+ KB



```python
# calculate some summary statistics
df.describe()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Cadmium_Chloride</th>
      <th>Caffeine</th>
      <th>Calcium_Chloride</th>
      <th>Cisplatin</th>
      <th>Cobalt_Chloride</th>
      <th>Congo_red</th>
      <th>Copper</th>
      <th>Cycloheximide</th>
      <th>Diamide</th>
      <th>E6_Berbamine</th>
      <th>...</th>
      <th>x6-Azauracil</th>
      <th>Xylose</th>
      <th>YNB</th>
      <th>YNB:ph3</th>
      <th>YNB:ph8</th>
      <th>YPD</th>
      <th>YPD:15C</th>
      <th>YPD:37C</th>
      <th>YPD:4C</th>
      <th>Zeocin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>799.000000</td>
      <td>1004.000000</td>
      <td>949.000000</td>
      <td>990.000000</td>
      <td>1007.000000</td>
      <td>979.000000</td>
      <td>972.000000</td>
      <td>1004.000000</td>
      <td>1003.000000</td>
      <td>1005.000000</td>
      <td>...</td>
      <td>1001.000000</td>
      <td>1003.000000</td>
      <td>1006.000000</td>
      <td>979.000000</td>
      <td>887.000000</td>
      <td>1006.000000</td>
      <td>1005.000000</td>
      <td>1003.000000</td>
      <td>806.000000</td>
      <td>957.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>-0.235763</td>
      <td>0.056543</td>
      <td>0.012039</td>
      <td>-0.080343</td>
      <td>-0.021220</td>
      <td>-0.580926</td>
      <td>0.067722</td>
      <td>0.157040</td>
      <td>0.139683</td>
      <td>0.264369</td>
      <td>...</td>
      <td>0.112644</td>
      <td>0.060641</td>
      <td>16.315382</td>
      <td>0.007996</td>
      <td>0.007482</td>
      <td>23.502212</td>
      <td>0.047823</td>
      <td>-0.140778</td>
      <td>-0.190777</td>
      <td>0.166791</td>
    </tr>
    <tr>
      <th>std</th>
      <td>6.306318</td>
      <td>0.910155</td>
      <td>0.773159</td>
      <td>1.045181</td>
      <td>2.808820</td>
      <td>6.411729</td>
      <td>1.808578</td>
      <td>1.683631</td>
      <td>4.749402</td>
      <td>7.083140</td>
      <td>...</td>
      <td>0.853002</td>
      <td>0.939542</td>
      <td>1.034526</td>
      <td>0.674851</td>
      <td>0.902919</td>
      <td>1.815418</td>
      <td>2.541834</td>
      <td>2.642301</td>
      <td>4.649077</td>
      <td>6.433438</td>
    </tr>
    <tr>
      <th>min</th>
      <td>-10.381891</td>
      <td>-4.141899</td>
      <td>-2.452206</td>
      <td>-4.557306</td>
      <td>-6.184913</td>
      <td>-13.488314</td>
      <td>-8.944908</td>
      <td>-5.980597</td>
      <td>-17.291493</td>
      <td>-12.513969</td>
      <td>...</td>
      <td>-3.327925</td>
      <td>-2.911638</td>
      <td>11.981303</td>
      <td>-2.570029</td>
      <td>-3.724701</td>
      <td>16.979292</td>
      <td>-6.493270</td>
      <td>-7.377041</td>
      <td>-9.588223</td>
      <td>-13.359268</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>-6.456470</td>
      <td>-0.507907</td>
      <td>-0.498740</td>
      <td>-0.741802</td>
      <td>-2.288283</td>
      <td>-6.620026</td>
      <td>-0.665649</td>
      <td>-0.963134</td>
      <td>-3.193845</td>
      <td>-4.880492</td>
      <td>...</td>
      <td>-0.449021</td>
      <td>-0.599287</td>
      <td>15.661717</td>
      <td>-0.451187</td>
      <td>-0.571156</td>
      <td>22.380371</td>
      <td>-1.879136</td>
      <td>-2.168742</td>
      <td>-3.680280</td>
      <td>-5.159223</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>1.022356</td>
      <td>0.067418</td>
      <td>0.042018</td>
      <td>-0.019068</td>
      <td>-0.377311</td>
      <td>-0.568692</td>
      <td>0.396090</td>
      <td>0.234640</td>
      <td>0.892107</td>
      <td>-2.475193</td>
      <td>...</td>
      <td>0.146145</td>
      <td>0.047475</td>
      <td>16.358272</td>
      <td>0.024450</td>
      <td>-0.003055</td>
      <td>23.403000</td>
      <td>-0.132709</td>
      <td>-0.047865</td>
      <td>-0.977779</td>
      <td>-2.015229</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>5.813172</td>
      <td>0.622523</td>
      <td>0.503318</td>
      <td>0.645479</td>
      <td>1.998527</td>
      <td>4.537839</td>
      <td>1.235156</td>
      <td>1.324370</td>
      <td>3.327036</td>
      <td>4.487006</td>
      <td>...</td>
      <td>0.700031</td>
      <td>0.717306</td>
      <td>17.006638</td>
      <td>0.479301</td>
      <td>0.628253</td>
      <td>24.629824</td>
      <td>1.832583</td>
      <td>1.966580</td>
      <td>2.605938</td>
      <td>5.751830</td>
    </tr>
    <tr>
      <th>max</th>
      <td>12.876948</td>
      <td>3.372236</td>
      <td>2.689781</td>
      <td>3.763224</td>
      <td>8.467436</td>
      <td>15.984204</td>
      <td>4.924384</td>
      <td>6.516210</td>
      <td>11.349569</td>
      <td>23.116821</td>
      <td>...</td>
      <td>2.816535</td>
      <td>3.286086</td>
      <td>19.067704</td>
      <td>1.773026</td>
      <td>2.956204</td>
      <td>29.985755</td>
      <td>7.287769</td>
      <td>7.474741</td>
      <td>15.552609</td>
      <td>16.662919</td>
    </tr>
  </tbody>
</table>
<p>8 rows × 46 columns</p>
</div>



### Accessing data in a DataFrame

Our columns in the dataframe object are individual Series of data. We can access them by referring to the column name e.g. dataframe['column-name']. For example to access the *Caffeine* column run this:


```python
print(df['Caffeine'].head())
print('============= or ===========')
print(df.Caffeine.head())
```

    A01_01    0.279993
    A01_02   -0.206326
    A01_03   -0.127960
    A01_04    0.878392
    A01_05   -1.353169
    Name: Caffeine, dtype: float64
    ============= or ===========
    A01_01    0.279993
    A01_02   -0.206326
    A01_03   -0.127960
    A01_04    0.878392
    A01_05   -1.353169
    Name: Caffeine, dtype: float64


To filter by row we have to use the *iloc* or *loc* function on the *dataframe*. The *iloc* function allows to filter by the integer index of the row while the *loc* function allows to index by the index label (accession id):


```python
df.iloc[2].Caffeine
```




    -0.127959825210201




```python
df.loc['A01_03'].Caffeine
```




    -0.127959825210201



It is also possible to retrieve multiple rows either by specifying the exact row index labels or integer positions:


```python
df.iloc[[1, 5, 6, 9]].Caffeine
```




    A01_02   -0.206326
    A01_06    1.655436
    A01_07   -1.423605
    A01_10   -2.787328
    Name: Caffeine, dtype: float64




```python
df.loc[['A01_02', 'A01_06', 'A01_07', 'A01_10']].Caffeine
```




    A01_02   -0.206326
    A01_06    1.655436
    A01_07   -1.423605
    A01_10   -2.787328
    Name: Caffeine, dtype: float64




```python
# retrieve the first 10 rows for the columns Caffeine (column 1 ) and Cisplatin (column 2)
df.iloc[:10,[1,3]]
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Caffeine</th>
      <th>Cisplatin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>A01_01</th>
      <td>0.279993</td>
      <td>1.658179</td>
    </tr>
    <tr>
      <th>A01_02</th>
      <td>-0.206326</td>
      <td>-0.918012</td>
    </tr>
    <tr>
      <th>A01_03</th>
      <td>-0.127960</td>
      <td>-2.712088</td>
    </tr>
    <tr>
      <th>A01_04</th>
      <td>0.878392</td>
      <td>-0.827358</td>
    </tr>
    <tr>
      <th>A01_05</th>
      <td>-1.353169</td>
      <td>-1.137234</td>
    </tr>
    <tr>
      <th>A01_06</th>
      <td>1.655436</td>
      <td>-2.044903</td>
    </tr>
    <tr>
      <th>A01_07</th>
      <td>-1.423605</td>
      <td>-1.714243</td>
    </tr>
    <tr>
      <th>A01_08</th>
      <td>1.750342</td>
      <td>0.306306</td>
    </tr>
    <tr>
      <th>A01_09</th>
      <td>-0.559696</td>
      <td>-0.879233</td>
    </tr>
    <tr>
      <th>A01_10</th>
      <td>-2.787328</td>
      <td>-0.239611</td>
    </tr>
  </tbody>
</table>
</div>



### Filtering data

We can also apply conditions to the data we are inspecting, such as to filter our data.
Let's examine the Caffeine column and check the lowest value.


```python
df.Caffeine.min()
```




    -4.14189858085831



Let's now filter the Caffeine column for values that are greater than 0 


```python
df.Caffeine > 0
```




    A01_01     True
    A01_02    False
    A01_03    False
    A01_04     True
    A01_05    False
    A01_06     True
    A01_07    False
    A01_08     True
    A01_09    False
    A01_10    False
    A01_11    False
    A01_12    False
    A01_13     True
    A01_14     True
    A01_15    False
    A01_16    False
    A01_17     True
    A01_18     True
    A01_19    False
    A01_20     True
    A01_21     True
    A01_22     True
    A01_23     True
    A01_24    False
    A01_25     True
    A01_26     True
    A01_27     True
    A01_28    False
    A01_29     True
    A01_30    False
              ...  
    A11_67    False
    A11_68     True
    A11_69     True
    A11_70    False
    A11_71    False
    A11_72     True
    A11_73     True
    A11_74    False
    A11_75    False
    A11_76    False
    A11_77     True
    A11_78     True
    A11_79     True
    A11_80     True
    A11_81    False
    A11_82    False
    A11_83    False
    A11_84    False
    A11_85     True
    A11_86     True
    A11_87     True
    A11_88    False
    A11_89     True
    A11_90     True
    A11_91     True
    A11_92    False
    A11_93     True
    A11_94     True
    A11_95    False
    A11_96    False
    Name: Caffeine, Length: 1008, dtype: bool



This returns a new *Series* of *True*/*False* values though. To actually filter the data, we need to use this *Series* to mask our original *DataFrame*


```python
df[df.Caffeine > 0]
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Cadmium_Chloride</th>
      <th>Caffeine</th>
      <th>Calcium_Chloride</th>
      <th>Cisplatin</th>
      <th>Cobalt_Chloride</th>
      <th>Congo_red</th>
      <th>Copper</th>
      <th>Cycloheximide</th>
      <th>Diamide</th>
      <th>E6_Berbamine</th>
      <th>...</th>
      <th>x6-Azauracil</th>
      <th>Xylose</th>
      <th>YNB</th>
      <th>YNB:ph3</th>
      <th>YNB:ph8</th>
      <th>YPD</th>
      <th>YPD:15C</th>
      <th>YPD:37C</th>
      <th>YPD:4C</th>
      <th>Zeocin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>A01_01</th>
      <td>-7.323520</td>
      <td>0.279993</td>
      <td>0.313118</td>
      <td>1.658179</td>
      <td>-1.604442</td>
      <td>5.841617</td>
      <td>-4.130950</td>
      <td>0.821226</td>
      <td>3.622602</td>
      <td>-0.378747</td>
      <td>...</td>
      <td>1.199054</td>
      <td>-0.309149</td>
      <td>17.470822</td>
      <td>0.055225</td>
      <td>-0.184268</td>
      <td>24.548971</td>
      <td>0.712171</td>
      <td>0.890842</td>
      <td>4.118372</td>
      <td>8.592818</td>
    </tr>
    <tr>
      <th>A01_04</th>
      <td>-6.147649</td>
      <td>0.878392</td>
      <td>-0.563331</td>
      <td>-0.827358</td>
      <td>-1.291270</td>
      <td>2.753851</td>
      <td>-2.268121</td>
      <td>0.867456</td>
      <td>-6.832079</td>
      <td>-1.970630</td>
      <td>...</td>
      <td>0.054694</td>
      <td>-0.739040</td>
      <td>15.575856</td>
      <td>-1.690867</td>
      <td>-0.840344</td>
      <td>20.893646</td>
      <td>-0.636280</td>
      <td>0.053258</td>
      <td>-3.196936</td>
      <td>0.294462</td>
    </tr>
    <tr>
      <th>A01_06</th>
      <td>9.394562</td>
      <td>1.655436</td>
      <td>0.531654</td>
      <td>-2.044903</td>
      <td>1.577325</td>
      <td>-5.769939</td>
      <td>1.355905</td>
      <td>2.336068</td>
      <td>4.038331</td>
      <td>-2.917101</td>
      <td>...</td>
      <td>0.591233</td>
      <td>-0.134019</td>
      <td>16.085601</td>
      <td>-0.100294</td>
      <td>0.710547</td>
      <td>23.549116</td>
      <td>-2.680231</td>
      <td>2.025716</td>
      <td>-0.889829</td>
      <td>-4.108786</td>
    </tr>
    <tr>
      <th>A01_08</th>
      <td>-7.585476</td>
      <td>1.750342</td>
      <td>0.941181</td>
      <td>0.306306</td>
      <td>-2.997501</td>
      <td>0.063007</td>
      <td>-4.352137</td>
      <td>2.634605</td>
      <td>3.147146</td>
      <td>8.292883</td>
      <td>...</td>
      <td>0.043294</td>
      <td>-0.826648</td>
      <td>16.210856</td>
      <td>NaN</td>
      <td>-0.538390</td>
      <td>23.595423</td>
      <td>-1.449024</td>
      <td>1.679693</td>
      <td>-2.227455</td>
      <td>-5.773160</td>
    </tr>
    <tr>
      <th>A01_13</th>
      <td>6.099126</td>
      <td>1.193630</td>
      <td>-0.460481</td>
      <td>-2.254822</td>
      <td>-1.737934</td>
      <td>-6.773013</td>
      <td>1.847423</td>
      <td>1.492433</td>
      <td>5.377797</td>
      <td>-2.763472</td>
      <td>...</td>
      <td>0.428124</td>
      <td>-0.447422</td>
      <td>15.707533</td>
      <td>1.328875</td>
      <td>-0.676125</td>
      <td>22.880547</td>
      <td>-4.488093</td>
      <td>3.399301</td>
      <td>-5.313911</td>
      <td>-4.854207</td>
    </tr>
    <tr>
      <th>A01_14</th>
      <td>-6.128808</td>
      <td>2.287569</td>
      <td>0.030860</td>
      <td>1.457627</td>
      <td>2.658909</td>
      <td>8.873294</td>
      <td>-2.725993</td>
      <td>2.081959</td>
      <td>-0.640953</td>
      <td>-2.948221</td>
      <td>...</td>
      <td>0.871464</td>
      <td>0.239443</td>
      <td>16.293701</td>
      <td>0.306043</td>
      <td>0.903620</td>
      <td>23.053594</td>
      <td>4.081697</td>
      <td>3.783942</td>
      <td>0.069884</td>
      <td>2.976971</td>
    </tr>
    <tr>
      <th>A01_17</th>
      <td>-6.435934</td>
      <td>0.331987</td>
      <td>0.701171</td>
      <td>0.330169</td>
      <td>-3.157943</td>
      <td>-1.098844</td>
      <td>2.099880</td>
      <td>3.261305</td>
      <td>-10.892045</td>
      <td>14.833187</td>
      <td>...</td>
      <td>0.778375</td>
      <td>1.257294</td>
      <td>16.432578</td>
      <td>0.030953</td>
      <td>1.637458</td>
      <td>23.623720</td>
      <td>3.763264</td>
      <td>-3.649057</td>
      <td>7.883571</td>
      <td>-5.824656</td>
    </tr>
    <tr>
      <th>A01_18</th>
      <td>-1.744416</td>
      <td>1.179753</td>
      <td>-0.869870</td>
      <td>-1.269823</td>
      <td>3.048698</td>
      <td>-6.121298</td>
      <td>0.008026</td>
      <td>-0.243670</td>
      <td>3.129125</td>
      <td>-0.310028</td>
      <td>...</td>
      <td>-0.658192</td>
      <td>0.014206</td>
      <td>17.374168</td>
      <td>-0.383067</td>
      <td>-0.997588</td>
      <td>23.653983</td>
      <td>-2.572794</td>
      <td>1.388221</td>
      <td>-6.245010</td>
      <td>-1.875007</td>
    </tr>
    <tr>
      <th>A01_20</th>
      <td>-6.362380</td>
      <td>0.986639</td>
      <td>-0.284413</td>
      <td>0.014248</td>
      <td>2.235446</td>
      <td>-1.681283</td>
      <td>-0.420546</td>
      <td>0.076087</td>
      <td>3.076295</td>
      <td>-6.065038</td>
      <td>...</td>
      <td>-1.904024</td>
      <td>0.127447</td>
      <td>14.506945</td>
      <td>-0.817359</td>
      <td>-0.907824</td>
      <td>20.581287</td>
      <td>0.125045</td>
      <td>-0.928468</td>
      <td>-2.542536</td>
      <td>6.886391</td>
    </tr>
    <tr>
      <th>A01_21</th>
      <td>5.925445</td>
      <td>1.238484</td>
      <td>-0.180890</td>
      <td>-0.785711</td>
      <td>0.856511</td>
      <td>-0.954674</td>
      <td>1.886783</td>
      <td>2.196050</td>
      <td>2.436579</td>
      <td>-0.643799</td>
      <td>...</td>
      <td>0.736404</td>
      <td>1.220645</td>
      <td>17.109577</td>
      <td>0.002582</td>
      <td>-0.689019</td>
      <td>24.960265</td>
      <td>-4.600436</td>
      <td>-3.240098</td>
      <td>-1.430459</td>
      <td>4.630062</td>
    </tr>
    <tr>
      <th>A01_22</th>
      <td>3.015350</td>
      <td>0.844175</td>
      <td>0.640409</td>
      <td>1.477814</td>
      <td>2.287242</td>
      <td>-0.343746</td>
      <td>1.038470</td>
      <td>-0.832654</td>
      <td>2.717067</td>
      <td>-4.706824</td>
      <td>...</td>
      <td>2.037066</td>
      <td>1.738069</td>
      <td>17.113177</td>
      <td>0.625722</td>
      <td>0.712437</td>
      <td>24.854792</td>
      <td>5.443331</td>
      <td>-4.111030</td>
      <td>5.459420</td>
      <td>10.382066</td>
    </tr>
    <tr>
      <th>A01_23</th>
      <td>5.368651</td>
      <td>1.648824</td>
      <td>-0.311006</td>
      <td>0.151763</td>
      <td>2.089597</td>
      <td>0.886900</td>
      <td>-0.827784</td>
      <td>-0.076783</td>
      <td>2.241636</td>
      <td>-5.830545</td>
      <td>...</td>
      <td>-0.369967</td>
      <td>-0.887642</td>
      <td>15.680565</td>
      <td>-0.132719</td>
      <td>0.793680</td>
      <td>23.211814</td>
      <td>1.160337</td>
      <td>3.339040</td>
      <td>0.908244</td>
      <td>13.591021</td>
    </tr>
    <tr>
      <th>A01_25</th>
      <td>2.361884</td>
      <td>1.207676</td>
      <td>-1.382897</td>
      <td>1.224807</td>
      <td>-1.735648</td>
      <td>-0.200892</td>
      <td>NaN</td>
      <td>-2.155111</td>
      <td>-6.857530</td>
      <td>-3.846204</td>
      <td>...</td>
      <td>0.909605</td>
      <td>-1.330609</td>
      <td>13.144011</td>
      <td>1.004012</td>
      <td>0.127184</td>
      <td>22.482902</td>
      <td>-0.605416</td>
      <td>0.017608</td>
      <td>0.398050</td>
      <td>-2.081629</td>
    </tr>
    <tr>
      <th>A01_26</th>
      <td>-5.888597</td>
      <td>1.748776</td>
      <td>-0.432020</td>
      <td>-0.303939</td>
      <td>-2.064806</td>
      <td>-0.087783</td>
      <td>0.418293</td>
      <td>1.550241</td>
      <td>1.862355</td>
      <td>4.625362</td>
      <td>...</td>
      <td>0.572039</td>
      <td>-0.514878</td>
      <td>16.622170</td>
      <td>-0.709576</td>
      <td>-0.502412</td>
      <td>23.804121</td>
      <td>-4.416801</td>
      <td>1.710262</td>
      <td>-2.832793</td>
      <td>-2.582201</td>
    </tr>
    <tr>
      <th>A01_27</th>
      <td>5.864971</td>
      <td>0.085989</td>
      <td>0.272710</td>
      <td>0.687241</td>
      <td>-2.272625</td>
      <td>-6.032576</td>
      <td>2.116587</td>
      <td>0.717661</td>
      <td>4.509182</td>
      <td>8.784337</td>
      <td>...</td>
      <td>0.984612</td>
      <td>0.663690</td>
      <td>17.433361</td>
      <td>-0.237397</td>
      <td>-0.214977</td>
      <td>22.998909</td>
      <td>1.925353</td>
      <td>4.612446</td>
      <td>2.617287</td>
      <td>7.365063</td>
    </tr>
    <tr>
      <th>A01_29</th>
      <td>8.291001</td>
      <td>0.602952</td>
      <td>-0.209818</td>
      <td>0.176575</td>
      <td>2.970487</td>
      <td>11.674431</td>
      <td>-5.015363</td>
      <td>0.942569</td>
      <td>-0.493707</td>
      <td>10.374494</td>
      <td>...</td>
      <td>0.222568</td>
      <td>-1.013385</td>
      <td>16.763518</td>
      <td>-0.206383</td>
      <td>0.672626</td>
      <td>26.448189</td>
      <td>1.866508</td>
      <td>0.239108</td>
      <td>-0.638321</td>
      <td>12.690415</td>
    </tr>
    <tr>
      <th>A01_31</th>
      <td>-6.398361</td>
      <td>0.166271</td>
      <td>-0.878706</td>
      <td>-0.078068</td>
      <td>-3.349890</td>
      <td>-1.069199</td>
      <td>1.163056</td>
      <td>-0.895535</td>
      <td>-0.087970</td>
      <td>-4.928491</td>
      <td>...</td>
      <td>-1.314301</td>
      <td>-1.375641</td>
      <td>15.544307</td>
      <td>-0.004930</td>
      <td>-0.744114</td>
      <td>24.094502</td>
      <td>-2.020393</td>
      <td>-0.891901</td>
      <td>-5.279599</td>
      <td>1.193205</td>
    </tr>
    <tr>
      <th>A01_32</th>
      <td>7.130669</td>
      <td>0.718797</td>
      <td>0.518366</td>
      <td>1.823296</td>
      <td>0.429805</td>
      <td>-1.103549</td>
      <td>0.740313</td>
      <td>2.506797</td>
      <td>2.778662</td>
      <td>-4.981721</td>
      <td>...</td>
      <td>0.632448</td>
      <td>1.494387</td>
      <td>16.222977</td>
      <td>0.751728</td>
      <td>-2.119434</td>
      <td>22.723083</td>
      <td>5.791591</td>
      <td>-0.958975</td>
      <td>1.065139</td>
      <td>-2.015229</td>
    </tr>
    <tr>
      <th>A01_33</th>
      <td>-7.996359</td>
      <td>0.922486</td>
      <td>0.264327</td>
      <td>-1.664310</td>
      <td>-5.532087</td>
      <td>-5.217066</td>
      <td>1.188328</td>
      <td>2.380337</td>
      <td>3.479604</td>
      <td>-2.757452</td>
      <td>...</td>
      <td>1.271010</td>
      <td>1.306615</td>
      <td>18.784166</td>
      <td>0.187778</td>
      <td>0.246367</td>
      <td>29.168126</td>
      <td>4.090078</td>
      <td>0.995169</td>
      <td>3.198552</td>
      <td>-10.757366</td>
    </tr>
    <tr>
      <th>A01_34</th>
      <td>-7.061253</td>
      <td>0.024929</td>
      <td>1.033085</td>
      <td>0.766978</td>
      <td>3.873184</td>
      <td>1.228287</td>
      <td>-0.019795</td>
      <td>-0.597636</td>
      <td>-0.869106</td>
      <td>-3.144067</td>
      <td>...</td>
      <td>1.212225</td>
      <td>1.713806</td>
      <td>17.565024</td>
      <td>0.101327</td>
      <td>-0.127652</td>
      <td>23.749063</td>
      <td>3.836577</td>
      <td>-2.537776</td>
      <td>6.718132</td>
      <td>-4.968092</td>
    </tr>
    <tr>
      <th>A01_35</th>
      <td>7.125313</td>
      <td>0.477383</td>
      <td>0.203855</td>
      <td>0.146072</td>
      <td>4.542455</td>
      <td>-4.875453</td>
      <td>2.360665</td>
      <td>-1.442190</td>
      <td>0.693330</td>
      <td>-6.215016</td>
      <td>...</td>
      <td>1.818472</td>
      <td>0.151740</td>
      <td>16.212551</td>
      <td>0.662079</td>
      <td>-0.147682</td>
      <td>23.430682</td>
      <td>-2.494964</td>
      <td>3.580115</td>
      <td>-2.705861</td>
      <td>-4.434123</td>
    </tr>
    <tr>
      <th>A01_38</th>
      <td>7.221967</td>
      <td>1.607407</td>
      <td>1.136931</td>
      <td>0.321945</td>
      <td>-2.917663</td>
      <td>6.128958</td>
      <td>-0.439967</td>
      <td>3.226792</td>
      <td>4.648551</td>
      <td>-3.809075</td>
      <td>...</td>
      <td>1.010044</td>
      <td>0.545939</td>
      <td>15.461926</td>
      <td>0.202358</td>
      <td>0.744799</td>
      <td>22.786138</td>
      <td>-0.951993</td>
      <td>-1.479317</td>
      <td>6.993651</td>
      <td>-5.792154</td>
    </tr>
    <tr>
      <th>A01_40</th>
      <td>7.386030</td>
      <td>0.685339</td>
      <td>0.552385</td>
      <td>1.627962</td>
      <td>4.060814</td>
      <td>-1.013393</td>
      <td>-0.051946</td>
      <td>0.805096</td>
      <td>5.205418</td>
      <td>0.235112</td>
      <td>...</td>
      <td>-0.328904</td>
      <td>0.065395</td>
      <td>16.315874</td>
      <td>0.539172</td>
      <td>-0.718097</td>
      <td>23.366239</td>
      <td>3.277305</td>
      <td>1.194231</td>
      <td>0.670884</td>
      <td>7.639392</td>
    </tr>
    <tr>
      <th>A01_44</th>
      <td>5.584843</td>
      <td>1.063073</td>
      <td>-0.673369</td>
      <td>0.292690</td>
      <td>-0.585619</td>
      <td>-1.411756</td>
      <td>0.982226</td>
      <td>-0.333864</td>
      <td>2.798290</td>
      <td>0.193106</td>
      <td>...</td>
      <td>-0.635281</td>
      <td>-0.836197</td>
      <td>15.972579</td>
      <td>0.842179</td>
      <td>1.050111</td>
      <td>22.333949</td>
      <td>-4.374102</td>
      <td>1.138760</td>
      <td>-5.530835</td>
      <td>5.771627</td>
    </tr>
    <tr>
      <th>A01_45</th>
      <td>-6.886116</td>
      <td>2.422459</td>
      <td>2.094146</td>
      <td>2.513203</td>
      <td>6.714054</td>
      <td>-0.397405</td>
      <td>-6.722845</td>
      <td>-1.037253</td>
      <td>-0.875755</td>
      <td>-5.357255</td>
      <td>...</td>
      <td>-1.635183</td>
      <td>0.518553</td>
      <td>15.862309</td>
      <td>1.405404</td>
      <td>1.074609</td>
      <td>20.873441</td>
      <td>-2.156279</td>
      <td>6.328426</td>
      <td>1.883678</td>
      <td>-3.488512</td>
    </tr>
    <tr>
      <th>A01_46</th>
      <td>-6.857821</td>
      <td>0.372263</td>
      <td>-1.493623</td>
      <td>-2.658450</td>
      <td>-0.895147</td>
      <td>-5.366424</td>
      <td>-0.115780</td>
      <td>-0.501187</td>
      <td>3.045313</td>
      <td>-5.753381</td>
      <td>...</td>
      <td>-1.029738</td>
      <td>-0.399975</td>
      <td>15.623093</td>
      <td>-0.450903</td>
      <td>-0.254102</td>
      <td>22.505698</td>
      <td>-3.920752</td>
      <td>-2.202890</td>
      <td>-1.563144</td>
      <td>1.055169</td>
    </tr>
    <tr>
      <th>A01_47</th>
      <td>-4.268287</td>
      <td>0.362259</td>
      <td>0.310740</td>
      <td>1.390570</td>
      <td>0.966619</td>
      <td>10.681893</td>
      <td>0.633434</td>
      <td>0.512259</td>
      <td>1.973278</td>
      <td>-6.997449</td>
      <td>...</td>
      <td>0.574923</td>
      <td>0.748745</td>
      <td>17.494740</td>
      <td>-0.075021</td>
      <td>0.094029</td>
      <td>23.578347</td>
      <td>-0.045091</td>
      <td>-2.543260</td>
      <td>1.207601</td>
      <td>4.475345</td>
    </tr>
    <tr>
      <th>A01_55</th>
      <td>4.074521</td>
      <td>1.005304</td>
      <td>-0.840341</td>
      <td>-0.156409</td>
      <td>1.439703</td>
      <td>-0.288215</td>
      <td>-2.433246</td>
      <td>-0.739372</td>
      <td>4.679368</td>
      <td>-5.967278</td>
      <td>...</td>
      <td>0.205970</td>
      <td>0.869975</td>
      <td>15.028113</td>
      <td>0.364559</td>
      <td>0.530884</td>
      <td>22.119766</td>
      <td>5.280116</td>
      <td>-4.113606</td>
      <td>-2.367438</td>
      <td>0.865415</td>
    </tr>
    <tr>
      <th>A01_58</th>
      <td>4.092310</td>
      <td>0.929132</td>
      <td>-0.885145</td>
      <td>0.854615</td>
      <td>-1.651898</td>
      <td>1.106198</td>
      <td>2.308981</td>
      <td>-1.584598</td>
      <td>2.258720</td>
      <td>6.459206</td>
      <td>...</td>
      <td>0.709761</td>
      <td>0.793777</td>
      <td>17.961373</td>
      <td>0.583428</td>
      <td>-0.001954</td>
      <td>27.061697</td>
      <td>-4.836116</td>
      <td>3.824283</td>
      <td>-2.061558</td>
      <td>-3.699731</td>
    </tr>
    <tr>
      <th>A01_59</th>
      <td>-7.267147</td>
      <td>0.317266</td>
      <td>0.353902</td>
      <td>0.173577</td>
      <td>3.859513</td>
      <td>-1.350063</td>
      <td>0.048835</td>
      <td>-1.142921</td>
      <td>0.293651</td>
      <td>-2.481264</td>
      <td>...</td>
      <td>0.165374</td>
      <td>1.366814</td>
      <td>16.484308</td>
      <td>-0.533237</td>
      <td>-0.487195</td>
      <td>24.985005</td>
      <td>0.075893</td>
      <td>2.709230</td>
      <td>2.936690</td>
      <td>5.137446</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>A11_44</th>
      <td>-7.455436</td>
      <td>0.065010</td>
      <td>0.708960</td>
      <td>-0.303781</td>
      <td>1.814606</td>
      <td>5.638196</td>
      <td>0.224649</td>
      <td>1.198512</td>
      <td>-1.343692</td>
      <td>-4.435628</td>
      <td>...</td>
      <td>-0.749211</td>
      <td>-0.148476</td>
      <td>17.105216</td>
      <td>0.123122</td>
      <td>NaN</td>
      <td>22.720722</td>
      <td>-2.992506</td>
      <td>-2.502814</td>
      <td>-1.839947</td>
      <td>5.751830</td>
    </tr>
    <tr>
      <th>A11_45</th>
      <td>2.907273</td>
      <td>0.377561</td>
      <td>0.008094</td>
      <td>1.516315</td>
      <td>0.471784</td>
      <td>-6.231109</td>
      <td>-3.230581</td>
      <td>-0.941268</td>
      <td>-11.222300</td>
      <td>-4.671810</td>
      <td>...</td>
      <td>0.790916</td>
      <td>0.722727</td>
      <td>16.467980</td>
      <td>1.132648</td>
      <td>NaN</td>
      <td>22.521577</td>
      <td>1.643338</td>
      <td>2.459915</td>
      <td>-5.901576</td>
      <td>-4.664778</td>
    </tr>
    <tr>
      <th>A11_46</th>
      <td>-6.928941</td>
      <td>0.077312</td>
      <td>0.227317</td>
      <td>1.109649</td>
      <td>2.180708</td>
      <td>-1.061959</td>
      <td>0.614301</td>
      <td>1.245527</td>
      <td>4.237199</td>
      <td>3.007887</td>
      <td>...</td>
      <td>0.791549</td>
      <td>1.060520</td>
      <td>15.742954</td>
      <td>0.688456</td>
      <td>NaN</td>
      <td>22.512751</td>
      <td>0.454164</td>
      <td>-2.993857</td>
      <td>2.341450</td>
      <td>5.118265</td>
    </tr>
    <tr>
      <th>A11_48</th>
      <td>7.183339</td>
      <td>0.983350</td>
      <td>2.090029</td>
      <td>0.001171</td>
      <td>0.420704</td>
      <td>-2.660331</td>
      <td>1.417940</td>
      <td>1.470723</td>
      <td>1.243041</td>
      <td>-7.825930</td>
      <td>...</td>
      <td>-0.111029</td>
      <td>-0.589826</td>
      <td>15.758187</td>
      <td>0.727027</td>
      <td>NaN</td>
      <td>22.681465</td>
      <td>-1.544399</td>
      <td>1.773595</td>
      <td>-3.088895</td>
      <td>-3.579066</td>
    </tr>
    <tr>
      <th>A11_51</th>
      <td>7.633948</td>
      <td>0.077103</td>
      <td>0.703641</td>
      <td>0.419895</td>
      <td>0.559865</td>
      <td>-6.942340</td>
      <td>1.111722</td>
      <td>0.174473</td>
      <td>2.517062</td>
      <td>-1.057514</td>
      <td>...</td>
      <td>0.772863</td>
      <td>-0.873542</td>
      <td>16.774581</td>
      <td>-0.384493</td>
      <td>NaN</td>
      <td>23.893079</td>
      <td>-0.050615</td>
      <td>-1.412976</td>
      <td>-6.528134</td>
      <td>7.276401</td>
    </tr>
    <tr>
      <th>A11_52</th>
      <td>-8.027743</td>
      <td>0.342941</td>
      <td>0.207606</td>
      <td>0.777800</td>
      <td>1.855766</td>
      <td>-6.155187</td>
      <td>0.929895</td>
      <td>-0.289071</td>
      <td>0.662585</td>
      <td>6.697976</td>
      <td>...</td>
      <td>0.621308</td>
      <td>0.035059</td>
      <td>16.362545</td>
      <td>0.487377</td>
      <td>NaN</td>
      <td>22.642174</td>
      <td>-0.752784</td>
      <td>-3.310424</td>
      <td>4.157145</td>
      <td>-5.710500</td>
    </tr>
    <tr>
      <th>A11_54</th>
      <td>-8.525728</td>
      <td>1.328282</td>
      <td>0.473115</td>
      <td>0.388526</td>
      <td>0.823639</td>
      <td>-6.179502</td>
      <td>0.822936</td>
      <td>-1.103175</td>
      <td>7.015429</td>
      <td>-4.235858</td>
      <td>...</td>
      <td>0.399033</td>
      <td>0.443633</td>
      <td>16.321728</td>
      <td>-0.326466</td>
      <td>NaN</td>
      <td>23.076351</td>
      <td>-3.822596</td>
      <td>-2.623999</td>
      <td>-6.319933</td>
      <td>-4.721783</td>
    </tr>
    <tr>
      <th>A11_57</th>
      <td>6.346367</td>
      <td>0.167243</td>
      <td>2.454003</td>
      <td>0.309949</td>
      <td>1.359860</td>
      <td>-0.526673</td>
      <td>2.606971</td>
      <td>0.754974</td>
      <td>1.809343</td>
      <td>-4.754368</td>
      <td>...</td>
      <td>-2.912437</td>
      <td>0.611290</td>
      <td>18.938192</td>
      <td>1.074502</td>
      <td>NaN</td>
      <td>22.313950</td>
      <td>-0.262279</td>
      <td>4.058366</td>
      <td>6.746140</td>
      <td>-5.175894</td>
    </tr>
    <tr>
      <th>A11_58</th>
      <td>-7.285724</td>
      <td>0.583391</td>
      <td>-0.163537</td>
      <td>-1.535646</td>
      <td>0.151295</td>
      <td>-6.545795</td>
      <td>NaN</td>
      <td>-0.749599</td>
      <td>1.642822</td>
      <td>2.715098</td>
      <td>...</td>
      <td>-0.324229</td>
      <td>-1.474309</td>
      <td>16.352664</td>
      <td>-0.068981</td>
      <td>NaN</td>
      <td>23.206375</td>
      <td>-4.699755</td>
      <td>0.792384</td>
      <td>-3.585440</td>
      <td>2.196232</td>
    </tr>
    <tr>
      <th>A11_60</th>
      <td>7.598761</td>
      <td>0.730103</td>
      <td>0.828262</td>
      <td>0.347327</td>
      <td>0.161159</td>
      <td>-8.610313</td>
      <td>2.052813</td>
      <td>-0.986483</td>
      <td>-8.726147</td>
      <td>-7.935309</td>
      <td>...</td>
      <td>-0.000856</td>
      <td>-0.865281</td>
      <td>15.641259</td>
      <td>1.351713</td>
      <td>NaN</td>
      <td>22.050501</td>
      <td>-2.512252</td>
      <td>2.951493</td>
      <td>-1.999519</td>
      <td>-8.702733</td>
    </tr>
    <tr>
      <th>A11_61</th>
      <td>-6.880364</td>
      <td>0.862238</td>
      <td>NaN</td>
      <td>-0.426329</td>
      <td>4.559201</td>
      <td>-8.452262</td>
      <td>0.926557</td>
      <td>2.548554</td>
      <td>1.555138</td>
      <td>-3.631661</td>
      <td>...</td>
      <td>0.768317</td>
      <td>-1.471348</td>
      <td>17.081808</td>
      <td>-0.140308</td>
      <td>NaN</td>
      <td>25.479476</td>
      <td>-0.750635</td>
      <td>3.074695</td>
      <td>NaN</td>
      <td>5.849204</td>
    </tr>
    <tr>
      <th>A11_62</th>
      <td>-8.045053</td>
      <td>0.069826</td>
      <td>-0.942733</td>
      <td>-0.277029</td>
      <td>0.725625</td>
      <td>-7.727957</td>
      <td>-0.064185</td>
      <td>-0.066813</td>
      <td>0.169675</td>
      <td>-3.480213</td>
      <td>...</td>
      <td>-0.039708</td>
      <td>-0.499990</td>
      <td>19.067704</td>
      <td>-0.319564</td>
      <td>NaN</td>
      <td>25.226514</td>
      <td>-4.369941</td>
      <td>3.546102</td>
      <td>-7.820512</td>
      <td>-6.362555</td>
    </tr>
    <tr>
      <th>A11_64</th>
      <td>-8.452961</td>
      <td>1.313303</td>
      <td>-0.180174</td>
      <td>-0.395152</td>
      <td>-3.073554</td>
      <td>11.475177</td>
      <td>-3.706662</td>
      <td>1.467159</td>
      <td>3.371451</td>
      <td>-4.024469</td>
      <td>...</td>
      <td>-1.072126</td>
      <td>-0.430788</td>
      <td>16.292964</td>
      <td>0.687982</td>
      <td>NaN</td>
      <td>24.538142</td>
      <td>2.667982</td>
      <td>2.440611</td>
      <td>6.249088</td>
      <td>8.189521</td>
    </tr>
    <tr>
      <th>A11_66</th>
      <td>9.448724</td>
      <td>0.311918</td>
      <td>0.164296</td>
      <td>1.179131</td>
      <td>1.776497</td>
      <td>-6.723817</td>
      <td>NaN</td>
      <td>3.236887</td>
      <td>3.533954</td>
      <td>-2.685802</td>
      <td>...</td>
      <td>1.692752</td>
      <td>0.418787</td>
      <td>15.430309</td>
      <td>1.211189</td>
      <td>NaN</td>
      <td>22.224755</td>
      <td>0.956364</td>
      <td>-0.765028</td>
      <td>-0.505661</td>
      <td>-4.559104</td>
    </tr>
    <tr>
      <th>A11_68</th>
      <td>-8.824487</td>
      <td>0.908554</td>
      <td>0.693707</td>
      <td>0.258778</td>
      <td>3.071328</td>
      <td>12.956824</td>
      <td>0.565678</td>
      <td>0.980528</td>
      <td>-4.295115</td>
      <td>-3.894834</td>
      <td>...</td>
      <td>1.005061</td>
      <td>0.611755</td>
      <td>18.096924</td>
      <td>-0.249243</td>
      <td>NaN</td>
      <td>24.928409</td>
      <td>4.742218</td>
      <td>-2.630358</td>
      <td>5.647873</td>
      <td>9.762167</td>
    </tr>
    <tr>
      <th>A11_69</th>
      <td>-5.625713</td>
      <td>0.622860</td>
      <td>-0.599341</td>
      <td>-0.301774</td>
      <td>0.571811</td>
      <td>11.847581</td>
      <td>0.201365</td>
      <td>0.066659</td>
      <td>2.859140</td>
      <td>-4.665133</td>
      <td>...</td>
      <td>-0.211785</td>
      <td>-0.645943</td>
      <td>16.710638</td>
      <td>-0.341715</td>
      <td>NaN</td>
      <td>23.724599</td>
      <td>1.796232</td>
      <td>0.607982</td>
      <td>0.693608</td>
      <td>6.862298</td>
    </tr>
    <tr>
      <th>A11_72</th>
      <td>-9.219612</td>
      <td>0.579977</td>
      <td>0.901603</td>
      <td>0.680549</td>
      <td>1.497327</td>
      <td>1.119451</td>
      <td>-2.112560</td>
      <td>-1.144666</td>
      <td>-0.532813</td>
      <td>-9.513557</td>
      <td>...</td>
      <td>-0.582535</td>
      <td>-1.300570</td>
      <td>16.447684</td>
      <td>0.858761</td>
      <td>NaN</td>
      <td>23.037218</td>
      <td>0.305084</td>
      <td>1.587400</td>
      <td>-1.889495</td>
      <td>-10.516313</td>
    </tr>
    <tr>
      <th>A11_73</th>
      <td>-5.636734</td>
      <td>0.453751</td>
      <td>NaN</td>
      <td>-1.238875</td>
      <td>-1.676514</td>
      <td>3.115243</td>
      <td>-5.534217</td>
      <td>-0.918070</td>
      <td>-6.196405</td>
      <td>-4.306319</td>
      <td>...</td>
      <td>-0.095414</td>
      <td>0.419458</td>
      <td>15.222614</td>
      <td>-0.870604</td>
      <td>NaN</td>
      <td>21.546315</td>
      <td>2.459713</td>
      <td>3.235806</td>
      <td>-3.884509</td>
      <td>4.981810</td>
    </tr>
    <tr>
      <th>A11_77</th>
      <td>-8.798892</td>
      <td>0.301123</td>
      <td>-0.323830</td>
      <td>0.432050</td>
      <td>2.895317</td>
      <td>10.033640</td>
      <td>0.595472</td>
      <td>-0.957188</td>
      <td>4.284933</td>
      <td>-3.969501</td>
      <td>...</td>
      <td>0.783732</td>
      <td>2.010373</td>
      <td>18.009701</td>
      <td>-0.453162</td>
      <td>NaN</td>
      <td>25.167154</td>
      <td>-1.830648</td>
      <td>-2.582527</td>
      <td>6.232993</td>
      <td>-2.549221</td>
    </tr>
    <tr>
      <th>A11_78</th>
      <td>-4.061045</td>
      <td>0.153991</td>
      <td>0.749857</td>
      <td>0.981008</td>
      <td>0.952446</td>
      <td>-7.626006</td>
      <td>0.546279</td>
      <td>1.417278</td>
      <td>2.061099</td>
      <td>-4.090833</td>
      <td>...</td>
      <td>0.792770</td>
      <td>2.142495</td>
      <td>16.595233</td>
      <td>0.516697</td>
      <td>NaN</td>
      <td>25.390230</td>
      <td>5.181662</td>
      <td>-2.575708</td>
      <td>12.377388</td>
      <td>8.398114</td>
    </tr>
    <tr>
      <th>A11_79</th>
      <td>-0.940648</td>
      <td>0.348818</td>
      <td>0.462085</td>
      <td>0.467866</td>
      <td>5.958515</td>
      <td>5.921017</td>
      <td>NaN</td>
      <td>0.267799</td>
      <td>2.001825</td>
      <td>2.009163</td>
      <td>...</td>
      <td>0.331876</td>
      <td>2.086463</td>
      <td>17.236369</td>
      <td>1.208932</td>
      <td>NaN</td>
      <td>24.567289</td>
      <td>4.332028</td>
      <td>-2.421702</td>
      <td>2.236019</td>
      <td>8.882389</td>
    </tr>
    <tr>
      <th>A11_80</th>
      <td>6.014177</td>
      <td>0.634675</td>
      <td>-0.431921</td>
      <td>1.064627</td>
      <td>-4.663311</td>
      <td>-7.117807</td>
      <td>1.670899</td>
      <td>-0.385115</td>
      <td>-4.388628</td>
      <td>-5.171900</td>
      <td>...</td>
      <td>-0.042121</td>
      <td>-0.469607</td>
      <td>16.598856</td>
      <td>0.260251</td>
      <td>NaN</td>
      <td>23.133193</td>
      <td>-1.978838</td>
      <td>2.192267</td>
      <td>-1.833746</td>
      <td>-3.248339</td>
    </tr>
    <tr>
      <th>A11_85</th>
      <td>NaN</td>
      <td>0.327573</td>
      <td>NaN</td>
      <td>-0.645948</td>
      <td>-2.031381</td>
      <td>7.609264</td>
      <td>-0.660415</td>
      <td>1.255510</td>
      <td>5.098531</td>
      <td>-4.039771</td>
      <td>...</td>
      <td>-0.470322</td>
      <td>0.624881</td>
      <td>15.913935</td>
      <td>0.221976</td>
      <td>NaN</td>
      <td>20.968383</td>
      <td>1.705758</td>
      <td>6.654459</td>
      <td>10.226067</td>
      <td>-1.580349</td>
    </tr>
    <tr>
      <th>A11_86</th>
      <td>3.189294</td>
      <td>2.307381</td>
      <td>NaN</td>
      <td>1.684777</td>
      <td>0.510847</td>
      <td>-1.193390</td>
      <td>-1.424114</td>
      <td>-0.993333</td>
      <td>-3.922912</td>
      <td>1.348560</td>
      <td>...</td>
      <td>-3.327925</td>
      <td>0.331371</td>
      <td>15.066564</td>
      <td>0.038268</td>
      <td>NaN</td>
      <td>16.979292</td>
      <td>-0.394850</td>
      <td>5.130092</td>
      <td>3.422867</td>
      <td>-0.946073</td>
    </tr>
    <tr>
      <th>A11_87</th>
      <td>-5.011391</td>
      <td>0.101890</td>
      <td>1.089817</td>
      <td>-1.230890</td>
      <td>-3.135374</td>
      <td>-7.961377</td>
      <td>0.164554</td>
      <td>0.359548</td>
      <td>-6.488684</td>
      <td>-4.194792</td>
      <td>...</td>
      <td>0.180122</td>
      <td>-0.699413</td>
      <td>16.162823</td>
      <td>-0.427215</td>
      <td>NaN</td>
      <td>22.939438</td>
      <td>1.632431</td>
      <td>-4.138147</td>
      <td>-3.984612</td>
      <td>-3.358585</td>
    </tr>
    <tr>
      <th>A11_89</th>
      <td>-0.781114</td>
      <td>1.079321</td>
      <td>0.219939</td>
      <td>0.771153</td>
      <td>1.955114</td>
      <td>5.559785</td>
      <td>4.924384</td>
      <td>2.472066</td>
      <td>2.813091</td>
      <td>14.695409</td>
      <td>...</td>
      <td>-0.372606</td>
      <td>0.096194</td>
      <td>16.820080</td>
      <td>0.619160</td>
      <td>NaN</td>
      <td>24.101895</td>
      <td>3.574523</td>
      <td>2.409968</td>
      <td>4.127189</td>
      <td>15.004739</td>
    </tr>
    <tr>
      <th>A11_90</th>
      <td>NaN</td>
      <td>0.645842</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>-0.890742</td>
      <td>13.016383</td>
      <td>-5.268835</td>
      <td>-0.468322</td>
      <td>2.211619</td>
      <td>-4.457251</td>
      <td>...</td>
      <td>0.058875</td>
      <td>-1.051974</td>
      <td>17.411414</td>
      <td>-0.035335</td>
      <td>NaN</td>
      <td>25.180528</td>
      <td>-1.240811</td>
      <td>3.017025</td>
      <td>-2.024357</td>
      <td>-5.818075</td>
    </tr>
    <tr>
      <th>A11_91</th>
      <td>-6.343767</td>
      <td>0.438523</td>
      <td>-0.322617</td>
      <td>1.973411</td>
      <td>4.515720</td>
      <td>-8.495414</td>
      <td>NaN</td>
      <td>-0.224670</td>
      <td>3.258043</td>
      <td>2.732642</td>
      <td>...</td>
      <td>0.114539</td>
      <td>1.206215</td>
      <td>16.871327</td>
      <td>1.084758</td>
      <td>NaN</td>
      <td>22.730956</td>
      <td>4.994928</td>
      <td>-2.030958</td>
      <td>-0.282208</td>
      <td>8.456099</td>
    </tr>
    <tr>
      <th>A11_93</th>
      <td>-10.147285</td>
      <td>1.145573</td>
      <td>0.048948</td>
      <td>-1.089350</td>
      <td>0.539833</td>
      <td>2.720952</td>
      <td>-3.445357</td>
      <td>0.891066</td>
      <td>3.003684</td>
      <td>-1.667592</td>
      <td>...</td>
      <td>-0.317916</td>
      <td>1.977915</td>
      <td>15.687951</td>
      <td>-0.098451</td>
      <td>NaN</td>
      <td>21.553930</td>
      <td>2.977722</td>
      <td>-1.913176</td>
      <td>4.279626</td>
      <td>-4.650492</td>
    </tr>
    <tr>
      <th>A11_94</th>
      <td>NaN</td>
      <td>0.321844</td>
      <td>NaN</td>
      <td>1.016712</td>
      <td>1.801923</td>
      <td>-4.157083</td>
      <td>-0.097598</td>
      <td>-1.954080</td>
      <td>-1.739499</td>
      <td>-6.520551</td>
      <td>...</td>
      <td>-1.698334</td>
      <td>1.363267</td>
      <td>17.357844</td>
      <td>0.390511</td>
      <td>NaN</td>
      <td>23.977236</td>
      <td>0.078992</td>
      <td>1.801422</td>
      <td>NaN</td>
      <td>-0.499069</td>
    </tr>
  </tbody>
</table>
<p>534 rows × 46 columns</p>
</div>



To check that there are no values below 0 run the *min()* function again


```python
df[df.Caffeine > 0].Caffeine.min()
```




    0.0032293928665531



Some of our values have missing values that are represented in the output as `NaN`. Often we can't work with them. One way to deal with them is to drop the rows that contain them. First let's check how many missing values we have.


```python
print(df.Caffeine[df.Caffeine.isnull()])
```

    A05_12   NaN
    A09_65   NaN
    A10_01   NaN
    A11_96   NaN
    Name: Caffeine, dtype: float64



```python
df.dropna(subset=['Caffeine'], inplace=True)
print(df.Caffeine[df.Caffeine.isnull()])
```

    Series([], Name: Caffeine, dtype: float64)


### Plotting pandas dataframes

The *DataFrame* has a built in plotting functions (i.e `pd.hist()` or `pd.plot()`) that allows to create simple plots using matplotlib internally. For more advanced plotting functions, one can use the *matplotlib* library directly.


```python
df[['Caffeine','Copper']].hist()
```


```python
df[['Caffeine','Copper']].plot(kind='hist')
```

---
## Matplotlib

*Matplotlib* is a Python package used for data plotting and visualisation. It is a useful complement to Pandas, and like Pandas, is a very feature-rich library which can produce a large variety of plots, charts, maps, and other visualisations.  
`matplotlib.pyplot` is a collection of functions that make matplotlib work like *MATLAB*. Majority of plotting commands in `pyplot` have *MATLAB* analogs with similar arguments. Let us take a couple of examples.  

Typically, if we are going to work on some plotting, we would import matplotlib like this:


```python
%matplotlib inline
import matplotlib.pyplot as plt
```


```python
# Example 1: Plotting a line graph
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
```


```python
# Example 2: Plotting a histogram
x = [21,22,23,4,5,6,77,8,9,10,31,32,33,34,35,36,37,18,49,50,100]
num_bins = 5
plt.hist(x, num_bins, facecolor='blue')
plt.show()
```

Let us try to plot our phenotype data.  Let’s have a look at the relationship between two phenotypes in our dataframe. To save typing later on, we can extract the Series for “Caffeine” and “Copper” by assigning each to a new variable, x and y, respectively.


```python
df.dropna(inplace=True)
x = df.Sorbitol
y = df.Trehalose
```

Then we can plot them as a scatter chart by adding:


```python
plt.scatter(x, y)
# or plt.savefig("name.png")
```

Now we are going to do some basic statistics so we can plot a line of linear regression on our chart.  
Let’s plot a linear regression through the data. Python has a library called scipy that contains a lot of statistics routines. We can import it:


```python
from scipy.stats import linregress
```

To get the statistics for the linear regression we can use a function called `linregress`


```python
stats = linregress(x, y)
m = stats.slope
b = stats.intercept
stats
```




    LinregressResult(slope=0.9622331502773507, intercept=-0.011936070484259736, rvalue=0.9064525118355159, pvalue=1.0446912214962477e-114, stderr=0.025839319479636295)



The equation for the straight line that describes linear regression is `y = mx + b`, where m is the *slope* and `b` is the *intercept*.  
(Hint: you have some extra information in the stats object - stats.rvalue and stats.pvalue.)

Therefore, we can then plot the line of linear regression by adding the following line:


```python
plt.scatter(x, y)
plt.plot(x, m * x + b, color="r")   # I've added a color argument here
```

Matplotlib figures are highly customisable, and there are so many options it is usually best to consult the [documentation]() first.  
We will take the above plot and customize *fontsize*, *linewidth*, *color* for example:


```python
# Change the default figure size
plt.figure(figsize=(10,10))

# Change the default marker for the scatter from circles to x's
plt.scatter(x, y, marker='x')

# Set the linewidth on the regression line to 3px
plt.plot(x, m * x + b, color="red", linewidth=3)

# Add x and y lables, and set their font size
plt.xlabel("Sorbitol", fontsize=20)
plt.ylabel("Trehalose", fontsize=20)

# Plot pvalue and r2 value
plt.text(-2.5, 3, 'pvalue: %s' % stats.pvalue,
         fontsize=14)
plt.text(-2.5, 2.8, 'r2: %s' % stats.rvalue,
         fontsize=14)

# Set the font size of the number lables on the axes
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.show()
```


```python

```

6d93b4f46ea15a84eaf0d53e8cebb176f896ad4f
ipynb
Jupyter Notebook
gwas-lecture-master/Lecture-1-Python-Data-Science-Guide.ipynb
krishdb38/Bio_python
e20ba5579e1e6b4ecf9fe72ef9e21a5960416ae4
MIT
gwas-lecture-master/Lecture-1-Python-Data-Science-Guide.ipynb
krishdb38/Bio_python
e20ba5579e1e6b4ecf9fe72ef9e21a5960416ae4
MIT
gwas-lecture-master/Lecture-1-Python-Data-Science-Guide.ipynb
krishdb38/Bio_python
e20ba5579e1e6b4ecf9fe72ef9e21a5960416ae4
MIT
Qwen/Qwen-72B
1. YES
2. YES