# Quantitative exploratory data analysis
* Univariate EDA for a quantitative variable is a way to make preliminary assessments about the population distribution of he variable using the data of the observed sample.
* The characteristics of the population distribution of a quantitative variable are its center, spread, modality (number of peaks in the pdf), shape (including “heaviness of the tails”), and outliers. 
* Our observed data represent just one sample out of an infinite number of possible samples. 
* The characteristics of our randomly observed sample are not inherently interesting, except to the degree that they represent the population that it came from.
* What we observe in the sample of measurements for a particular variable that  we select for our particular experiment is the “sample distribution”.
* We need to recognize that this would be different each time we might repeat the same experiment, due to selection of a different random sample, a different treatment randomization, and different random (incompletely controlled) experimental conditions. 
* In addition we can calculate “sample statistics” from the data, such as sample mean, sample variance, sample standard deviation, sample skewness and sample kurtosis.
* These again would vary for each repetition of the experiment, so they don’t represent any deep truth, but rather represent some uncertain information about the underlying population distribution and its parameters, which are what we really care about.

<b>

1. Introduction to summary statistics: The sample mean and median
    1. Computing means
2. Percentiles, outliers, and box plots
    1. Compute percentiles
    2. Comparing percentiles to ECDF
    3. Box-and-whisker plot
3. Variance and standard deviation
    1. Computing the variance
    2. The standard deviation and the variance
4. Covariance and the Pearson correlation coefficient
    1. Scatter plots
    2. Computing the covariance
    3. Computing the Pearson correlation coefficient


</b>


```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```


```
path = "https://raw.githubusercontent.com/reddyprasade/Data-Analysis-with-Python/main/Statistics/Data/iris_lables.csv"
```


```
sns.set()

df = pd.read_csv(path)
renamed_columns = ['sepal length (cm)', 'sepal width (cm)', 
                   'petal length (cm)', 'petal width (cm)', 'species']
df.columns = renamed_columns


versicolor_petal_length = df[df['species'] == 'Versicolor']['petal length (cm)']
setosa_petal_length = df[df['species'] == 'Setosa']['petal length (cm)']
virginica_petal_length = df[df['species'] == 'Virginica']['petal length (cm)']
versicolor_petal_width = df[df['species'] == 'Versicolor']['petal width (cm)']
```


```
df.columns = renamed_columns
df.head()
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>sepal length (cm)</th>
      <th>sepal width (cm)</th>
      <th>petal length (cm)</th>
      <th>petal width (cm)</th>
      <th>species</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>5.1</td>
      <td>3.5</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Setosa</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4.9</td>
      <td>3.0</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Setosa</td>
    </tr>
    <tr>
      <th>2</th>
      <td>4.7</td>
      <td>3.2</td>
      <td>1.3</td>
      <td>0.2</td>
      <td>Setosa</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4.6</td>
      <td>3.1</td>
      <td>1.5</td>
      <td>0.2</td>
      <td>Setosa</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5.0</td>
      <td>3.6</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Setosa</td>
    </tr>
  </tbody>
</table>
</div>



```
df.tail()
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>sepal length (cm)</th>
      <th>sepal width (cm)</th>
      <th>petal length (cm)</th>
      <th>petal width (cm)</th>
      <th>species</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>145</th>
      <td>6.7</td>
      <td>3.0</td>
      <td>5.2</td>
      <td>2.3</td>
      <td>Virginica</td>
    </tr>
    <tr>
      <th>146</th>
      <td>6.3</td>
      <td>2.5</td>
      <td>5.0</td>
      <td>1.9</td>
      <td>Virginica</td>
    </tr>
    <tr>
      <th>147</th>
      <td>6.5</td>
      <td>3.0</td>
      <td>5.2</td>
      <td>2.0</td>
      <td>Virginica</td>
    </tr>
    <tr>
      <th>148</th>
      <td>6.2</td>
      <td>3.4</td>
      <td>5.4</td>
      <td>2.3</td>
      <td>Virginica</td>
    </tr>
    <tr>
      <th>149</th>
      <td>5.9</td>
      <td>3.0</td>
      <td>5.1</td>
      <td>1.8</td>
      <td>Virginica</td>
    </tr>
  </tbody>
</table>
</div>



```
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 150 entries, 0 to 149
    Data columns (total 5 columns):
     #   Column             Non-Null Count  Dtype  
    ---  ------             --------------  -----  
     0   sepal length (cm)  150 non-null    float64
     1   sepal width (cm)   150 non-null    float64
     2   petal length (cm)  150 non-null    float64
     3   petal width (cm)   150 non-null    float64
     4   species            150 non-null    object 
    dtypes: float64(4), object(1)
    memory usage: 6.0+ KB



```
df.describe()
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>sepal length (cm)</th>
      <th>sepal width (cm)</th>
      <th>petal length (cm)</th>
      <th>petal width (cm)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>150.000000</td>
      <td>150.000000</td>
      <td>150.000000</td>
      <td>150.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>5.843333</td>
      <td>3.057333</td>
      <td>3.758000</td>
      <td>1.199333</td>
    </tr>
    <tr>
      <th>std</th>
      <td>0.828066</td>
      <td>0.435866</td>
      <td>1.765298</td>
      <td>0.762238</td>
    </tr>
    <tr>
      <th>min</th>
      <td>4.300000</td>
      <td>2.000000</td>
      <td>1.000000</td>
      <td>0.100000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>5.100000</td>
      <td>2.800000</td>
      <td>1.600000</td>
      <td>0.300000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>5.800000</td>
      <td>3.000000</td>
      <td>4.350000</td>
      <td>1.300000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>6.400000</td>
      <td>3.300000</td>
      <td>5.100000</td>
      <td>1.800000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>7.900000</td>
      <td>4.400000</td>
      <td>6.900000</td>
      <td>2.500000</td>
    </tr>
  </tbody>
</table>
</div>



```
df.isnull().sum()
```


    sepal length (cm)    0
    sepal width (cm)     0
    petal length (cm)    0
    petal width (cm)     0
    species              0
    dtype: int64


1. Introduction to summary statistics: The sample mean and median
    1. Mean
    2. Median
    3. Variance
    4. Stand Devation
The mean of all measurements gives an indication of the typical magnitude of a measurement. It is computed using np.mean(). 


```
# compute the mean: mean_length_vers
mean_length_vers = np.mean(setosa_petal_length)

# Print the result with some nice formatting
print('I. Setosa:', mean_length_vers, 'cm')
```

    I. Setosa: 1.4620000000000002 cm



```
# compute the mean: mean_length_vers
mean_length_vers = np.mean(versicolor_petal_length)

# Print the result with some nice formatting
print('I. versicolor:', mean_length_vers, 'cm')
```

    I. versicolor: 4.26 cm



```

std_length_vers = np.std(versicolor_petal_length)

# Print the result with some nice formatting
print('I. versicolor:', std_length_vers, 'cm')
```

    I. versicolor: 0.4651881339845204 cm



```

var_length_vers = np.var(versicolor_petal_length)

# Print the result with some nice formatting
print('I. versicolor:', var_length_vers, 'cm')
```

    I. versicolor: 0.21640000000000012 cm



```

median_length_vers = np.median(versicolor_petal_length)

# Print the result with some nice formatting
print('I. versicolor:', median_length_vers, 'cm')
```

    I. versicolor: 4.35 cm


## Percentiles, outliers, and box plots

### Compute percentiles
In this exercise, you will compute the percentiles of petal length of Iris versicolor.



```
# Specify array of percentiles: percentiles
percentiles = np.array([2.5, 25, 50, 75, 97.5])

# Compute percentiles: ptiles_vers
ptiles_vers = np.percentile(versicolor_petal_length, percentiles)

# Print the result
print(ptiles_vers)
```

    [3.3    4.     4.35   4.6    4.9775]


### Comparing percentiles to ECDF
To see how the percentiles relate to the ECDF, you will plot the percentiles of Iris versicolor petal lengths you calculated in the last exercise on the ECDF plot you generated in chapter 1. 

 Note that to ensure the Y-axis of the ECDF plot remains between 0 and 1, you will need to rescale the percentiles array accordingly - in this case, dividing it by 100.


```
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n= len(data)
    #x data foe ecdf
    x = np.sort(data)
    # y data for ecdf
    y = np.arange(1,n+1)/n

    return x,y
```


```
x_vers ,y_vers  = ecdf(versicolor_petal_length)
```


```
# Plot the ECDF
plt.figure(figsize=(20,9))
_ = plt.plot(x_vers, y_vers, '.')
_ = plt.xlabel('petal length (cm)')
_ = plt.ylabel('ECDF')

_ = plt.plot(ptiles_vers,percentiles/100, marker='D', color='red', linestyle='none')
```

### Box-and-whisker plot
Making a box plot for the petal lengths is unnecessary because the iris data set is not too large and the bee swarm plot works fine. However, it is always good to get some practice. Make a box plot of the iris petal lengths. You have a pandas DataFrame, df, which contains the petal length data, in your namespace. Inspect the data frame df in the IPython shell using ```df.head()``` to make sure you know what the pertinent columns are.

For your reference, the code used to produce the box plot in the video is provided below:
```python
_ = sns.boxplot(x='east_west', y='dem_share', data=df_all_states)
_ = plt.xlabel('region')
_ = plt.ylabel('percent of vote for Obama')
```
In the IPython Shell, you can use ```sns.boxplot?``` or ```help(sns.boxplot)``` for more details on how to make box plots using seaborn.


```
plt.Figure(figsize=(20,9))
_ = sns.boxplot(x='species', y='petal length (cm)', data=df)
_ = plt.xlabel('species')
_ = plt.ylabel('petal length (cm)')
```

## Variance and standard deviation

$$ variance = \dfrac{1}{n}\sum^{n}_{i=1}(x_i - \bar{x})^2 $$

### Computing the variance
It is important to have some understanding of what commonly-used functions are doing under the hood. Though you may already know how to compute variances, this is a beginner course that does not assume so. In this exercise, we will explicitly compute the variance of the petal length of Iris veriscolor using the equations discussed in the videos. We will then use ```np.var()``` to compute it.


```
# Array of differnces to mean: differences
differences = np.array(versicolor_petal_length - np.mean(versicolor_petal_length))
differences
```


```
# Square the differences: diff_sq
diff_sq = differences ** 2
diff_sq
```


```
# Compute the mean square differences: variance_explicit
variance_explicit = np.mean(diff_sq)
variance_explicit
```


    0.21640000000000004



```
# Compute the variance using NumPy: variance_np
variance_np = np.var(differences)
variance_np
```


    0.21640000000000004


### The standard deviation and the variance
As mentioned in the video, the standard deviation is the square root of the variance. You will see this for yourself by computing the standard deviation using ```np.std()``` and comparing it to what you get by computing the variance with ```np.var()``` and then computing the square root.


```
# Compute the variance: variance
variance = np.var(versicolor_petal_length)

# Print the square root of the variance
print(np.sqrt(variance))

# Print the standard deviation
print(np.std(versicolor_petal_length))
```

    0.4651881339845204
    0.4651881339845204


## Covariance and the Pearson correlation coefficient

$$ covariance = \dfrac{1}{n}\sum^{n}_{i=1}(x_i - \bar{x})(y_i - \bar{y})$$
$$ \begin{align} \rho &= \text{Pearson correlation} = \dfrac{\text{covariance}}{(\text{std of x})(\text{std of y})} \\ &= \dfrac{\text{variability due to codependence}}{\text{independent variability}} \end{align}$$

### Scatter plots
When you made bee swarm plots, box plots, and ECDF plots in previous exercises, you compared the petal lengths of different species of iris. But what if you want to compare two properties of a single species? This is exactly what we will do in this exercise. We will make a scatter plot of the petal length and width measurements of Anderson's Iris versicolor flowers. If the flower scales (that is, it preserves its proportion as it grows), we would expect the length and width to be correlated.

For your reference, the code used to produce the scatter plot in the video is provided below:
```python
_ = plt.plot(total_votes/1000, dem_share, marker='.', linestyle='none')
_ = plt.xlabel('total votes (thousands)')
_ = plt.ylabel('percent of vote for Obama')
```


```
# Make a scatter plot
_ = plt.plot(versicolor_petal_length, versicolor_petal_width, marker='.', linestyle='none')

# Label the axes
_ = plt.xlabel('petal length')
_ = plt.ylabel('petal width')
```

### Computing the covariance
The covariance may be computed using the Numpy function ```np.cov()```. For example, we have two sets of data ```x``` and ```y```, ```np.cov(x, y)``` returns a 2D array where entries ```[0,1]``` and ```[1,0]``` are the covariances. Entry ```[0,0]``` is the variance of the data in x, and entry ```[1,1]``` is the variance of the data in y. This 2D output array is called the covariance matrix, since it organizes the self- and covariance.


```
# Compute the covariance matrix: covariance_matrix
covariance_matrix = np.cov(versicolor_petal_length, versicolor_petal_width)

# Print covariance matrix
print(covariance_matrix)

# Extract covariance of length and width of petals: petal_cov
petal_cov = covariance_matrix[0, 1]

# Print the length/width covariance
print(petal_cov)
```

    [[0.22081633 0.07310204]
     [0.07310204 0.03910612]]
    0.07310204081632653


### Computing the Pearson correlation coefficient
In this exercise, you will write a function, ```pearson_r(x, y)``` that takes in two arrays and returns the Pearson correlation coefficient. You will then use this function to compute it for the petal lengths and widths of I. versicolor.


```
def pearson_r(x, y):
    """Compute Pearson correlation coefficient between two arrays
    
    Args:
        x: arrays
        y: arrays
        
    returns:
        r: int
    """
    # Compute correlation matrix: corr_mat
    coor_mat = np.corrcoef(x,y)

    # return entry[0,1]
    return coor_mat[0,1]
```


```
r = pearson_r(versicolor_petal_length,versicolor_petal_width)
print(r)
```

    0.7866680885228169



```
pearson_r(versicolor_petal_length,virginica_petal_length)
```


    -0.08782058063891776



```

```

8332219ea3d77963678eb383c46974e212f6bee3
ipynb
Jupyter Notebook
Statistics/STAST-EN-2021-105-Quantitative exploratory data analysis.ipynb
reddyprasade/Data-Analysis-with-Python-
2440e23486856eea5556c8262467b3a618032bc2
MIT
2021-06-29T23:15:05.000Z
2021-06-29T23:15:05.000Z
Statistics/STAST-EN-2021-105-Quantitative exploratory data analysis.ipynb
reddyprasade/Data-Analysis-with-Python-
2440e23486856eea5556c8262467b3a618032bc2
MIT
Statistics/STAST-EN-2021-105-Quantitative exploratory data analysis.ipynb
reddyprasade/Data-Analysis-with-Python-
2440e23486856eea5556c8262467b3a618032bc2
MIT
2021-12-20T10:04:53.000Z
2021-12-20T10:04:53.000Z
Qwen/Qwen-72B
1. YES
2. YES