```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
from sympy.stats import E
import warnings
warnings.filterwarnings('ignore')
np.random.seed(0)
```

### 1. Population and Sample



```python
items = ['t-shirt', 'sleeveless', 'sweat', 'pull-over', 'pants',
'trousers', 'dress', 'jacket', 'nightdress', 'underwear']
pop = np.random.choice(items, size=10000, replace=True)
samp = np.random.choice(pop, 1000)

labels, pop_counts = np.unique(pop, return_counts=True)
_, samp_counts = np.unique(samp, return_counts=True)
```


```python
fig, axs = plt.subplots(nrows = 1, ncols = 2, figsize=(10,3))
ax1, ax2 = axs[0], axs[1]
fontdict = {'fontsize':17}
ax1.bar(labels, pop_counts, align='center')
ax1.set_xticklabels(labels, rotation=80)
ax1.set_xlabel('Population', fontdict=fontdict)
ax2.bar(labels, samp_counts, align='center')
ax2.set_xticklabels(labels, rotation=80)
ax2.set_xlabel('Sample', fontdict=fontdict)
plt.show()
```

### 2. Probability distributions


```python
probs = pop_counts/np.sum(pop_counts)
plt.bar(labels, probs)
plt.xticks(rotation=75)
plt.show()
```

### 3. Normal distribution



```python
def norm(x, mean, std):
    f = np.exp(-0.5 * ((x - mean)/std**2)**2) /(std * (np.sqrt(2*np.pi)))
    return f

x = np.linspace(-4, 4, 1000)
y = [norm(i, 0, 1) for i in x]
plt.plot(x,y)
plt.grid()
plt.show()
```

### 4. Measures of central tendency


```python
print(f'mean = {np.mean(pop_counts)}')
print(f'median = {np.median(pop_counts)}')
```

    mean = 1000.0
    median = 997.0


### 5. Measures of dispersion 


```python
print(f'std = {np.std(pop_counts):.2f}')
print(f'IQR = {stats.iqr(pop_counts)}')
print(f'range = {np.max(pop_counts) - np.min(pop_counts)}')
```

    std = 32.39
    IQR = 60.75
    range = 86


### 6. Expected value of a random variable



```python
print(f'E = {np.sum(pop_counts*probs):.2f}')
```

    E = 1001.05


### 7. Linear regression
#### 7.2. Example
We will create a randomly distributed dataset $x$ for which we will find the best linear model $y$ describing $x$, that is find the values of the **slope** $\beta_1$ and the **intercept** $\beta_0$ such that $y = \beta_0 + \beta_1x$.


```python
x = np.random.randn(100)
y = 1.5*x + np.random.default_rng().random(100)
res = stats.linregress(x, y)
y_hat = res.intercept + res.slope*x
plt.plot(x, y_hat, label='Data', color='red')
plt.scatter(x, y, label='Best fit')
plt.grid()
plt.legend()
plt.show()
```

### 8. Over sampling and under sampling


### 9. Probability
Let's consider the random experiment **Rolling a dice** and calculate the probabilities of few possible outcomes. Whatever the outcome, it is an integer that falls between 1 and 6, included.

In this case, $\Omega = \{1,2,3,4,5,6\}$. 

Consider the following events:
- A : "Roll a 3"
- B : "Roll an even digit"
- C : "Roll a digit les than or equal to 4"
- D : "Roll an odd digit greater tha 2"
- E : "Roll 7"


```python
omega = {1,2,3,4,5,6}
A = {3}
B = {2,4,6}
C = {1,2,3,4}
D = {3,5}
E = {}
print(f'P(A) = {len(A)}/{len(omega)}\nP(B) = {len(B)}/{len(omega)}\n\
P(C) = {len(C)}/{len(omega)}\nP(D) = {len(D)}/{len(omega)}\nP(E) = {len(E)/len(omega)}\n')
```

    P(A) = 1/6
    P(B) = 3/6
    P(C) = 4/6
    P(D) = 2/6
    P(E) = 0.0
    


### 10. Combinations and Permutations
### Examples
- a) Permutation : How many ways (noted X) can first, second and third place be awarded to 10 people? 
- b) Combination : In how many different ways (noted Y) a team of 3 people can be formed from a group of 10?


```python
# a) 
X = np.math.factorial(10)/np.math.factorial(10-3)
Y = np.math.factorial(10)/(np.math.factorial(10-3)*np.math.factorial(3))
print(f"X = {int(X)}\nY = {int(Y)}")
```

    X = 720
    Y = 120


## Conclusion
Data Science is a generic field of knowledge that leverages the power of advanced mathematics to draw useful insights from data. The tenants of mathematics for data science are statistics and probability theory. The concepts presented throughout this post are just few of a large set of concepts and tools statistics provide us with, to help data scientists succeed in our data story telling.  

You can find read more about this notebook [here](https://www.datainsightonline.com/post/10-must-know-statistical-concepts-for-data-science).



ebea90148f2df65273b8d240a468c574810086aa
ipynb
Jupyter Notebook
notebook.ipynb
tem-ctrl/statitistics_for_data_science
e7f654e9eec2b26657085dd7851187575ba86159
MIT
notebook.ipynb
tem-ctrl/statitistics_for_data_science
e7f654e9eec2b26657085dd7851187575ba86159
MIT
notebook.ipynb
tem-ctrl/statitistics_for_data_science
e7f654e9eec2b26657085dd7851187575ba86159
MIT
Qwen/Qwen-72B
1. YES
2. YES