# 분류 최적화<br>Classification Optimization


```python
import matplotlib.pyplot as plt
import numpy as np
import numpy.random as nr
import scipy.optimize as so


```

$(x_1, x_2)$ 데이터 집합 두개 생성<br>Generating two data sets




```python
n = 1000

set_0 = nr.normal([2, 0], [1, 1], (n, 2))
set_1 = nr.normal([0, 2], [1, 1], (n, 2))


```

생성한 두 데이터 집합을 표시<br>
Visualizing the two data sets




```python
def plot_two_sets(set_a, set_b):

    plt.plot(set_a[:, 0], set_a[:, 1], '.', label="y=0", alpha=0.5)
    plt.plot(set_b[:, 0], set_b[:, 1], '+', label="y=1", alpha=0.5)
    plt.grid(True)
    plt.axis('equal')
    plt.xlabel("$x_1$")
    plt.ylabel("$x_2$")


```


```python
plot_two_sets(set_0, set_1)

plt.legend(loc=0)
plt.show()
plt.close();


```

## 데이터 준비<br>Prepare data



모든 입력 값과 출력 값을 하나의 배열 안에 모음<br>
Collect all input & output values into one `numpy.ndarray`

| 입력 input | 출력 output |
|:----------:|:-----------:|
| $(x_1, x_2)$ | 0 |
| $(x_1, x_2)$ | 1 |




```python
y0 = np.zeros((len(set_0), 1))
y1 = np.ones((len(set_1), 1))

data_0 = np.concatenate([set_0, y0], axis=1)
data_1 = np.concatenate([set_1, y1], axis=1)

data = np.concatenate([
        data_0,
        data_1
    ], axis=0)


```

행과 열의 갯수 확인<br>
Check the number of rows and columns




```python
data.shape


```

처음 10개의 data<br>First 10 data points




```python
data[:10, :]


```

마지막 10개의 data<br>Last 10 data points




```python
data[-10:, :]


```

## 첫번째 (순진한) 시도<br>First (naive) attempt



### 모델<br>Model



$x_1$, $x_2$ 로부터 $y$ 값을 추정<br>
Estimate $y$ from $x_1$ and $x_2$



$$
\hat y = w_1 x_1 +  x_2 + w_2
$$




```python
def wx(w:np.ndarray, x_y:np.ndarray) -> np.ndarray:
    w1 = w[0]
    w2 = w[1]
    
    x1 = x_y[:, 0]
    x2 = x_y[:, 1]

    return w1 * x1 + x2 + w2


```

이 $\hat y$ 값이 0.5 보다 크면 1, 아니면 0 인 것으로 가정.<br>Let's assume it was one if this $\hat y$ is larger than 0.5. Zero otherwise.

### 비용 함수<br>Cost function



$$
C = \frac{1}{n}\sum_{i=1}^{n} \left( \hat y_i - y_i \right)^2
$$

예상되는 문제점?<br>
Can you expect any possible issues?




```python
def cost_function_first_attempt(w:np.ndarray, x_y:np.ndarray) -> float:
    n = len(x_y)
    y_hat = wx(w, x_y)
    y = x_y[:, -1]

    error = y_hat - y
    error_sqr = error ** 2
    result = error_sqr.sum() / n

    return result


```

최적화<br>
Optimize




```python
result = so.fmin(cost_function_first_attempt, x0=nr.rand(2,), args=(data,), full_output=True)
weights, cost_value, n_iter, n_call, warning = result
result


```

$\hat y = 0.5$



$$
\begin{align}
    \hat y &= 0.5 \\
    w_1 x_1 +  x_2 + w_2 &= 0.5 \\
     x_2 &= -w_1 x_1 -w_2 + 0.5
\end{align}
$$




```python
def plot_decision_boundary(x_min, x_max, weights_array):
    x1_array = np.linspace(x_min, x_max)
    x2_array = - weights_array[0] * x1_array - weights_array[1] + 0.5

    plt.plot(x1_array, x2_array, label="$\hat y = 0.5$")


```


```python
plot_two_sets(set_0, set_1)
plot_decision_boundary(data[:, 0].min(), data[:, 0].max(), weights)

plt.legend(loc=0)
plt.show()
plt.close();


```

## 계단함수<br>Step function



0 또는 1로 바꾸어주는 함수<br>
A function generating 0 or 1<br>



$$
s(z)=
    \begin{cases}
        0, z < 0\\
        1, z >= 0 \\
    \end{cases}
$$




```python
def step(z):
    return np.heaviside(z, 0)


```


```python
z_array = np.linspace(-10, 10)
g_z_array = step(z_array)
plt.plot(z_array, g_z_array)
plt.grid(True)
plt.xlabel('$z$')
plt.ylabel("$g(z)$")
plt.show()


```

계단 함수를 사용하는 비용함수<br>
Cost function using the sigmoid function




```python
def cost_function_step(w:np.ndarray, x_y:np.ndarray) -> float:
    n = len(x_y)
    y_hat = step(wx(w, x_y))
    y = x_y[:, -1]

    error = y_hat - y
    error_sqr = error ** 2
    result = error_sqr.sum() / n

    return result


```

최적화<br>
Optimize




```python
result = so.fmin(cost_function_step, x0=nr.rand(2,), args=(data,), full_output=True)
weights, cost_value, n_iter, n_call, warning = result
result


```


```python
plot_two_sets(set_0, set_1)
plot_decision_boundary(data[:, 0].min(), data[:, 0].max(), weights)

plt.legend(loc=0)
plt.show()
plt.close();


```

## Sigmoid function



0 과 1 사이를 부드럽게 연결하는 함수<br>
A function connecting 0 and 1 smoothly<br>
ref : [](https://youtu.be/PIjno6paszY?t=650)



$$
z = w_1 x_1 +  x_2 + w_2
$$



$$
\hat y =  g(z)=\frac{1}{1+exp\left(-z\right)}
$$



$$
C = \frac{1}{n}\sum_{i=1}^{n} \left( \hat y_i - y_i \right)^2
$$


```python
def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


```


```python
z_array = np.linspace(-10, 10)
g_z_array = sigmoid(z_array)
plt.plot(z_array, g_z_array)
plt.grid(True)
plt.xlabel('$z$')
plt.ylabel("$g(z)$")
plt.show()


```

시그모이드 함수를 사용하는 비용함수<br>
Cost function using the sigmoid function




```python
def cost_function_sigmoid(w:np.ndarray, x_y:np.ndarray) -> float:
    n = len(x_y)
    y_hat = sigmoid(wx(w, x_y))
    y = x_y[:, -1]

    error = y_hat - y
    error_sqr = error ** 2
    result = error_sqr.sum() / n

    return result


```

최적화<br>
Optimize




```python
result = so.fmin(cost_function_sigmoid, x0=nr.rand(2,), args=(data,), full_output=True)
weights, cost_value, n_iter, n_call, warning = result
result


```


```python
plot_two_sets(set_0, set_1)
plot_decision_boundary(data[:, 0].min(), data[:, 0].max(), weights)

plt.legend(loc=0)
plt.show()
plt.close();


```

## 교차 엔트로피 비용함수<br>Cross entropy cost function



국소 최소점을 피해 전역 최소점을 찾기 위해 사용<br>To find global and avoid local minimum.



ref : (14:23)
[](https://youtu.be/6vzchGYEJBc)



$$
C = \frac{1}{n}\sum_{i=1}^{n} \left[ -y_i log \left( \hat y_i \right) - \left(1 - y_i \right) log \left( 1 - \hat y_i \right)\right]
$$


```python
def cost_function_cross_entropy(w:np.ndarray, x_y:np.ndarray) -> float:
    n = len(x_y)
    y_hat = sigmoid(wx(w, x_y))
    y = x_y[:, -1]

    cost = -y * np.log2(y_hat) - (1 - y) * np.log2(1 - y_hat)

    return np.mean(cost)


```

최적화<br>
Optimize




```python
result = so.fmin(cost_function_cross_entropy, x0=nr.rand(2,), args=(data,), full_output=True)
weights, cost_value, n_iter, n_call, warning = result
result


```


```python
plot_two_sets(set_0, set_1)
plot_decision_boundary(data[:, 0].min(), data[:, 0].max(), weights)

plt.legend(loc=0)
plt.show()
plt.close();


```

## Final Bell<br>마지막 종




```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");


```


```python

```

d4fdd082dbf9f37955077590742c0cda0a67818a
ipynb
Jupyter Notebook
15_Optimization/030_Classification_Optimization.ipynb
kangwonlee/2109eca-nmisp-template
2e078870757fa06222df62d0ff8f4f4f288af51a
BSD-3-Clause
15_Optimization/030_Classification_Optimization.ipynb
kangwonlee/2109eca-nmisp-template
2e078870757fa06222df62d0ff8f4f4f288af51a
BSD-3-Clause
15_Optimization/030_Classification_Optimization.ipynb
kangwonlee/2109eca-nmisp-template
2e078870757fa06222df62d0ff8f4f4f288af51a
BSD-3-Clause
Qwen/Qwen-72B
1. YES
2. YES