# Problem set 2

Getting the mnist dataset from keras and importing other libraries that would help in this problem statement.

Sources: https://www.youtube.com/watch?v=81ZGOib7DTk
https://www.datacamp.com/community/tutorials/svm-classification-scikit-learn-python
https://towardsdatascience.com/write-markdown-latex-in-the-jupyter-notebook-10985edb91fd, https://nlp.stanford.edu/IR-book/html/htmledition/support-vector-machines-the-linearly-separable-case-1.html#:~:text=The%20SVM%20in%20particular%20defines,the%20margin%20of%20the%20classifier.&text=A%20classifier%20with%20a%20large%20margin%20makes%20no%20low%20certainty%20classification%20decisions.


TASK 1: Write code to train a multi-class support vector classifier with dot-product kernel and 1-norm
soft margin using the MNIST training data set.Then report the performance using MNIST test data set. There is a hyper-parameter that sets the trade-off between the margin and the training error | tune this hyper-parameter through cross-validation.

Linear Kernel: A linear kernel can be used as normal dot product any two given observations. The product between two vectors is the sum of the multiplication of each pair of input values.


```python
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
import math
from sklearn import svm
```

Loading the testing(10000) and training data(60000) samples, creating SVM classifier, training the model using training sets, and also predicting the data.


```python
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000,784) #had to reshape it to make equal dimensions 
x_test = x_test.reshape(10000,784) #had to reshape it to make equal dimensions
print(x_train.shape, y_train.shape, x_test.shape, y_test.shape)
clf = svm.LinearSVC()
clf.fit(x_train, y_train) #training the data 
y_pred = clf.predict(x_test) #predict the response of test dataset

```

    (60000, 784) (60000,) (10000, 784) (10000,)


    C:\Users\lovey\anaconda3\lib\site-packages\sklearn\svm\_base.py:976: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      warnings.warn("Liblinear failed to converge, increase "


In this the accuracy of dataset is calculated for that I am importing metrics from sklearn


```python
from sklearn import metrics
accuracy = metrics.accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy*100, "%")
```

    Accuracy: 87.26 %


TASK2:

Identify the Lagrange dual problem of the following primal problem:
Given Features $(x_1,y_1),....(x_N,y_N)$ $where y_1,...y_N\in{-1,1}$ 
\begin{equation}
Minimize \; w^T.w+C\sum_{i=1}^N\xi_i \;
\
Subject\;to\;y_i(w^T.x_i)\geq1-\xi_i and \;\;\xi_i\geq0 
\end{equation}
inequality constraints can be expressed as and also lagrange form is: 
\begin{equation}
2-y_i(w^T.x_i)-\xi_i\leq0 where\ 1-\xi\leq0
\\
L(w,\alpha) = w^T.w+C\sum_{i=1}^N\xi_i + \alpha_1(2-y_i(w^T.x_i)-\xi_i)+\alpha_2(1-\xi)
\end{equation}
where $\alpha_1$ & $\alpha_2$ are the Lagrange Multipliers where $\alpha_i\geq0$.
Set the partial derivative function to Zero.
\begin{equation}
\partial_wL = 2w - \alpha_1y_ix_i= 0
\\
w = \frac{\alpha_1y_ix_i}{2}
\end{equation}
The margin is:
\begin{equation}
\gamma = y\frac{w^Tw+b}{||w||}
\end{equation}

The SVM in particular defines the criterion to be looking for a decision surface that is maximally far away from any data point. This distance from the decision surface to the closest data point determines the margin of the classifier.

Benefits of maximizing the margin is this allows to handle minor  errors in the data sets. 

Solving Dual problem advantages: In the Dual problem we take $\alpha_i$ which help us to decide the importance of the subjected features also computation is fast as well as optimzation is easy.

Solving Primal Problem Disadvantages: we only obtain the optimal $w$ also Computing the scalar product of $w^Tx$ which makes the computation slow. 



09c94078842ea36d99456f57b9c905b707fbfce8
ipynb
Jupyter Notebook
Problem_set_2.ipynb
AshishSingh-Glitch90/Pattern-Recognition
3874b786607cae78e4b101cbe04a353c4de43006
MIT
2022-01-09T11:24:27.000Z
2022-01-09T11:24:27.000Z
Problem_set_2.ipynb
AshishSingh-Glitch90/Pattern-Recognition
3874b786607cae78e4b101cbe04a353c4de43006
MIT
Problem_set_2.ipynb
AshishSingh-Glitch90/Pattern-Recognition
3874b786607cae78e4b101cbe04a353c4de43006
MIT
Qwen/Qwen-72B
1. YES
2. YES