<center><h1>Least Squares Support Vector Classifier</h1></center>

## Summary:
1. [Introduction](#introduction)


2. [Using the classifier](#using_classifier)

    2.1 [CPU/Numpy version](#cpu_version)
    
    2.2 [GPU/PyTorch version](#gpu_version)

# 1. Introduction <a class="anchor" id="introduction"></a>

The Least Squares Support Vector Machine (LSSVM) is a variation of the original Support Vector Machine (SVM) in which we have a slight change in the objective and restriction functions that results in a big simplification of the optimization problem.

First, let's see the optimization problem of an SVM:

$$ 
\begin{align}
    minimize && f_o(\vec{w},\vec{\xi})=\frac{1}{2} \vec{w}^T\vec{w} + C \sum_{i=1}^{n} \xi_i &&\\
    s.t. && y_i(\vec{w}^T\vec{x}_i+b)\geq 1 - \xi_i, && i = 1,..., n \\
         && \xi_i \geq 0,                            && i = 1,..., n
\end{align}
$$

In this case, we have a set of inequality restrictions and when solving the optimization problem by it's dual we find a discriminative function, adding the kernel trick, of the type:


$$ f(\vec{x}) = sign \ \Big( \sum_{i=1}^{n} \alpha_i^o y_i K(\vec{x}_i,\vec{x}) + b_o \Big) $$

Where $\alpha_i^o$ and $b_o$ denote optimum values. Giving enough regularization (smaller values of $C$) we get a lot of $\alpha_i^o$ nulls, resulting in a sparse model in which we only need to save the pairs $(\vec{x}_i,y_i)$ which have the optimum dual variable not null. The vectors $\vec{x}_i$ with not null $\alpha_i^o$ are known as support vectors (SV).



In the LSSVM case, we change the inequality restrictions to equality restrictions. As the $\xi_i$ may be negative we square its values in the objective function:

$$ 
\begin{align}
    minimize && f_o(\vec{w},\vec{\xi})=\frac{1}{2} \vec{w}^T\vec{w} + \gamma \frac{1}{2}\sum_{i=1}^{n} \xi_i^2 &&\\
    s.t. && y_i(\vec{w}^T\vec{x}_i+b) = 1 - \xi_i, && i = 1,..., n
\end{align}
$$


The dual of this optimization problem results in a system of linear equations, a set of Karush-Khun-Tucker (KKT) equations:

$$
\begin{bmatrix} 
    0 & \vec{d}^T \\
    \vec{y} & \Omega + \gamma^{-1} I 
\end{bmatrix}
\
\begin{bmatrix} 
    b  \\
    \vec{\alpha}
\end{bmatrix}
=
\begin{bmatrix} 
    0 \\
    \vec{1}
\end{bmatrix}
$$

Where, with the kernel trick, &nbsp; $\Omega_{i,j} = y_i y_j K(\vec{x}_i,\vec{x}_j)$,  &nbsp;  $\vec{y} = [y_1 \ y_2 \ ... \ y_n]^T$, &nbsp; $\vec{\alpha} = [\alpha_1 \ \alpha_2 \ ... \ \alpha_n]^T$ &nbsp;  e &nbsp; $\vec{1} = [1 \ 1 \ ... \ 1]^T$.

The discriminative function of the LSSVM has the same form of the SVM but the $\alpha_i^o$ aren't usually null, resulting in a bigger model. The big advantage of the LSSVM is in finding it's parameters, which is reduced to solving the linear system of the type:

$$ A\vec{z} = \vec{b} $$

A well-known solution of the linear system is when we minimize the square of the residues, that can be written as the optimization problem:

$$
\begin{align}
    minimize && f_o(\vec{z})=\frac{1}{2}||A\vec{z} - \vec{b}||^2\\
\end{align}
$$

And have the analytical solution:

$$ \vec{z} = A^{\dagger} \vec{b} $$

Where $A^{\dagger}$ is the pseudo-inverse defined as:

$$ A^{\dagger} = (A^T A)^{-1} A^T$$

# 2. Using the classifier <a class="anchor" id="using_classifier"></a>


```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_digits
from sklearn.preprocessing import MinMaxScaler

from lssvm import LSSVC, LSSVC_GPU
from utils.encoding import dummie2multilabel
```


```python
# Preprocessing

# Import digits recognition dataset (from sklearn)
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=2020)

# Scaling features (from sklearn)
scaler = MinMaxScaler()
scaler.fit(X_train)
X_tr_norm = scaler.transform(X_train)
X_ts_norm = scaler.transform(X_test)

# Get information about input and outputs
print(f"X_train.shape: {X_train.shape}")
print(f"X_test.shape:  {X_test.shape}")
print(f"y_train.shape: {y_train.shape}")
print(f"y_test.shape:  {y_test.shape}")
print(f"np.unique(y_train): {np.unique(y_train)}")
print(f"np.unique(y_test):  {np.unique(y_test)}")
```

    X_train.shape: (1078, 64)
    X_test.shape:  (719, 64)
    y_train.shape: (1078,)
    y_test.shape:  (719,)
    np.unique(y_train): [0 1 2 3 4 5 6 7 8 9]
    np.unique(y_test):  [0 1 2 3 4 5 6 7 8 9]


## 2.1 CPU/Numpy version <a class="anchor" id="cpu_version"></a>


```python
# Use the classifier with different kernels

print('Gaussian kernel:')
lssvc = LSSVC(gamma=1, kernel='rbf', sigma=.5) # Class instantiation
lssvc.fit(X_tr_norm, y_train) # Fitting the model
y_pred = lssvc.predict(X_ts_norm) # Making predictions with the trained model
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)) # Calculate Accuracy
print('acc_test = ', acc, '\n')

print('Polynomial kernel:')
lssvc = LSSVC(gamma=1, kernel='poly', d=2)
lssvc.fit(X_tr_norm, y_train)
y_pred = lssvc.predict(X_ts_norm)
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred))
print('acc_test = ', acc, '\n')

print('Linear kernel:')
lssvc = LSSVC(gamma=1, kernel='linear')
lssvc.fit(X_tr_norm, y_train)
y_pred = lssvc.predict(X_ts_norm)
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred))
print('acc_test = ', acc, '\n')
```

    Gaussian kernel:
    acc_test =  0.9680111265646731 
    
    Polynomial kernel:
    acc_test =  0.9944367176634215 
    
    Linear kernel:
    acc_test =  0.9791376912378303 
    


If there are any questions about a specific method, the user can ask for help:


```python
help(LSSVC.fit)
```

    Help on function fit in module lssvm.LSSVC:
    
    fit(self, X, y)
        Fits the model given the set of X attribute vectors and y labels.
        - X: ndarray of shape (n_samples, n_attributes)
        - y: ndarray of shape (n_samples,) or (n_samples, n)
            If the label is represented by an array of n elements, the y 
            parameter must have n columns.
    


The user can also have an overview of the class:


```python
help(LSSVC)
```

    Help on class LSSVC in module lssvm.LSSVC:
    
    class LSSVC(builtins.object)
     |  LSSVC(gamma=1, kernel='rbf', **kernel_params)
     |  
     |  A class that implements the Least Squares Support Vector Machine 
     |  for classification tasks.
     |  
     |  It uses Numpy pseudo-inverse function to solve the dual optimization 
     |  problem with ordinary least squares. In multiclass classification 
     |  problems the approach used is one-vs-all, so, a model is fit for each 
     |  class while considering the others a single set of the same class.
     |  
     |  # Parameters:
     |  - gamma: float, default = 1.0
     |      Constant that control the regularization of the model, it may vary 
     |      in the set (0, +infinity). The closer gamma is to zero, the more 
     |      regularized the model will be.
     |  - kernel: {'linear', 'poly', 'rbf'}, default = 'rbf'
     |      The kernel used for the model, if set to 'linear' the model 
     |      will not take advantage of the kernel trick, and the LSSVC maybe only
     |      useful for linearly separable problems.
     |  - kernel_params: **kwargs, default = depends on 'kernel' choice
     |      If kernel = 'linear', these parameters are ignored. If kernel = 'poly',
     |      'd' is accepted to set the degree of the polynomial, with default = 3. 
     |      If kernel = 'rbf', 'sigma' is accepted to set the radius of the 
     |      gaussian function, with default = 1. 
     |   
     |  # Attributes:
     |  - All hyperparameters of section "Parameters".
     |  - alpha: ndarray of shape (1, n_support_vectors) if in binary 
     |           classification and (n_classes, n_support_vectors) for 
     |           multiclass problems
     |      Each column is the optimum value of the dual variable for each model
     |      (using the one-vs-all approach we have n_classes == n_classifiers), 
     |      it can be seen as the weight given to the support vectors 
     |      (sv_x, sv_y). As usually there is no alpha == 0, we have 
     |      n_support_vectors == n_train_samples.
     |  - b: ndarray of shape (1,) if in binary classification and (n_classes,) 
     |       for multiclass problems 
     |      The optimum value of the bias of the model.
     |  - sv_x: ndarray of shape (n_support_vectors, n_features)
     |      The set of the supporting vectors attributes, it has the shape 
     |      of the training data.
     |  - sv_y: ndarray of shape (n_support_vectors, n)
     |      The set of the supporting vectors labels. If the label is represented 
     |      by an array of n elements, the sv_y attribute will have n columns.
     |  - y_labels: ndarray of shape (n_classes, n)
     |      The set of unique labels. If the label is represented by an array 
     |      of n elements, the y_label attribute will have n columns.
     |  - K: function, default = rbf()
     |      Kernel function.
     |  
     |  Methods defined here:
     |  
     |  __init__(self, gamma=1, kernel='rbf', **kernel_params)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |  
     |  dump(self, filepath='model', only_hyperparams=False)
     |      This method saves the model in a JSON format.
     |      - filepath: string, default = 'model'
     |          File path to save the model's json.
     |      - only_hyperparams: boolean, default = False
     |          To either save only the model's hyperparameters or not, it 
     |          only affects trained/fitted models.
     |  
     |  fit(self, X, y)
     |      Fits the model given the set of X attribute vectors and y labels.
     |      - X: ndarray of shape (n_samples, n_attributes)
     |      - y: ndarray of shape (n_samples,) or (n_samples, n)
     |          If the label is represented by an array of n elements, the y 
     |          parameter must have n columns.
     |  
     |  predict(self, X)
     |      Predicts the labels of data X given a trained model.
     |      - X: ndarray of shape (n_samples, n_attributes)
     |  
     |  ----------------------------------------------------------------------
     |  Class methods defined here:
     |  
     |  load(filepath, only_hyperparams=False) from builtins.type
     |      This class method loads a model from a .json file.
     |      - filepath: string
     |          The model's .json file path.
     |      - only_hyperparams: boolean, default = False
     |          To either load only the model's hyperparameters or not, it 
     |          only has effects when the dump of the model as done with the
     |          model's parameters.
     |  
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |  
     |  __dict__
     |      dictionary for instance variables (if defined)
     |  
     |  __weakref__
     |      list of weak references to the object (if defined)
    


The user may also save the model in JSON format: 


```python
lssvc.dump('model')
loaded_model = LSSVC.load('model')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(loaded_model.predict(X_ts_norm))
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303


## 2.2 GPU/PyTorch version <a class="anchor" id="gpu_version"></a>    

It has the same functionalities and syntax of the CPU version, the difference is the use of PyTorch to run the operations on the GPU.


```python
# Use the classifier with different kernels

print('Gaussian kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='rbf', sigma=.5) 
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')

print('Polynomial kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='poly', d=2)
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')

print('Linear kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='linear')
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')
```

    Gaussian kernel:
    acc_test =  0.9680111265646731 
    
    Polynomial kernel:
    acc_test =  0.9944367176634215 
    
    Linear kernel:
    acc_test =  0.9791376912378303 
    


The user may also save the model in JSON format: 


```python
lssvc_gpu.dump('model')
loaded_model = LSSVC_GPU.load('model')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(loaded_model.predict(X_ts_norm).cpu())
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303


The `model.json` is the same for the CPU and GPU version, giving the developer the possibility to train a model in GPU, dumping it in a .json, and loading in CPU version (the other way around is also possible).


```python
lssvc.dump('model_from_cpu')
lssvc_gpu = LSSVC_GPU.load('model_from_cpu')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303



```python
lssvc_gpu.dump('model_from_gpu')
lssvc = LSSVC.load('model_from_gpu')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test), 
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303


9f0492458df13a6dd9cf1edb3b2f3ffbf664eae5
ipynb
Jupyter Notebook
LSSVC.ipynb
RomuloDrumond/LSSVM
092c486218c8a6bd0690fa6ebf712dddb488c397
MIT
2019-07-25T08:51:20.000Z
2022-03-19T06:03:35.000Z
LSSVC.ipynb
rcminitor/LSSVM
092c486218c8a6bd0690fa6ebf712dddb488c397
MIT
2019-07-25T09:48:25.000Z
2022-03-11T23:55:38.000Z
LSSVC.ipynb
rcminitor/LSSVM
092c486218c8a6bd0690fa6ebf712dddb488c397
MIT
2019-10-25T04:55:43.000Z
2022-03-20T23:49:00.000Z
Qwen/Qwen-72B
1. YES
2. YES