# Extreme Learning Machine Tutorial

Sample code just for quickly teaching how to build an ELM network using Numpy and using a quick computation algorithm.

## Introduction

This machine learning algorithm was designed by Huang et al. [1], using a single layer *feed forward* network.
One of their main features is that both weights $\mathbf{W}$ and biases $\mathbf{b}$ are random and they don't require fitting. Thus, their output weights are independent.
The output $\beta$ is obtained using the following operation
$$ \begin{align}
H\beta &= T \\
\beta &= H^{\dagger} T \\
H^{\dagger} &= \left(H^{T} H \right)^{-1} H^{T}\\
\end{align}$$
WHere $H^{\dagger}$ is the **_Moore-Penrose Pseudoinverse_**, and $H$ is a full-rank column matrix

## Moore-Penrose PseudoInverse
This is a generalization form of the inverse matrix, which allows solving least-squared systems which don't have a well-defined solution.

A matrix $G$ of size $m\times n$ has a Pseudoinverse $G^{\dagger}$ of size $n\times m$ which has the following properties:

$$G G^{\dagger} G = G$$
$$G^{\dagger} G G^{\dagger} = G^{\dagger}$$
$$\left( G G^{\dagger} \right)' = G G^{\dagger}$$
$$\left(G^{\dagger} G \right)' = G^{\dagger} G$$

This calculation is computationally complex. Normally, the least-squares method or Eigen-value Decomposition are used (most commonly on `MATLAB` and `SciPy`).

Courrieu [2] proposed a more computationally simple algorithm, based on reverse ordering and Cholesky factorization of a symmetric full-rank positive matrix.


```python
import numpy as np
from time import time
```


```python
#Defining Moore-Penrose PseudoInverse as defined by Courrieu (original in Matlab)
def mp_pinv(G):
    m,n = G.shape #Shape of matrix
    transpose = False
    
    if m < n:
        transpose = True #If m is bigger we multiply it with its transpose
        A = np.matmul(G,G.T) 
        n = m
    else:
        A = np.matmul(G.T,G) #Else, it's backwards
    
    #Full-rank Choilesky
    L = np.linalg.cholesky(A) 
    
    #This part is doing the same without the NumPy function, I left it here just if you want to test it.
    #dA = np.diag(A)
    #tol = np.min(dA[dA>0])*1e-9
    #L = np.zeros_like(A,dtype=np.float) #Zeros-like matrix like A
    #r = 0
    #for k in range(n):
    #    L[k:n,r] = A[k:n,k]-np.matmul(L[k:n,0:(r)],L[k,0:(r)].T)
    #    #If is higher than tolerance value:
    #    if L[k,r] > tol:
    #        L[k,r]=np.sqrt(L[k,r]) #We get square-root
    #        if k<n:
    #            L[(k+1):n,r]= L[(k+1):n,r]/L[k,r]; #And divide it
    #    else:
    #        r-=1;
    #    r+=1 
    #L=L[:,0:r]
    
    #We compute generalized inverse
    M = np.linalg.inv(np.matmul(L.T,L))
    if transpose:
        #Y = G.T * L * M * M * L.T
        Y = np.matmul(np.matmul(np.matmul(np.matmul(G.T,L),M),M),L.T)
    else:
        #Y = L* M * M *L.T * G.T
        Y = np.matmul(np.matmul(np.matmul(np.matmul(L,M),M),L.T),G.T)
    
    return Y
```

### Comparing this method with NumPy implementation


```python
matrix = np.random.rand(800).reshape(25,32) #Random Matrix as input

t0 = time()
m01 =np.linalg.pinv(matrix)
print("Metodo SVD numpy: {:.3f} ms".format((time()-t0)*1000))

t0 = time()
m02 = mp_pinv(matrix)
print("\nCourrieu method: {:.3f} ms".format((time()-t0)*1000))
print("\nMean Squared Error between these two: {}".format(np.mean((m01-m02)**2)))
```

    Metodo SVD numpy: 0.719 ms
    
    Courrieu method: 1.379 ms
    
    Mean Squared Error between these two: 1.1021139939204808e-28


*_ TODO: compile using numba and do a better comparison..._*

## Implementing ELM


```python
#Loading Breast Cancer
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X,Y = load_breast_cancer(return_X_y=True)
X = (X- X.min(axis=0))/(X.max(axis=0)-X.min(axis=0)) #Normalization between 0 - 1

#Splitting Train and test Data
Xtrain,Xtest,Ytrain,Ytest = train_test_split(X,Y,train_size = 0.7, test_size=0.3)
```


```python
#Params
n_train = len(Xtrain)
n_test = len(Xtest)
n_data = Xtrain.shape[-1] #Number of samples
n_hidden = 1000 #Hidden neurons

#Training
true = Ytrain.reshape(n_train,1).T #Outputs
w_i = np.random.rand(n_hidden,n_data)*2 -1 #Input weights
b_i = np.random.rand(n_hidden,1) #Input biases

b_m = b_i * np.ones(n_train) #Bias matrix for all values
temp_H = np.matmul(w_i,Xtrain.T)+b_m # wX+b

#sigmoid function
def sigmoid(x):
    return 1/(1+np.exp(-x))

H = sigmoid(temp_H) #Activation function
# Computing Beta with the Pseudoinverse
H_cross = mp_pinv(H.T)
beta_i = np.matmul(H_cross,true.T) #And multiply it with the output vector
Y = np.matmul(H.T,beta_i) #And our training prediction
```


```python
#Test
b_mt = b_i *np.ones(n_test) 
temp_Htest = np.matmul(w_i,Xtest.T) + b_mt # wX + b
H_test = sigmoid(temp_Htest) 
testY = np.matmul(H_test.T,beta_i) # Prediction

Ypred = np.where(testY <= 0.5,0,1) #If it's higher than 0.5, then is positive, else not.

from sklearn.metrics import accuracy_score
print("Test Accuracy: {:.2f}%".format(accuracy_score(Ytest,Ypred)*100))
```

    Test Accuracy: 85.96%


# Future Work
* I plan to do a Tensorflow or PyTorch implementation of this same code. If someone wants to use it
* Compiling in order to better compare Courrieu's method, specially without using Cholesky from numpy...

This code is way far from optimal, but can be a simple example just for fun.

## References
1. Huang, G. B., Zhu, Q. Y., & Siew, C. K. (2004, July). Extreme learning machine: a new learning scheme of feedforward neural networks. In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on (Vol. 2, pp. 985-990). IEEE.
2. Courrieu, P. (2008). Fast computation of Moore-Penrose inverse matrices. arXiv preprint arXiv:0804.4809. Obtained from https://arxiv.org/abs/0804.4809
3. Akusok, A., Björk, K. M., Miche, Y., & Lendasse, A. (2015). High-Performance Extreme Learning Machines: A Complete Toolbox for Big Data Applications. IEEE Access, 3, 1011–1025. https://doi.org/10.1109/ACCESS.2015.2450498



```python

```

d92e002c521a50cf7b80e8bda30e7657e0408d20
ipynb
Jupyter Notebook
ELM.ipynb
dmelladoc/ML-examples
994e5149ba4b4d73fdcc3f891fffb74a15e49252
MIT
ELM.ipynb
dmelladoc/ML-examples
994e5149ba4b4d73fdcc3f891fffb74a15e49252
MIT
ELM.ipynb
dmelladoc/ML-examples
994e5149ba4b4d73fdcc3f891fffb74a15e49252
MIT
Qwen/Qwen-72B
1. YES
2. YES