# Backpropagation Radial basis function

So we try radial basis function for the last classification after 5 layers of convolution and two pooling layer.
    \begin{equation}
    f(\mathbf{x})=e^{-\beta |\mathbf{c-x}|^{2}} -O
    \end{equation}
$O$ being an offset for the reason we will give later. For each patch we want to classify whether is it forest or not. If it is forest we would expect $f(\mathbf{x})$ to be 1. So now we can calculate the error:
    \begin{equation}
    E=\frac{1}{2} |t-f(\mathbf{x})|^{2} 
    \end{equation}
    

So if we have forest in our training patch than $t=1$. First we try to find suitable values for $\beta$ and $\mathbf{c}$. With back propagation we find:
    \begin{equation}
    \Delta \beta= -\eta \frac{dE}{d \beta}= -\eta \frac{dE}{d f(\mathbf{x})} \frac{d f(\mathbf{x})}{d \beta}= \eta(t-f( \mathbf{x} )) \cdot -|\mathbf{c-x}|^2 e^{-\beta |\mathbf{c-x}|^{2}} =\eta \left ( t-f( \mathbf{x} ) \right ) \cdot -|\mathbf{c-x}|^2 f(\mathbf{x}) 
    \end{equation}

\begin{equation}
\Delta c_{i}= -\eta \frac{dE}{d c_{i}}= -\eta \frac{dE}{d f(\mathbf{x})} \frac{d f(\mathbf{x})}{d c_{i}}= \eta |t-f(\mathbf{x})| \frac{d}{d c_{i}} \left (  e^{- \beta  \sum_{i} (c_{i}- x_{i})^{2}}    \right )
\end{equation}

\begin{equation}
= \eta |t-f(\mathbf{x})| \cdot -\beta f(\mathbf{x}) \cdot 2(c_{i}-x_{i})
\end{equation}
We see that if $f(x)$ is zero we do not have an update. We can overcome this problem by setting an offset in the RBF. Now 0.5 will be the maximum peak value and -0.5 indicatates the lowest value.

# Le-Net 1 based architecture

We start with 50X50 (I) after first convolution we have 46X46 (L1). The next pooling layer takes twice as big steps (instead of 1 pixel) reducing to an output image of 23X23 with 2x2 pooling kernels (L2). Then we apply different types of convolution 4x4 kernels on the L2 layer resulting in 20x20 (L3) . Then followed by pooling 5X5 resulting in 4x4 output map (L4). So we have 16 connection for each element in layer L4 (which depend on the amount of different Covolutions in L3) 


\begin{equation}
f(x)=\frac{1}{1+e^{-x}} \\
F_{k}= f( \sum_{i} \mathbf{W_{i} \cdot y_{i}}-b_{k})
\end{equation}


\begin{equation}
E=\sum_{k} \frac{1}{2}|t_k-F_{k}|^{2} \\
\Delta W_{ij}= - \eta \frac{dE}{d W_{ij}}
\end{equation}

\begin{equation}
\Delta W_{ij}= \sum_{k} - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{dW_{ij}}=\sum_{k} \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \frac{dx_{k}}{dW_{ij}} \\
= \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} y_{ij}
\end{equation}

\begin{equation}
\Delta b_{k}= - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{b_{k}}=\eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \cdot-1
\end{equation}


```
(np.exp(-x))/(1+np.exp(-x)**2)
```


```
%%timeit
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
import Image

def Sigmoid(x):
    return (1/(1+np.exp(-x)))

def FileExistCheck(file):
    try:
        img = Image.open(file)
        return('exist')
    except: return('not exist')
        
def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

WC11=np.array([
    [1,1,1,1,1],
    [1,2,2,2,1],
    [1,2,4,2,1],
    [1,2,2,2,1],
    [1,1,1,1,1],    
    ])

WC12=np.array([
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    ])

WC13=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [0.5,0.5,0.5,0.5,0.5],
    [0,0,0,0,0],
    [0,0,0,0,0]
    ])

WC14=np.array([
    [0.5,1,1,1,1],
    [0,0.5,1,1,1],
    [0,0,0.5,1,1],
    [0,0,0,0.5,1],
    [0,0,0,0,0.5]
    ])

WC5=np.array([
    [0.5,0,0,0,0],
    [1,0.5,0,0,0],
    [1,1,0.5,0,0],
    [1,1,1,0.5,0],
    [1,1,1,1,0.5]
    ])
# averaging kernel in L2
WP=np.array([ 
    [1,1],
    [1,1],
    ])

# WC31=np.array([ 
#     [1,1,0,0],
#     [1,1,0,0],
#     [1,1,0,0],
#     [1,1,0,0],
#     ])

# WC32=np.array([ 
#     [1,1,1,1],
#     [1,1,1,1],
#     [0,0,0,0],
#     [0,0,0,0],
#     ])
WC31=np.array([ 
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    ])

WC32=np.array([ 
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    ])



WP4=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    ])


W1=np.zeros(16)
W2=np.zeros(16)
W3=np.zeros(16)
W4=np.zeros(16)
W5=np.zeros(16)
W6=np.zeros(16)
W7=np.zeros(16)
W8=np.zeros(16)
b1=0
b2=0

n1=1*10**-12
n2=1*10**2


# root='Forest\images\F1_'
# t1=1 # f1 is classifier for forest
# t2=0 # classifier for no forest
Input=1
k=0
for u in range(1,800): #it will start from 1 due to filenames
#     print(u)
    if(Input==1):
#         print('forest')
        root='Forest\images\F1_'
        t1=1 # f1 is classifier for forest
        t2=0 # classifier for no forest
        Input=2
        k=k+1
    elif(Input==2):
#         print('city')
        root='City\images\C1_'
        t1=0 # f1 is classifier for forest
        t2=1 # classifier for no forest
        Input=3
        
    elif(Input==3):
#         print('forest 2')
        root='Forest\images\F2_'
        t1=1 # f1 is classifier for forest
        t2=0 # classifier for no forest
        Input=4
 
    elif(Input==4):
#         print('city 2')
        root='City\images\C2_'
        t1=0 # f1 is classifier for forest
        t2=1 # classifier for no forest
        Input=1
    
    if (k<10): file=root+'0'+str(k)+'.png'
    else: file=str(root)+str(k)+'.png' 
    if(FileExistCheck(file)=='not exist'): continue # if file does not exist it stops the current iteration
    img = Image.open(file)
    data=img.convert('RGB')
    data = np.asarray( data, dtype="int32" )
    data=np.transpose(data) # This can be improved, but this was easy since we have een 3D matrix
    
    #L1
    ConvImg1=sig.convolve(data[0,:,:],WC11,mode='valid') 
    ConvImg2=sig.convolve(data[0,:,:],WC12,mode='valid') 
    ConvImg3=sig.convolve(data[0,:,:],WC13,mode='valid') 
    ConvImg4=sig.convolve(data[0,:,:],WC14,mode='valid') 
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    PoolImg1_41=Pool(ConvImg1_31,WP4) # reducing dimensionality by 2.
    PoolImg1_42=Pool(ConvImg1_32,WP4) # reducing dimensionality by 2.
    
    PoolImg2_43=Pool(ConvImg2_33,WP4) # reducing dimensionality by 2.
    PoolImg2_44=Pool(ConvImg2_34,WP4) # reducing dimensionality by 2.
    
    PoolImg3_45=Pool(ConvImg3_35,WP4) # reducing dimensionality by 2.
    PoolImg3_46=Pool(ConvImg3_36,WP4) # reducing dimensionality by 2.
    
    PoolImg4_47=Pool(ConvImg4_37,WP4) # reducing dimensionality by 2.
    PoolImg4_48=Pool(ConvImg4_38,WP4) # reducing dimensionality by 2.
    

    # Now we have the outputs and we make 
    x1=np.inner(PoolImg1_41.flatten(),W1)+np.inner(PoolImg2_43.flatten(),W3)+np.inner(PoolImg3_45.flatten(),W5)+np.inner(PoolImg4_47.flatten(),W7)
    x2=np.inner(PoolImg1_42.flatten(),W2)+np.inner(PoolImg2_44.flatten(),W4)+np.inner(PoolImg3_46.flatten(),W6)+np.inner(PoolImg4_48.flatten(),W8)
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
    W1=W1+ n1*(t1-f1)*PoolImg1_41.flatten()
    W3=W3+ n1*(t1-f1)*PoolImg2_43.flatten()
    W5=W5+ n1*(t1-f1)*PoolImg3_45.flatten()
    W7=W7+ n1*(t1-f1)*PoolImg4_47.flatten()
        
    W2=W2+ n1*(t2-f2)*PoolImg1_42.flatten()
    W4=W4+ n1*(t2-f2)*PoolImg2_44.flatten() 
    W6=W6+ n1*(t2-f2)*PoolImg3_46.flatten() 
    W8=W8+ n1*(t2-f2)*PoolImg4_48.flatten()
    b1=b1-n2*(t1-f1)
    b2=b2-n2*(t2-f2)
#     print(f1,f2)


```

    1 loops, best of 3: 19.3 s per loop


This training duration is 19.3 seconds


```

im=[]
root='City\images\C4_'
for k in range(1,250): #it will start from 1 due to filenames
    if (k<10): file=root+'0'+str(k)+'.png'
    else: file=str(root)+str(k)+'.png'
    if(FileExistCheck(file)=='not exist'): 
        continue # if file does not exist it stops the current iteration
    img = Image.open(file)
    data=img.convert('RGB')
    data = np.asarray( data, dtype="int32" )
    data=np.transpose(data) # This can be improved, but this was easy since we have een 3D matrix
    #L1
    ConvImg1=sig.convolve(data[0,:,:],WC11,mode='valid') 
    ConvImg2=sig.convolve(data[0,:,:],WC12,mode='valid') 
    ConvImg3=sig.convolve(data[0,:,:],WC13,mode='valid') 
    ConvImg4=sig.convolve(data[0,:,:],WC14,mode='valid') 
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    PoolImg1_41=Pool(ConvImg1_31,WP4) # reducing dimensionality by 2.
    PoolImg1_42=Pool(ConvImg1_32,WP4) # reducing dimensionality by 2.
    
    PoolImg2_43=Pool(ConvImg2_33,WP4) # reducing dimensionality by 2.
    PoolImg2_44=Pool(ConvImg2_34,WP4) # reducing dimensionality by 2.
    
    PoolImg3_45=Pool(ConvImg3_35,WP4) # reducing dimensionality by 2.
    PoolImg3_46=Pool(ConvImg3_36,WP4) # reducing dimensionality by 2.
    
    PoolImg4_47=Pool(ConvImg4_37,WP4) # reducing dimensionality by 2.
    PoolImg4_48=Pool(ConvImg4_38,WP4) # reducing dimensionality by 2.
    

    # Now we have the outputs and we make 
    x1=np.inner(PoolImg1_41.flatten(),W1)+np.inner(PoolImg2_43.flatten(),W3)+np.inner(PoolImg3_45.flatten(),W5)+np.inner(PoolImg4_47.flatten(),W7)
    x2=np.inner(PoolImg1_42.flatten(),W2)+np.inner(PoolImg2_44.flatten(),W4)+np.inner(PoolImg3_46.flatten(),W6)+np.inner(PoolImg4_48.flatten(),W8)
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)  
#     print(f1,f2)
    if(f1>0.5): im=np.append(im,file)
```


```
import IPython.display as dis

for k in range(0,len(im)):
    dis.display(dis.Image(filename=im[k]))
 
```

# Patching in Python


```
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
import Image
RGB=2
file='City\C4.png'
img = Image.open(file)
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=150 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatched=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])
```


```
dataPatched.shape
```




    (3L, 8L, 150L, 150L)




```
inputPatch=dataPatched[0,6,:,:]
img = Image.fromarray( np.asarray( np.clip(inputPatch,0,255), dtype="uint8"), "L" )
img.save('patch'+str(RGB)+'.png' )
```

# Training with patches from Python

First let us patch all the images we want to use!


```
# %%timeit
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
import Image
RGB=1 #selects layer R=0,G=1,B=2

img = Image.open('Forest\F1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

```


```
def Sigmoid(x):
    return (1/(1+np.exp(-x)))

def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

WC11=np.array([
    [1,1,1,1,1],
    [1,2,2,2,1],
    [1,2,4,2,1],
    [1,2,2,2,1],
    [1,1,1,1,1],    
    ])

WC12=np.array([
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    ])

WC13=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [0.5,0.5,0.5,0.5,0.5],
    [0,0,0,0,0],
    [0,0,0,0,0]
    ])

WC14=np.array([
    [0.5,1,1,1,1],
    [0,0.5,1,1,1],
    [0,0,0.5,1,1],
    [0,0,0,0.5,1],
    [0,0,0,0,0.5]
    ])

WC5=np.array([
    [0.5,0,0,0,0],
    [1,0.5,0,0,0],
    [1,1,0.5,0,0],
    [1,1,1,0.5,0],
    [1,1,1,1,0.5]
    ])
# averaging kernel in L2
WP=np.array([ 
    [1,1],
    [1,1],
    ])

WC31=np.array([ 
    [1,1,1,1],
    [0,1,1,1],
    [0,0,1,1],
    [0,0,0,1],
    ])

WC32=np.array([ 
    [1,0,0,0],
    [1,1,0,0],
    [1,1,1,0],
    [1,1,1,1],
    ])



WP4=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    ])


W1=np.zeros(16)
W2=np.zeros(16)
W3=np.zeros(16)
W4=np.zeros(16)
W5=np.zeros(16)
W6=np.zeros(16)
W7=np.zeros(16)
W8=np.zeros(16)
b1=0
b2=0

n1=1*10**-12
n2=1*10**2

Input=1
k=0
i=0
j=0
for u in range(0,2000):   
    if(Input==1):
        if(j==25 and i==10):
            print('stop')
            break # Searchspace of 2000x2000 is achieved and the loop will stop
        if(i==10): # i must increase to cover all patches over y-direction 
            j=j+1 # selects new patch in x direction
            i=0
        i=i+1 
        inputPatch=dataPatchedF1[i,j,:,:] 
        t1=1 # f1 is classifier for forest
        t2=0 # classifier for no forest
        Input=2

    elif(Input==2):
        inputPatch=dataPatchedC1[i,j,:,:]
        t1=0 
        t2=1 
        Input=3        
    elif(Input==3):
        inputPatch=dataPatchedF2[i,j,:,:]
        t1=1
        t2=0 
        Input=4       
    elif(Input==4):
        inputPatch=dataPatchedC2[i,j,:,:]
        t1=0
        t2=1 
        Input=1

    #L1
    ConvImg1=sig.convolve(inputPatch,WC11,mode='valid') 
    ConvImg2=sig.convolve(inputPatch,WC12,mode='valid') 
    ConvImg3=sig.convolve(inputPatch,WC13,mode='valid') 
    ConvImg4=sig.convolve(inputPatch,WC14,mode='valid') 
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    PoolImg1_41=Pool(ConvImg1_31,WP4) # reducing dimensionality by 2.
    PoolImg1_42=Pool(ConvImg1_32,WP4) # reducing dimensionality by 2.
    
    PoolImg2_43=Pool(ConvImg2_33,WP4) # reducing dimensionality by 2.
    PoolImg2_44=Pool(ConvImg2_34,WP4) # reducing dimensionality by 2.
    
    PoolImg3_45=Pool(ConvImg3_35,WP4) # reducing dimensionality by 2.
    PoolImg3_46=Pool(ConvImg3_36,WP4) # reducing dimensionality by 2.
    
    PoolImg4_47=Pool(ConvImg4_37,WP4) # reducing dimensionality by 2.
    PoolImg4_48=Pool(ConvImg4_38,WP4) # reducing dimensionality by 2.
    

    # Now we have the outputs and we make 
    x1=np.inner(PoolImg1_41.flatten(),W1)+np.inner(PoolImg2_43.flatten(),W3)+np.inner(PoolImg3_45.flatten(),W5)+np.inner(PoolImg4_47.flatten(),W7)
    x2=np.inner(PoolImg1_42.flatten(),W2)+np.inner(PoolImg2_44.flatten(),W4)+np.inner(PoolImg3_46.flatten(),W6)+np.inner(PoolImg4_48.flatten(),W8)
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
    W1=W1+ n1*(t1-f1)*PoolImg1_41.flatten()
    W3=W3+ n1*(t1-f1)*PoolImg2_43.flatten()
    W5=W5+ n1*(t1-f1)*PoolImg3_45.flatten()
    W7=W7+ n1*(t1-f1)*PoolImg4_47.flatten()
        
    W2=W2+ n1*(t2-f2)*PoolImg1_42.flatten()
    W4=W4+ n1*(t2-f2)*PoolImg2_44.flatten() 
    W6=W6+ n1*(t2-f2)*PoolImg3_46.flatten() 
    W8=W8+ n1*(t2-f2)*PoolImg4_48.flatten()
    b1=b1-n2*(t1-f1)
    b2=b2-n2*(t2-f2)
#     print(f1,f2,x1,b1)
```


```
def FileExistCheck(file):
    try:
        img = Image.open(file)
        return('exist')
    except: return('not exist')
im=[]
root='City\images\C4_'
for k in range(1,250): #it will start from 1 due to filenames
    if (k<10): file=root+'0'+str(k)+'.png'
    else: file=str(root)+str(k)+'.png'
    if(FileExistCheck(file)=='not exist'): 
        continue # if file does not exist it stops the current iteration
    img = Image.open(file)
    data=img.convert('RGB')
    data = np.asarray( data, dtype="int32" )
    data=np.transpose(data) # This can be improved, but this was easy since we have een 3D matrix
    #L1
    ConvImg1=sig.convolve(data[0,:,:],WC11,mode='valid') 
    ConvImg2=sig.convolve(data[0,:,:],WC12,mode='valid') 
    ConvImg3=sig.convolve(data[0,:,:],WC13,mode='valid') 
    ConvImg4=sig.convolve(data[0,:,:],WC14,mode='valid') 
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    PoolImg1_41=Pool(ConvImg1_31,WP4) # reducing dimensionality by 2.
    PoolImg1_42=Pool(ConvImg1_32,WP4) # reducing dimensionality by 2.
    
    PoolImg2_43=Pool(ConvImg2_33,WP4) # reducing dimensionality by 2.
    PoolImg2_44=Pool(ConvImg2_34,WP4) # reducing dimensionality by 2.
    
    PoolImg3_45=Pool(ConvImg3_35,WP4) # reducing dimensionality by 2.
    PoolImg3_46=Pool(ConvImg3_36,WP4) # reducing dimensionality by 2.
    
    PoolImg4_47=Pool(ConvImg4_37,WP4) # reducing dimensionality by 2.
    PoolImg4_48=Pool(ConvImg4_38,WP4) # reducing dimensionality by 2.
    

    # Now we have the outputs and we make 
    x1=np.inner(PoolImg1_41.flatten(),W1)+np.inner(PoolImg2_43.flatten(),W3)+np.inner(PoolImg3_45.flatten(),W5)+np.inner(PoolImg4_47.flatten(),W7)
    x2=np.inner(PoolImg1_42.flatten(),W2)+np.inner(PoolImg2_44.flatten(),W4)+np.inner(PoolImg3_46.flatten(),W6)+np.inner(PoolImg4_48.flatten(),W8)
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)  
#     print(f1,f2)
    if(f1>0.5): im=np.append(im,file)
```


```
import IPython.display as dis

for k in range(0,len(im)):
    dis.display(dis.Image(filename=im[k]))
```

# More classes!


```
# %%timeit
import matplotlib.pyplot as plt
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
import Image
RGB=1 #selects layer R=0,G=1,B=2

img = Image.open('Forest\F1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F4.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF4=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C4.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C5.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC4=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])


img = Image.open('Water\W1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])


```


```
dataPatchedF4.shape
```




    (11L, 26L, 50L, 50L)




```
def Sigmoid(x): # a is used to adjust the slope of the function 
    return (1/(1+np.exp(-x)))

def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

scale=1*10**-3 # at last output-layer
scale_WP=1*10**-4 #to reduce the end-values those can become very high!
WC11=np.array([
    [1,1,1,1,1],
    [1,2,2,2,1],
    [1,2,4,2,1],
    [1,2,2,2,1],
    [1,1,1,1,1],    
    ])

WC12=np.array([
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    ])

WC13=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [0.5,0.5,0.5,0.5,0.5],
    [0,0,0,0,0],
    [0,0,0,0,0]
    ])

WC14=np.array([
    [0.5,1,1,1,1],
    [0,0.5,1,1,1],
    [0,0,0.5,1,1],
    [0,0,0,0.5,1],
    [0,0,0,0,0.5]
    ])

WC5=np.array([
    [0.5,0,0,0,0],
    [1,0.5,0,0,0],
    [1,1,0.5,0,0],
    [1,1,1,0.5,0],
    [1,1,1,1,0.5]
    ])
# averaging kernel in L2
WP=np.array([ 
    [1,1],
    [1,1],
    ])

WC31=np.array([ 
    [1,1,1,1],
    [1,1,1,2],
    [1,1,1,2],
    [1,1,1,1],
    ])

WC32=np.array([ 
    [1,0,0,0],
    [1,1,0,0],
    [1,1,1,0],
    [1,1,1,1],
    ])



WP4=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    ])*scale_WP

# we initialize random weight to prevent oscillations while training
W1_1=np.random.rand(1,16)
W1_2=np.random.rand(1,16)
W1_3=np.random.rand(1,16)
W1_4=np.random.rand(1,16)
W1_5=np.random.rand(1,16)
W1_6=np.random.rand(1,16)
W1_7=np.random.rand(1,16)
W1_8=np.random.rand(1,16)

W2_1=np.random.rand(1,16)
W2_2=np.random.rand(1,16)
W2_3=np.random.rand(1,16)
W2_4=np.random.rand(1,16)
W2_5=np.random.rand(1,16)
W2_6=np.random.rand(1,16)
W2_7=np.random.rand(1,16)
W2_8=np.random.rand(1,16)

W3_1=np.random.rand(1,16)
W3_2=np.random.rand(1,16)
W3_3=np.random.rand(1,16)
W3_4=np.random.rand(1,16)
W3_5=np.random.rand(1,16)
W3_6=np.random.rand(1,16)
W3_7=np.random.rand(1,16)
W3_8=np.random.rand(1,16)

b1=0
b2=0
b3=0

Input=1
i=0
j=0
for u in range(0,4):
    if(Input==1):
        if(j==24 and i==9):
            print('stop')
            break # Searchspace of 2000x2000 is achieved and the loop will stop
        if(i==9): # i must increase to cover all patches over y-direction 
            j=j+1 # selects new patch in x direction
            i=0
        i=i+1
        classType='Forest'
        inputPatch=dataPatchedF1[i,j,:,:] 
        t1=1 # f1 is classifier for forest
        t2=0 # classifier for city
        t3=0
        Input=2
    elif(Input==2):
        classType='City'
        inputPatch=dataPatchedC1[i,j,:,:]
        t1=0 
        t2=1 
        t3=0
        Input=3        
    elif(Input==3):
        classType='Water'
        inputPatch=dataPatchedW1[i,j,:,:]
        t1=0
        t2=0
        t3=1
        Input=4       
    elif(Input==4):
        classType='Forest'
        inputPatch=dataPatchedF2[i,j,:,:]
        t1=1
        t2=0
        t3=0
        Input=5
    elif(Input==5):
        classType='City'
        inputPatch=dataPatchedC2[i,j,:,:]
        t1=0
        t2=1
        t3=0
        Input=6
    elif(Input==6):
        classType='Water'
        inputPatch=dataPatchedW2[i,j,:,:]
        t1=0
        t2=0
        t3=1
        Input=7
    elif(Input==7):
        classType='Forest'
        inputPatch=dataPatchedF3[i,j,:,:]
        t1=1
        t2=0
        t3=0
        Input=8
    elif(Input==8):
        classType='City'
        inputPatch=dataPatchedC3[i,j,:,:]
        t1=0
        t2=1
        t3=0
        Input=9
    elif(Input==9):
        classType='Water'
        inputPatch=dataPatchedW3[i,j,:,:]
        t1=0
        t2=0
        t3=1
        Input=10
    elif(Input==10):
        classType='Forest'
        inputPatch=dataPatchedF4[i,j,:,:]
        t1=1
        t2=0
        t3=0
        Input=11
    elif(Input==11):
        classType='City'
        inputPatch=dataPatchedC4[i,j,:,:]
        t1=0
        t2=1
        t3=0
        Input=1

    if(classType=='Water'):continue
    #L1
    ConvImg1=sig.convolve(inputPatch,WC11,mode='valid')
    ConvImg2=sig.convolve(inputPatch,WC12,mode='valid')
    ConvImg3=sig.convolve(inputPatch,WC13,mode='valid')
    ConvImg4=sig.convolve(inputPatch,WC14,mode='valid')
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    # reducing dimensionality by 2.
    PoolImg1_41=Pool(ConvImg1_31,WP4)*scale
    PoolImg1_42=Pool(ConvImg1_32,WP4)*scale 
    
    PoolImg2_43=Pool(ConvImg2_33,WP4)*scale 
    PoolImg2_44=Pool(ConvImg2_34,WP4)*scale 
    
    PoolImg3_45=Pool(ConvImg3_35,WP4)*scale
    PoolImg3_46=Pool(ConvImg3_36,WP4)*scale 
    
    PoolImg4_47=Pool(ConvImg4_37,WP4)*scale
    PoolImg4_48=Pool(ConvImg4_38,WP4)*scale
#     print(PoolImg4_48)
    
    n1=5*10**0
    n2=1*10**-1
    
    # Now we have the outputs
    x1=(
        np.inner(PoolImg1_41.flatten(),W1_1)+ 
        np.inner(PoolImg1_42.flatten(),W1_2)+
        np.inner(PoolImg2_43.flatten(),W1_3)+
        np.inner(PoolImg2_44.flatten(),W1_4)+
        np.inner(PoolImg3_45.flatten(),W1_5)+
        np.inner(PoolImg3_46.flatten(),W1_6)+
        np.inner(PoolImg4_47.flatten(),W1_7)+
        np.inner(PoolImg4_48.flatten(),W1_8)
        )
    x2=(
        np.inner(PoolImg1_41.flatten(),W2_1)+ 
        np.inner(PoolImg1_42.flatten(),W2_2)+
        np.inner(PoolImg2_43.flatten(),W2_3)+
        np.inner(PoolImg2_44.flatten(),W2_4)+
        np.inner(PoolImg3_45.flatten(),W2_5)+
        np.inner(PoolImg3_46.flatten(),W2_6)+
        np.inner(PoolImg4_47.flatten(),W2_7)+
        np.inner(PoolImg4_48.flatten(),W2_8)
        )
#     x3=(np.inner(PoolImg1_41.flatten(),W3_1)+ 
#         np.inner(PoolImg1_42.flatten(),W3_2)+
#         np.inner(PoolImg2_43.flatten(),W3_3)+
#         np.inner(PoolImg2_44.flatten(),W3_4)+
#         np.inner(PoolImg3_45.flatten(),W3_5)+
#         np.inner(PoolImg3_46.flatten(),W3_6)+
#         np.inner(PoolImg4_47.flatten(),W3_7)+
#         np.inner(PoolImg4_48.flatten(),W3_8)
#         )
    x1=x1[0]
    x2=x2[0]
#     x3=x3[0]
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
#     f3=Sigmoid(x3-b3)
    
    f11=1 #(np.exp(-x1))/(1+np.exp(-x1)**2)
    f22=1 #(np.exp(-x2))/(1+np.exp(-x2)**2)
    
    W1_1=W1_1+ n1*(t1-f1)*PoolImg1_41.flatten()*f11
    W1_2=W1_2+ n1*(t1-f1)*PoolImg1_42.flatten()*f11
    W1_3=W1_3+ n1*(t1-f1)*PoolImg2_43.flatten()*f11
    W1_4=W1_4+ n1*(t1-f1)*PoolImg2_44.flatten()*f11
    W1_5=W1_5+ n1*(t1-f1)*PoolImg3_45.flatten()*f11
    W1_6=W1_6+ n1*(t1-f1)*PoolImg3_46.flatten()*f11
    W1_7=W1_7+ n1*(t1-f1)*PoolImg4_47.flatten()*f11
    W1_8=W1_8+ n1*(t1-f1)*PoolImg4_48.flatten()*f11
    
    W2_1=W2_1+ n1*(t2-f2)*PoolImg1_41.flatten()*f22
    W2_2=W2_2+ n1*(t2-f2)*PoolImg1_42.flatten()*f22
    W2_3=W2_3+ n1*(t2-f2)*PoolImg2_43.flatten()*f22
    W2_4=W2_4+ n1*(t2-f2)*PoolImg2_44.flatten()*f22
    W2_5=W2_5+ n1*(t2-f2)*PoolImg3_45.flatten()*f22
    W2_6=W2_6+ n1*(t2-f2)*PoolImg3_46.flatten()*f22
    W2_7=W2_7+ n1*(t2-f2)*PoolImg4_47.flatten()*f22
    W2_8=W2_8+ n1*(t2-f2)*PoolImg4_48.flatten()*f22
    
#     W3_1=W3_1+ n1*(t3-f3)*PoolImg1_41.flatten()
#     W3_2=W3_2+ n1*(t3-f3)*PoolImg1_42.flatten()
#     W3_3=W3_3+ n1*(t3-f3)*PoolImg2_43.flatten()
#     W3_4=W3_4+ n1*(t3-f3)*PoolImg2_44.flatten()
#     W3_5=W3_5+ n1*(t3-f3)*PoolImg3_45.flatten()
#     W3_6=W3_6+ n1*(t3-f3)*PoolImg3_46.flatten()
#     W3_7=W3_7+ n1*(t3-f3)*PoolImg4_47.flatten()
#     W3_8=W3_8+ n1*(t3-f3)*PoolImg4_48.flatten()
    b1=b1-n2*(t1-f1)
    b2=b2-n2*(t2-f2)
#     b3=b3-n2*(t3-f3)
#     if(u%2==0): print(classType,f1,f2,x1,b1)
```


```

```


```
%matplotlib inline
img = Image.open('City\C2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])
i=0
j=0
for k in range(1,1000): #it will start from 1 due to filenames
    if(j==24 and i==9):
            print('stop')
            break # Searchspace of 2000x2000 is achieved and the loop will stop
    if(i==9): # i must increase to cover all patches over y-direction 
        j=j+1 # selects new patch in x direction
        i=0
    i=i+1
    inputPatch=dataPatchedW2[i,j,:,:]
     #L1
    ConvImg1=sig.convolve(inputPatch,WC11,mode='valid')
    ConvImg2=sig.convolve(inputPatch,WC12,mode='valid')
    ConvImg3=sig.convolve(inputPatch,WC13,mode='valid')
    ConvImg4=sig.convolve(inputPatch,WC14,mode='valid')
    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid')
    
    ConvImg2_33=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg2_34=sig.convolve(PoolImg2,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg3_35=sig.convolve(PoolImg3,WC31,mode='valid') 
    ConvImg3_36=sig.convolve(PoolImg3,WC32,mode='valid')
    
    ConvImg4_37=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg4_38=sig.convolve(PoolImg4,WC32,mode='valid')
    
    #L4
    # reducing dimensionality by 2.
    PoolImg1_41=Pool(ConvImg1_31,WP4)*scale
    PoolImg1_42=Pool(ConvImg1_32,WP4)*scale 
    
    PoolImg2_43=Pool(ConvImg2_33,WP4)*scale 
    PoolImg2_44=Pool(ConvImg2_34,WP4)*scale 
    
    PoolImg3_45=Pool(ConvImg3_35,WP4)*scale
    PoolImg3_46=Pool(ConvImg3_36,WP4)*scale 
    
    PoolImg4_47=Pool(ConvImg4_37,WP4)*scale
    PoolImg4_48=Pool(ConvImg4_38,WP4)*scale
    
    # Now we have the outputs
    x1=(
        np.inner(PoolImg1_41.flatten(),W1_1)+ 
        np.inner(PoolImg1_42.flatten(),W1_2)+
        np.inner(PoolImg2_43.flatten(),W1_3)+
        np.inner(PoolImg2_44.flatten(),W1_4)+
        np.inner(PoolImg3_45.flatten(),W1_5)+
        np.inner(PoolImg3_46.flatten(),W1_6)+
        np.inner(PoolImg4_47.flatten(),W1_7)+
        np.inner(PoolImg4_48.flatten(),W1_8)
        )
    x2=(
        np.inner(PoolImg1_41.flatten(),W2_1)+ 
        np.inner(PoolImg1_42.flatten(),W2_2)+
        np.inner(PoolImg2_43.flatten(),W2_3)+
        np.inner(PoolImg2_44.flatten(),W2_4)+
        np.inner(PoolImg3_45.flatten(),W2_5)+
        np.inner(PoolImg3_46.flatten(),W2_6)+
        np.inner(PoolImg4_47.flatten(),W2_7)+
        np.inner(PoolImg4_48.flatten(),W2_8)
        )
#     x3=(np.inner(PoolImg1_41.flatten(),W3_1)+ 
#         np.inner(PoolImg1_42.flatten(),W3_2)+
#         np.inner(PoolImg2_43.flatten(),W3_3)+
#         np.inner(PoolImg2_44.flatten(),W3_4)+
#         np.inner(PoolImg3_45.flatten(),W3_5)+
#         np.inner(PoolImg3_46.flatten(),W3_6)+
#         np.inner(PoolImg4_47.flatten(),W3_7)+
#         np.inner(PoolImg4_48.flatten(),W3_8)
#         )
    x1=x1[0]
    x2=x2[0]
#     x3=x3[0]
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
#     print(f1,f2)
#     if(f1>0.7): 
#         plt.figure(k)
#         im=plt.imshow(inputPatch)
```

# Try something else

\begin{equation}
f(x)=\frac{1}{1+e^{-x}} \\
F_{k}= f( \sum_{i} \mathbf{W_{i} \cdot y_{i}}-b_{k})
\end{equation}


\begin{equation}
E=\sum_{k} \frac{1}{2}|t_k-F_{k}|^{2} \\
\Delta W_{ij}= - \eta \frac{dE}{d W_{ij}}
\end{equation}

\begin{equation}
\Delta W_{ij}= \sum_{k} - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{dW_{ij}}=\sum_{k} \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \frac{dx_{k}}{dW_{ij}} \\
= \eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} y_{ij}
\end{equation}

\begin{equation}
\Delta b_{k}= - \eta \frac{dE}{d F_{k}} \frac{dF_{k}}{dx_{k}} \frac{dx_{k}}{b_{k}}=\eta (t_{k}-F_{k})\frac{e^{-x_{k}}}{(1+e^{-x_{k}})^{2}} \cdot-1
\end{equation}


```
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from numpy import linalg as lin
import scipy.signal as sig
import Image
RGB=2 #selects layer R=0,G=1,B=2

img = Image.open('Forest\F1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F5.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F4.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF4=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Forest\F5.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedF5=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])


img = Image.open('City\C3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C4.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C5.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C6.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC4=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('City\C7.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedC5=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])


img = Image.open('Water\W1.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW1=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W2.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW2=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W3.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW3=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W4.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW4=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])

img = Image.open('Water\W5.png')
data=img.convert('RGB')
data = np.asarray( data, dtype="int32" )
patchSize=50 # patchsize this must be 50 since our network can only handle this value
Yamount=data.shape[0]/patchSize # Counts how many times the windowsize fits in the picture
Xamount=data.shape[1]/patchSize # Counts how many times the windowsize fits in the picture
dataPatchedW5=np.array([[data[j*patchSize:(j+1)*patchSize,i*patchSize:(i+1)*patchSize,RGB] for i in range(0,Xamount)] for j in range(0,Yamount)])



```


```
dataPatchedW4.shape
```




    (11L, 26L, 50L, 50L)




```
def Sigmoid(x): # a is used to adjust the slope of the function 
    return (1/(1+np.exp(-x)))

def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

scale=1*10**-3 # at last output-layer
scale_WP=1*10**-4 #to reduce the end-values those can become very high!
WC11=np.array([
    [1,1,1,1,1],
    [1,2,2,2,1],
    [1,2,4,2,1],
    [1,2,2,2,1],
    [1,1,1,1,1],    
    ])

WC12=np.array([
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    ])

WC13=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [0.5,0.5,0.5,0.5,0.5],
    [0,0,0,0,0],
    [0,0,0,0,0]
    ])

WC14=np.array([
    [0.5,1,1,1,1],
    [0,0.5,1,1,1],
    [0,0,0.5,1,1],
    [0,0,0,0.5,1],
    [0,0,0,0,0.5]
    ])

WC15=np.array([
    [0.5,0,0,0,0],
    [1,0.5,0,0,0],
    [1,1,0.5,0,0],
    [1,1,1,0.5,0],
    [1,1,1,1,0.5]
    ])

WC16=np.array([
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1]
    ])

# averaging kernel in L2
WP=np.array([ 
    [1,1],
    [1,1],
    ])

WC31=np.array([ 
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    ])

WC32=np.array([ 
    [1,0,0,0],
    [1,1,0,0],
    [1,1,1,0],
    [1,1,1,1],
    ])



WP4=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    ])*scale_WP

# we initialize random weight to prevent oscillations while training
W1_1=np.random.rand(1,16)
W1_2=np.random.rand(1,16)
W1_3=np.random.rand(1,16)
W1_4=np.random.rand(1,16)
W1_5=np.random.rand(1,16)
W1_6=np.random.rand(1,16)
W1_7=np.random.rand(1,16)
W1_8=np.random.rand(1,16)

W2_1=np.random.rand(1,16)
W2_2=np.random.rand(1,16)
W2_3=np.random.rand(1,16)
W2_4=np.random.rand(1,16)
W2_5=np.random.rand(1,16)
W2_6=np.random.rand(1,16)
W2_7=np.random.rand(1,16)
W2_8=np.random.rand(1,16)

W3_1=np.random.rand(1,16)
W3_2=np.random.rand(1,16)
W3_3=np.random.rand(1,16)
W3_4=np.random.rand(1,16)
W3_5=np.random.rand(1,16)
W3_6=np.random.rand(1,16)
W3_7=np.random.rand(1,16)
W3_8=np.random.rand(1,16)

b1=0
b2=0
b3=0




data_list=([[dataPatchedF1,dataPatchedF2,dataPatchedF3,dataPatchedF4,dataPatchedF5],
           [dataPatchedC1,dataPatchedC2,dataPatchedC3,dataPatchedC4,dataPatchedC5],
           [dataPatchedW1,dataPatchedW2,dataPatchedW3,dataPatchedW4,dataPatchedW5]
            ])

class_list=[[1,0,0],
            [0,1,0],
            [0,0,1]
            ]

i=0
j=0
d1=0
d2=0

for u in range(0,4000):
    if(d1==3): 
        d1=0 
        d2=d2+1 
    if(d2==5): 
        d2=0
        if(j==24 and i==9):
            print('stop')
            break # Searchspace of 2000x2000 is achieved and the loop will stop
        if(i==9): # i must increase to cover all patches over y-direction 
            j=j+1 # selects new patch in x direction
            i=0
        i=i+1
    inputPatch=data_list[d1][d2][i,j,:,:]  
    t1,t2,t3=class_list[d1]
#     print(t1,t2)
    d1=d1+1
        
    #L1
    ConvImg1=sig.convolve(inputPatch,WC11,mode='valid')
    ConvImg2=sig.convolve(inputPatch,WC12,mode='valid')
    ConvImg3=sig.convolve(inputPatch,WC13,mode='valid')
    ConvImg4=sig.convolve(inputPatch,WC14,mode='valid')
    ConvImg5=sig.convolve(inputPatch,WC15,mode='valid')
    ConvImg6=sig.convolve(inputPatch,WC16,mode='valid')


    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    PoolImg5=Pool(ConvImg5,WP)
    PoolImg6=Pool(ConvImg6,WP)
    
    #L3
    ConvImg1_1=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg2_1=sig.convolve(PoolImg2,WC31,mode='valid') 
    ConvImg3_1=sig.convolve(PoolImg3,WC31,mode='valid')  
    ConvImg4_1=sig.convolve(PoolImg4,WC31,mode='valid') 
    ConvImg5_1=sig.convolve(PoolImg5,WC31,mode='valid')
    ConvImg6_1=sig.convolve(PoolImg6,WC31,mode='valid')

    #L4
    # reducing dimensionality by 2.
    PoolImg1_1=Pool(ConvImg1_1,WP4)*scale
    PoolImg2_1=Pool(ConvImg2_1,WP4)*scale 
    PoolImg3_1=Pool(ConvImg3_1,WP4)*scale 
    PoolImg4_1=Pool(ConvImg4_1,WP4)*scale 
    PoolImg5_1=Pool(ConvImg5_1,WP4)*scale
    PoolImg6_1=Pool(ConvImg6_1,WP4)*scale
#     plt.figure()
#     plt.imshow(PoolImg6_1)
#     plt.xlabel(classType)
    
    n1=2*10**-1
    n2=1*10**0
    
    # Now we have the outputs
    x1=(
        np.inner(PoolImg1_1.flatten(),W1_1)+ 
        np.inner(PoolImg2_1.flatten(),W1_2)+
        np.inner(PoolImg3_1.flatten(),W1_3)+
        np.inner(PoolImg4_1.flatten(),W1_4)+
        np.inner(PoolImg5_1.flatten(),W1_5)+
        np.inner(PoolImg6_1.flatten(),W1_6)
        )
    x2=(
        np.inner(PoolImg1_1.flatten(),W2_1)+ 
        np.inner(PoolImg2_1.flatten(),W2_2)+
        np.inner(PoolImg3_1.flatten(),W2_3)+
        np.inner(PoolImg4_1.flatten(),W2_4)+
        np.inner(PoolImg5_1.flatten(),W2_5)+
        np.inner(PoolImg6_1.flatten(),W2_6)
        )
    x3=(
        np.inner(PoolImg1_1.flatten(),W3_1)+ 
        np.inner(PoolImg2_1.flatten(),W3_2)+
        np.inner(PoolImg3_1.flatten(),W3_3)+
        np.inner(PoolImg4_1.flatten(),W3_4)+
        np.inner(PoolImg5_1.flatten(),W3_5)+
        np.inner(PoolImg6_1.flatten(),W3_6)
        )
    x1=x1[0]
    x2=x2[0]
    x3=x3[0]
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
    f3=Sigmoid(x3-b3)
    
    W1_1=W1_1+ n1*(t1-f1)*PoolImg1_1.flatten()
    W1_2=W1_2+ n1*(t1-f1)*PoolImg2_1.flatten()
    W1_3=W1_3+ n1*(t1-f1)*PoolImg3_1.flatten()
    W1_4=W1_4+ n1*(t1-f1)*PoolImg4_1.flatten()
    W1_5=W1_5+ n1*(t1-f1)*PoolImg5_1.flatten()
    W1_6=W1_6+ n1*(t1-f1)*PoolImg6_1.flatten()
    
    W2_1=W2_1+ n1*(t2-f2)*PoolImg1_1.flatten()
    W2_2=W2_2+ n1*(t2-f2)*PoolImg2_1.flatten()
    W2_3=W2_3+ n1*(t2-f2)*PoolImg3_1.flatten()
    W2_4=W2_4+ n1*(t2-f2)*PoolImg4_1.flatten()
    W2_5=W2_5+ n1*(t2-f2)*PoolImg5_1.flatten()
    W2_6=W2_6+ n1*(t2-f2)*PoolImg6_1.flatten()
    
    W3_1=W3_1+ n1*(t3-f3)*PoolImg1_1.flatten()
    W3_2=W3_2+ n1*(t3-f3)*PoolImg2_1.flatten()
    W3_3=W3_3+ n1*(t3-f3)*PoolImg3_1.flatten()
    W3_4=W3_4+ n1*(t3-f3)*PoolImg4_1.flatten()
    W3_5=W3_5+ n1*(t3-f3)*PoolImg5_1.flatten()
    W3_6=W3_6+ n1*(t3-f3)*PoolImg6_1.flatten()
    
    
    b1=b1-n2*(t1-f1)
    b2=b2-n2*(t2-f2)
    b3=b2-n2*(t3-f3)

    if(u%10==0): print(t1,t2,t3,'|',f1,f2,f3)
# 
```

    (1, 0, 0, '|', 0.99973426289487932, 0.99967912899052147, 0.99980861302617696)
    (0, 1, 0, '|', 0.9254186840985501, 0.88055604384809738, 0.91230611051673927)
    (0, 0, 1, '|', 0.72499375197092653, 0.97318795713578432, 0.93049086569275852)
    (1, 0, 0, '|', 0.068306907897917726, 0.23367962217743365, 0.23641246453133813)
    (0, 1, 0, '|', 0.53999243479344083, 0.61234660017052434, 0.61749459774796001)
    (0, 0, 1, '|', 0.064996944467150625, 0.93847679512230775, 0.32479323380793768)
    (1, 0, 0, '|', 0.17640949724811947, 0.20499209593914333, 0.24127490649969055)
    (0, 1, 0, '|', 0.24098170142264264, 0.51076733813463771, 0.4872664470111881)
    (0, 0, 1, '|', 0.053751647506200144, 0.95557981648139578, 0.90313660068985313)
    (1, 0, 0, '|', 0.20411104332037336, 0.25119843459922964, 0.35067995566870491)
    (0, 1, 0, '|', 0.11378462327293028, 0.53834038361272241, 0.36873311439832218)
    (0, 0, 1, '|', 0.025595661798408911, 0.96431795249413965, 0.87011791687455287)
    (1, 0, 0, '|', 0.16901908015616507, 0.36741690118128778, 0.40323912358301089)
    (0, 1, 0, '|', 0.14354391294796937, 0.50031728638180328, 0.285492070165206)
    (0, 0, 1, '|', 0.016462275353789125, 0.96072563987386728, 0.83870646076451638)
    (1, 0, 0, '|', 0.10853461285883233, 0.40953584688343925, 0.46191921394403646)
    (0, 1, 0, '|', 0.090738520854325544, 0.58698469197161085, 0.48547081175008988)
    (0, 0, 1, '|', 0.020936048027135928, 0.94555299692637129, 0.50098168424145662)
    (1, 0, 0, '|', 0.1635396859333004, 0.31678839435081341, 0.37629092318895174)
    (0, 1, 0, '|', 0.042901311156176514, 0.84909890799057197, 0.80277769274068245)
    (0, 0, 1, '|', 0.010088169012123307, 0.95563400405542942, 0.85686809988183976)
    (1, 0, 0, '|', 0.29818818984371204, 0.18424039147040813, 0.29989154611777108)
    (0, 1, 0, '|', 0.086999591718110961, 0.471076543433541, 0.32019205214538271)
    (0, 0, 1, '|', 0.0029323167397823863, 0.97034141339931179, 0.91020108907202013)
    (1, 0, 0, '|', 0.32339260182040175, 0.1968526156926646, 0.29093548242297929)
    (0, 1, 0, '|', 0.074378068623673646, 0.48875709324865357, 0.24484230557017272)
    (0, 0, 1, '|', 0.0056502365531948505, 0.94886931148542042, 0.76621926732501133)
    (1, 0, 0, '|', 0.10506614456782608, 0.41460649903406066, 0.45536637221810528)
    (0, 1, 0, '|', 0.16059364176453342, 0.33160729075039674, 0.20462618557783083)
    (0, 0, 1, '|', 0.0055500054681886079, 0.95491079682891333, 0.78425997896466526)
    (1, 0, 0, '|', 0.092734975386852575, 0.39708120965563565, 0.4849342382822599)
    (0, 1, 0, '|', 0.017642580646379906, 0.74294988093353365, 0.63206971282952029)
    (0, 0, 1, '|', 0.0025604856244509418, 0.95283837510492198, 0.8656506611571364)
    (1, 0, 0, '|', 0.21177511705266561, 0.28883884168987284, 0.41283578246742675)
    (0, 1, 0, '|', 0.02496128332748429, 0.63615994562166855, 0.60661317025901895)
    (0, 0, 1, '|', 0.002693504524916567, 0.93007757170503125, 0.8329232494170633)
    (1, 0, 0, '|', 0.29150267420977233, 0.19543599090703836, 0.2727544586545137)
    (0, 1, 0, '|', 0.032776194563342784, 0.51667126702950805, 0.38851790408851039)
    (0, 0, 1, '|', 0.00098943121767206162, 0.95864991998511839, 0.9180837601212678)
    (1, 0, 0, '|', 0.21963676186136283, 0.25437171826588573, 0.3510587776454549)
    (0, 1, 0, '|', 0.045252145484916191, 0.51979767300971802, 0.35600406655606737)
    (0, 0, 1, '|', 0.0028160140771098605, 0.93799348494839252, 0.83237384336363651)
    (1, 0, 0, '|', 0.069218861432156498, 0.45454813425386698, 0.64246138410489573)
    (0, 1, 0, '|', 0.021393941486862196, 0.65560437799536442, 0.56669507673577724)
    (0, 0, 1, '|', 0.0016846744149035882, 0.94343957807833911, 0.7928447887929938)
    (1, 0, 0, '|', 0.0426477975735313, 0.59666958757782551, 0.66727421883780691)
    (0, 1, 0, '|', 0.057884029152824749, 0.50053848476519003, 0.44344097927362675)
    (0, 0, 1, '|', 0.0014977264332955229, 0.93935010107461969, 0.88704903782164846)
    (1, 0, 0, '|', 0.2234847594565115, 0.29324125362121406, 0.39501117517157053)
    (0, 1, 0, '|', 0.051528848008108502, 0.4664254306617282, 0.35890520520338837)
    (0, 0, 1, '|', 0.0013513482423153512, 0.89005034481480927, 0.75320609389494997)
    (1, 0, 0, '|', 0.16098968155115342, 0.34108495290505741, 0.45892244313280572)
    (0, 1, 0, '|', 0.043928358664075677, 0.47667942367674199, 0.38880201017822963)
    (0, 0, 1, '|', 0.0014970541003075264, 0.93512667426932106, 0.86589994330387121)
    (1, 0, 0, '|', 0.25301013228856994, 0.28787745688455907, 0.41138298204492285)
    (0, 1, 0, '|', 0.10257929938684482, 0.32671569365516523, 0.34168384971694105)
    (0, 0, 1, '|', 0.0034480826102531608, 0.93155179210934569, 0.75007540739536449)
    (1, 0, 0, '|', 0.064238144466353994, 0.47120084681384489, 0.54754918396350427)
    (0, 1, 0, '|', 0.036603545784904698, 0.50931930203402365, 0.43005866109356888)
    (0, 0, 1, '|', 0.0011035792228370259, 0.9505585194220596, 0.85934916722417642)
    (1, 0, 0, '|', 0.080061918513161315, 0.45297596608857499, 0.58178777487696742)
    (0, 1, 0, '|', 0.06621352916296043, 0.36512139946084621, 0.37288849572010552)
    (0, 0, 1, '|', 0.0019179461176408715, 0.93879338027712667, 0.82386251296508972)
    (1, 0, 0, '|', 0.28847037993436753, 0.27458987106597166, 0.39954059253498075)
    (0, 1, 0, '|', 0.10769367428948524, 0.3985669739420698, 0.1922281623972599)
    (0, 0, 1, '|', 0.001453156754917742, 0.91814640896762001, 0.81852104754510979)
    (1, 0, 0, '|', 0.33367076073670765, 0.21546562769362076, 0.3282510411899251)
    (0, 1, 0, '|', 0.069377743568971015, 0.41208760878564848, 0.32445977223492195)
    (0, 0, 1, '|', 0.00094950817114740707, 0.92250493267340472, 0.88609911406866293)
    (1, 0, 0, '|', 0.073785288415251615, 0.40927289904947112, 0.51907232502847811)
    (0, 1, 0, '|', 0.032680387604700961, 0.52826388482406239, 0.41905186255838961)
    (0, 0, 1, '|', 0.0011097334974455582, 0.96097851561343495, 0.85362769875309297)
    (1, 0, 0, '|', 0.052226922598877838, 0.48031262327804403, 0.61663726987791601)
    (0, 1, 0, '|', 0.083384921067188617, 0.34825726368796894, 0.37748420736415561)
    (0, 0, 1, '|', 0.00081313497815626266, 0.93248279046769866, 0.89035845014503701)
    (1, 0, 0, '|', 0.17009237597285096, 0.31675883484250972, 0.54066926244205693)
    (0, 1, 0, '|', 0.019522465533023392, 0.47672725492242302, 0.52074379982803543)
    (0, 0, 1, '|', 0.00085465967321340702, 0.87560161308142015, 0.79915901305890902)
    (1, 0, 0, '|', 0.30740081173620676, 0.22833494349718003, 0.37537142270480445)
    (0, 1, 0, '|', 0.058559791170448385, 0.34972916804186271, 0.40195612290155491)
    (0, 0, 1, '|', 0.0012882378671997608, 0.86107291916858131, 0.80055493874125672)
    (1, 0, 0, '|', 0.3615504223962358, 0.2283392273669391, 0.39789023603134172)
    (0, 1, 0, '|', 0.034442878355307495, 0.41861063794601977, 0.33665041709199212)
    (0, 0, 1, '|', 0.0020232797326247697, 0.91672563161064458, 0.76358143672536205)
    (1, 0, 0, '|', 0.13590238190187198, 0.3945014390265027, 0.47942554156210099)
    (0, 1, 0, '|', 0.078486496479758555, 0.35231053471124768, 0.34413668895308902)
    (0, 0, 1, '|', 0.00091054881461760488, 0.89564132245953398, 0.83390105615714338)
    (1, 0, 0, '|', 0.098751732114695095, 0.38511676092117259, 0.52994907576124839)
    (0, 1, 0, '|', 0.04842460195256728, 0.37739473689854602, 0.41058967157172549)
    (0, 0, 1, '|', 0.0010520587732132717, 0.86394583508538347, 0.85028347339736265)
    (1, 0, 0, '|', 0.10203887706345295, 0.35241164298385214, 0.61442186346971739)
    (0, 1, 0, '|', 0.058923920866330713, 0.29588941944598451, 0.36284735647658267)
    (0, 0, 1, '|', 0.00087190927960381615, 0.82931771744904748, 0.83626569116207317)
    (1, 0, 0, '|', 0.22937436370887096, 0.2770915461484319, 0.42823726241456972)
    (0, 1, 0, '|', 0.051626467307624463, 0.41448340825903796, 0.36373808661393348)
    (0, 0, 1, '|', 0.0015468389176655762, 0.80093852866118442, 0.7719069868574715)
    (1, 0, 0, '|', 0.10941202353205079, 0.36149897573423034, 0.53248502187485336)
    (0, 1, 0, '|', 0.088060882702767546, 0.39945434007685876, 0.33480472325907046)
    (0, 0, 1, '|', 0.00088681414942510953, 0.87853019259127252, 0.8480519247254924)
    (1, 0, 0, '|', 0.12007813572225284, 0.33915272983695438, 0.51268290495511726)
    (0, 1, 0, '|', 0.078151810181146808, 0.33326701723204105, 0.39276459478729475)
    (0, 0, 1, '|', 0.0014170939040826479, 0.85422539496736383, 0.82685825229701471)
    (1, 0, 0, '|', 0.073299174844590489, 0.37808140580826638, 0.62927764224103111)
    (0, 1, 0, '|', 0.07471889386181145, 0.35669073735287293, 0.37798630595306698)
    (0, 0, 1, '|', 0.00097490257890251591, 0.78904175472667792, 0.84822121361667935)
    (1, 0, 0, '|', 0.18292898816636147, 0.28879097136429643, 0.5211614427649921)
    (0, 1, 0, '|', 0.11811833703045085, 0.26230289699591947, 0.31510878648694923)
    (0, 0, 1, '|', 0.0007008000213041387, 0.75540127628783971, 0.83917907200541741)
    (1, 0, 0, '|', 0.33680649203768487, 0.25652087891089276, 0.46228658650918847)
    (0, 1, 0, '|', 0.057387554803801956, 0.30934646677997962, 0.32690726789867192)
    (0, 0, 1, '|', 0.0013488513253292018, 0.87303923374266434, 0.69692189789295345)
    (1, 0, 0, '|', 0.039395206349424935, 0.45528023982649896, 0.63231716485813649)
    (0, 1, 0, '|', 0.082058150806660554, 0.30854279876981111, 0.36587779896658013)
    (0, 0, 1, '|', 0.001730738589254995, 0.90872060431887924, 0.73569760329665956)
    (1, 0, 0, '|', 0.080770214802000806, 0.38483169915617171, 0.56948795999351798)
    (0, 1, 0, '|', 0.096195387670832952, 0.33124105169035362, 0.28206774656412725)
    (0, 0, 1, '|', 0.0014257284508236068, 0.85566033177324452, 0.79627581043004403)
    (1, 0, 0, '|', 0.073715826210016586, 0.36205839854159727, 0.62097945303155555)
    (0, 1, 0, '|', 0.070112685555468318, 0.39076836534677373, 0.36029169172979936)
    (0, 0, 1, '|', 0.0012501155502603293, 0.74441367778879508, 0.79506888434383527)
    (1, 0, 0, '|', 0.13850430049047022, 0.3084857398079488, 0.56948797096839621)
    (0, 1, 0, '|', 0.035765219337700487, 0.4008827143774647, 0.39655409100442124)
    (0, 0, 1, '|', 0.00075968161976541434, 0.82500759313171435, 0.74062294607876034)
    (1, 0, 0, '|', 0.083228658080153325, 0.37893823339082833, 0.56765341099595934)
    (0, 1, 0, '|', 0.1329661282011125, 0.31056821433908255, 0.27182958805209761)
    (0, 0, 1, '|', 0.0025423066004061005, 0.88206841524252855, 0.71123071537350901)
    (1, 0, 0, '|', 0.074172244752274796, 0.39523138367399802, 0.5957350544859692)
    (0, 1, 0, '|', 0.034214254597064189, 0.4260840699430406, 0.41711124982738729)
    (0, 0, 1, '|', 0.0031828173826903379, 0.86588951071006937, 0.71718358715262664)
    (1, 0, 0, '|', 0.11050120080901925, 0.35785163179314794, 0.54711429651271548)
    (0, 1, 0, '|', 0.055293332950480931, 0.25049424041111068, 0.43839853127316408)
    (0, 0, 1, '|', 0.0030493323003167008, 0.8070268734458993, 0.70531519182776836)
    (1, 0, 0, '|', 0.11358206009937871, 0.33606540119674205, 0.54479066623020911)
    (0, 1, 0, '|', 0.037484713326981438, 0.37041477350284774, 0.48187096833106724)
    (0, 0, 1, '|', 0.0023138365386537281, 0.73327300149139896, 0.77389193188463723)
    (1, 0, 0, '|', 0.098727504010643466, 0.39731058211459391, 0.50571366811025709)
    (0, 1, 0, '|', 0.045555837782422698, 0.40264021302055364, 0.43639660747832792)
    (0, 0, 1, '|', 0.0017834338896284231, 0.9076691248595169, 0.8144118752510825)
    (1, 0, 0, '|', 0.081108493023989287, 0.36341162667594012, 0.62905922084667232)
    (0, 1, 0, '|', 0.22945787362977019, 0.28237332909990975, 0.22267126700389483)
    (0, 0, 1, '|', 0.0014452960772694948, 0.87739620193738144, 0.82945485662317842)
    (1, 0, 0, '|', 0.088265183034796912, 0.32645792361766662, 0.57603118973313794)
    (0, 1, 0, '|', 0.29918126531074224, 0.21905176006842772, 0.23193228147088055)
    (0, 0, 1, '|', 0.0053422568035385717, 0.87048399084607275, 0.59178677747725172)
    (1, 0, 0, '|', 0.071504710643875063, 0.38570022455100855, 0.58654332638327766)
    (0, 1, 0, '|', 0.049391521883582166, 0.4473101180776794, 0.36760367252359716)
    (0, 0, 1, '|', 0.0034375254667976061, 0.7411838585180256, 0.73598716696384581)
    (1, 0, 0, '|', 0.054678871073048932, 0.42499828062465789, 0.57644613012956403)
    (0, 1, 0, '|', 0.056363437129875363, 0.37935165907626522, 0.35792872642560181)
    (0, 0, 1, '|', 0.0037317063451647926, 0.7513067686281969, 0.76179310370433906)
    (1, 0, 0, '|', 0.12807298171227197, 0.35133812252515439, 0.52385599132768967)
    (0, 1, 0, '|', 0.10195149528116713, 0.32582229052615957, 0.37631315440267737)
    (0, 0, 1, '|', 0.001064556929216215, 0.90123420371476026, 0.86220035903655878)
    (1, 0, 0, '|', 0.086127478428512996, 0.37491917607048708, 0.60753219905132338)
    (0, 1, 0, '|', 0.061702530460612123, 0.34690623380103874, 0.36105285436600071)
    (0, 0, 1, '|', 0.0016820085037815291, 0.82914180750699196, 0.83103336695776031)
    (1, 0, 0, '|', 0.079667614230280029, 0.41578955387799077, 0.59197066722675562)
    (0, 1, 0, '|', 0.12211844692177613, 0.33414510187786717, 0.28718798751579128)
    (0, 0, 1, '|', 0.0062452614006804135, 0.77527998544359045, 0.69069109459105349)
    (1, 0, 0, '|', 0.084065728212394386, 0.37164110069815581, 0.5897945526414754)
    (0, 1, 0, '|', 0.14547869836905775, 0.27918493944383693, 0.32444599957205927)
    (0, 0, 1, '|', 0.0040531387891914036, 0.73554059808371119, 0.74711139307104546)
    (1, 0, 0, '|', 0.060866270915543232, 0.43056102018128539, 0.59545704351277751)
    (0, 1, 0, '|', 0.036832130164657299, 0.40731485161582731, 0.36147721239659203)
    (0, 0, 1, '|', 0.0030257900043585972, 0.87646261306283835, 0.74994957664559292)
    (1, 0, 0, '|', 0.12280209804726644, 0.36581466402728996, 0.55476725697975693)
    (0, 1, 0, '|', 0.12514346529712395, 0.30113588325947827, 0.34476953437021857)
    (0, 0, 1, '|', 0.002371339043402147, 0.83711432409059416, 0.8158038833811817)
    (1, 0, 0, '|', 0.075026717098296822, 0.37610025730397784, 0.6488784840768943)
    (0, 1, 0, '|', 0.070285280643533468, 0.3583149396209902, 0.3807279345243928)
    (0, 0, 1, '|', 0.004034352566903536, 0.69702780372606099, 0.76487589267971523)
    (1, 0, 0, '|', 0.12145352862959961, 0.33641675106003616, 0.54854460564793084)
    (0, 1, 0, '|', 0.11211734291394354, 0.36628920627464345, 0.36448618623292017)
    (0, 0, 1, '|', 0.0080803279969311591, 0.73122097977276024, 0.63531443600347404)
    (1, 0, 0, '|', 0.087573853908082797, 0.39719818034349119, 0.56577562728589614)
    (0, 1, 0, '|', 0.032627736681494539, 0.48866132341632512, 0.28839452949511002)
    (0, 0, 1, '|', 0.0076519680923050875, 0.70801609456733872, 0.5689294221492408)
    (1, 0, 0, '|', 0.10062750511828661, 0.35823900291135391, 0.48671779843930335)
    (0, 1, 0, '|', 0.11147528310978075, 0.2781462666111173, 0.3695309664742798)
    (0, 0, 1, '|', 0.0020659252332876432, 0.8886465014425583, 0.84859530550025919)
    (1, 0, 0, '|', 0.10111281539767178, 0.36935216580211228, 0.58775784427385613)
    (0, 1, 0, '|', 0.11522760530211079, 0.29427399318772957, 0.34551056673714525)
    (0, 0, 1, '|', 0.0074419417235103268, 0.8250237069105657, 0.72178671657694604)
    (1, 0, 0, '|', 0.10859232785170762, 0.3502220587865601, 0.59935493253624916)
    (0, 1, 0, '|', 0.093221172597186791, 0.33865671798634139, 0.37002654400845286)
    (0, 0, 1, '|', 0.0053518944960174271, 0.69127310921209273, 0.71149776217085692)
    (1, 0, 0, '|', 0.12915104744302489, 0.32966061064682478, 0.51663527765789008)
    (0, 1, 0, '|', 0.10100745550801693, 0.35884824300022727, 0.31631345766364255)
    (0, 0, 1, '|', 0.0044432398721447148, 0.78596889819624782, 0.70532939327139632)
    (1, 0, 0, '|', 0.0984804006627093, 0.37755631528496364, 0.52607311417477731)
    (0, 1, 0, '|', 0.072629650377706897, 0.40169045513498075, 0.32681798422944874)
    (0, 0, 1, '|', 0.0016729927139831251, 0.90599749814172803, 0.72129192594319169)
    (1, 0, 0, '|', 0.1264001077785506, 0.31986415382266215, 0.49186275719170636)
    (0, 1, 0, '|', 0.12256223556345525, 0.2943901368902816, 0.31904629818607316)
    (0, 0, 1, '|', 0.0017614391546916565, 0.8665831694664945, 0.79210891237769221)
    (1, 0, 0, '|', 0.093128708219904932, 0.38867232221289816, 0.5876239941018977)
    (0, 1, 0, '|', 0.18791131752554557, 0.25215722767904342, 0.27594835574811571)
    (0, 0, 1, '|', 0.004304754544275197, 0.81915210087197488, 0.78677127062093521)
    (1, 0, 0, '|', 0.13843638799782451, 0.32227953757479488, 0.52774122772449372)
    (0, 1, 0, '|', 0.017665623425627452, 0.51701134216938027, 0.54436176619596643)
    (0, 0, 1, '|', 0.0033676468358334053, 0.82566254752291113, 0.70972931029167441)
    (1, 0, 0, '|', 0.14025155436927955, 0.34950911764151948, 0.52215354366539946)
    (0, 1, 0, '|', 0.047038198165000253, 0.3574192058986132, 0.41014545452606671)
    (0, 0, 1, '|', 0.0028054833608984779, 0.77557716458244508, 0.72658734716225426)
    (1, 0, 0, '|', 0.33121222372926518, 0.21994798232669585, 0.34063106714827235)
    (0, 1, 0, '|', 0.10549442830288547, 0.33046689930723189, 0.29349195730636779)
    (0, 0, 1, '|', 0.0017253459130696913, 0.86825552157333663, 0.79225310534425908)
    (1, 0, 0, '|', 0.064318327450169827, 0.42929070834839639, 0.65279314541441968)
    (0, 1, 0, '|', 0.023237261274899643, 0.45360908107204928, 0.49232703435551195)
    (0, 0, 1, '|', 0.0032076501800901345, 0.84118169066078929, 0.7669855134457868)
    (1, 0, 0, '|', 0.094124071079963026, 0.37211924443982564, 0.57388752847547164)
    (0, 1, 0, '|', 0.097695063439384833, 0.3222470371936651, 0.30137892629823948)
    (0, 0, 1, '|', 0.0045679508115824266, 0.80859476640951899, 0.70223657412062046)
    (1, 0, 0, '|', 0.084981139600587494, 0.43393961412096022, 0.60035315314808702)
    (0, 1, 0, '|', 0.091132583902708009, 0.26663296483678484, 0.22464895910452329)
    (0, 0, 1, '|', 0.0022829215517572317, 0.758981920814501, 0.71083812508828992)
    (1, 0, 0, '|', 0.080189611943183428, 0.40405016901086432, 0.55763378180769307)
    (0, 1, 0, '|', 0.055902401399773996, 0.36834633840923647, 0.37107992663561729)
    (0, 0, 1, '|', 0.0025128665596374504, 0.89553523400364909, 0.76025247923207573)
    (1, 0, 0, '|', 0.16288864486399371, 0.31705754573439265, 0.51525633921053415)
    (0, 1, 0, '|', 0.11434093929023478, 0.28649596488941709, 0.34079451621452383)
    (0, 0, 1, '|', 0.0056984747643403577, 0.84177838674611871, 0.68779316942106095)
    (1, 0, 0, '|', 0.11645623026641624, 0.3416023449281112, 0.56898603233163825)
    (0, 1, 0, '|', 0.052983337780431766, 0.35741260538832742, 0.42046972297015217)
    (0, 0, 1, '|', 0.0062281668950835249, 0.83773380750101756, 0.68685356812981113)
    (1, 0, 0, '|', 0.080093763967461692, 0.39379364579137566, 0.58208106916121483)
    (0, 1, 0, '|', 0.093349442379184119, 0.3429936494894355, 0.38112429274597404)
    (0, 0, 1, '|', 0.002206513166445124, 0.80142304022048616, 0.74550846426446338)
    (1, 0, 0, '|', 0.063951684090068975, 0.4202119170806306, 0.58710535269044672)
    (0, 1, 0, '|', 0.052102656929842957, 0.40736255784696257, 0.39806342936607914)
    (0, 0, 1, '|', 0.0039829988707050946, 0.85708677775692854, 0.69488873079681523)
    (1, 0, 0, '|', 0.38603485931207376, 0.19419960376613726, 0.33361524761310085)
    (0, 1, 0, '|', 0.077079439351597501, 0.33166725985538359, 0.40038701308248259)
    (0, 0, 1, '|', 0.0015157748153372854, 0.89859485637243175, 0.81658234040492739)
    (1, 0, 0, '|', 0.16628884571965122, 0.31949557538793805, 0.47542883590466994)
    (0, 1, 0, '|', 0.04657857314271751, 0.37559034309565753, 0.46292622005020539)
    (0, 0, 1, '|', 0.0033260041935596609, 0.84838616274225054, 0.61971350577100903)
    (1, 0, 0, '|', 0.24467553348073323, 0.26063144142778694, 0.43857234886958579)
    (0, 1, 0, '|', 0.041743691887670374, 0.4315999654127568, 0.34559195439170248)
    (0, 0, 1, '|', 0.0015850749783337184, 0.86361289331405766, 0.79783170706778506)
    (1, 0, 0, '|', 0.070346528647124193, 0.40838860386813186, 0.59889625326569318)
    (0, 1, 0, '|', 0.083283089040409958, 0.31460351421268368, 0.31599146692026514)
    (0, 0, 1, '|', 0.0019656515325727881, 0.84105836345629259, 0.7674334253842241)
    (1, 0, 0, '|', 0.068024425847332537, 0.41461751040419076, 0.60683097782848006)
    (0, 1, 0, '|', 0.034150681382154505, 0.41917824712882978, 0.44290993836517323)
    (0, 0, 1, '|', 0.00089185239036695366, 0.8977113096043805, 0.82086745723976828)
    (1, 0, 0, '|', 0.2847595465300119, 0.25367263560272085, 0.3945910478323062)
    (0, 1, 0, '|', 0.06601971500422614, 0.34788424910096727, 0.37244479117684354)
    (0, 0, 1, '|', 0.001793944634489315, 0.85983640364623681, 0.78877702278187933)
    (1, 0, 0, '|', 0.34370379597506018, 0.22178518716299447, 0.38033743942665815)
    (0, 1, 0, '|', 0.14465240132416768, 0.34615849021668466, 0.2401964231359838)
    (0, 0, 1, '|', 0.0020791916129353563, 0.83949685508049987, 0.75173555250881674)
    (1, 0, 0, '|', 0.21245496556694998, 0.29266133359023128, 0.46426991273951551)
    (0, 1, 0, '|', 0.044850372005049488, 0.29178433387553349, 0.45819900989191348)
    (0, 0, 1, '|', 0.0037129568657459913, 0.82150110870709658, 0.60357366779885824)
    (1, 0, 0, '|', 0.1059053508191551, 0.36759216215791962, 0.54261881844896831)
    (0, 1, 0, '|', 0.18743812027446557, 0.2761207337965022, 0.19625023847470635)
    (0, 0, 1, '|', 0.0036709026212593935, 0.80676134579906666, 0.72145436201824098)
    (1, 0, 0, '|', 0.090169626063593356, 0.38296636858544714, 0.49106933145116699)
    (0, 1, 0, '|', 0.039352205336477653, 0.41195004940812979, 0.43984033619738455)
    (0, 0, 1, '|', 0.0019308403001243974, 0.88746872423611411, 0.76737378451736349)
    (1, 0, 0, '|', 0.070162408738592308, 0.38470504841109765, 0.54050486760598493)
    (0, 1, 0, '|', 0.069463881005960071, 0.37128236267262615, 0.3323482150501702)
    (0, 0, 1, '|', 0.0048509748334481873, 0.83548747569856108, 0.68459274667203185)
    (1, 0, 0, '|', 0.23533512387625383, 0.25221437513918088, 0.45739136938250807)
    (0, 1, 0, '|', 0.048381965364662084, 0.4192790024701456, 0.42831551139968616)
    (0, 0, 1, '|', 0.0016006258322962923, 0.80091987289125188, 0.77691797437311338)
    (1, 0, 0, '|', 0.11530602276283272, 0.30753642908242068, 0.5249442595665319)
    (0, 1, 0, '|', 0.16776520851207824, 0.32576340537006876, 0.23463036368311929)
    (0, 0, 1, '|', 0.0068742567024439273, 0.80298064654846213, 0.6159697363424842)
    (1, 0, 0, '|', 0.11159073394018611, 0.37138937257898125, 0.52435972542477527)
    (0, 1, 0, '|', 0.028558063948047798, 0.48208137195115203, 0.23503069201621793)
    (0, 0, 1, '|', 0.00072399439291260625, 0.95320146997862354, 0.8139830339855606)
    (1, 0, 0, '|', 0.13918177900056175, 0.31839495496499992, 0.42662584482332827)
    (0, 1, 0, '|', 0.043276787307737208, 0.43502655012338382, 0.38826684530047123)
    (0, 0, 1, '|', 0.0022717547463620758, 0.89565724436895844, 0.77706514935770554)
    (1, 0, 0, '|', 0.097322873230555482, 0.36458611956303694, 0.5265879952123127)
    (0, 1, 0, '|', 0.084212374969207318, 0.27739950661111118, 0.36001711794052604)
    (0, 0, 1, '|', 0.0020078377210996918, 0.86694706595080384, 0.78410553447554487)
    (1, 0, 0, '|', 0.15357484453536382, 0.31702667166633891, 0.51202632760630795)
    (0, 1, 0, '|', 0.045182515822551082, 0.36494590745244632, 0.46550260064487453)
    (0, 0, 1, '|', 0.001950727375323429, 0.80963466122495098, 0.76350325933968122)
    (1, 0, 0, '|', 0.13155313783566389, 0.3458662433081891, 0.5171948200209201)
    (0, 1, 0, '|', 0.11058738373528375, 0.30716459867205231, 0.26365503297088339)
    (0, 0, 1, '|', 0.0082027531488634113, 0.76538885520393052, 0.572485034373868)
    (1, 0, 0, '|', 0.10027210266648977, 0.3758322219027142, 0.52592040063668399)
    (0, 1, 0, '|', 0.013707371816544706, 0.48926890855574545, 0.54158243313300714)
    (0, 0, 1, '|', 0.00074391353711931573, 0.93878118457662973, 0.80388534401550837)
    (1, 0, 0, '|', 0.15112717822134064, 0.30743197642127801, 0.48016472484177053)
    (0, 1, 0, '|', 0.082701218327975667, 0.36919430934708075, 0.32600419952701676)
    (0, 0, 1, '|', 0.0021400962387945937, 0.89672812012529801, 0.63301994781599547)
    (1, 0, 0, '|', 0.13948092302746362, 0.34793533967301221, 0.51457983511144367)
    (0, 1, 0, '|', 0.072611202485833429, 0.34595371617396481, 0.37454077650606926)
    (0, 0, 1, '|', 0.00143546413566676, 0.87339813059539773, 0.79544791423259931)
    (1, 0, 0, '|', 0.14118436423615033, 0.30464454006768343, 0.54777331689587661)
    (0, 1, 0, '|', 0.15727439081294259, 0.25051921658488008, 0.22248738720197608)
    (0, 0, 1, '|', 0.0032684970971889864, 0.79131045592632709, 0.64281963773655881)
    (1, 0, 0, '|', 0.13941049640495601, 0.34563177857785549, 0.49382655689732669)
    (0, 1, 0, '|', 0.071801320154897735, 0.31198071620530277, 0.26830613835715067)
    (0, 0, 1, '|', 0.0001983125128125463, 0.95869285273851734, 0.87953900804654328)
    (1, 0, 0, '|', 0.28806114241763242, 0.22851277562182579, 0.3971279890799565)
    (0, 1, 0, '|', 0.033703539597531418, 0.38960386554606169, 0.39244555518490898)
    (0, 0, 1, '|', 0.00037678204230100014, 0.92432412861760271, 0.59452392229502105)
    (1, 0, 0, '|', 0.23384881143976854, 0.24962470760644728, 0.38192032868116188)
    (0, 1, 0, '|', 0.047394180723382563, 0.3403030440205641, 0.39933455620675923)
    (0, 0, 1, '|', 0.00063520726198449556, 0.88757369665539043, 0.83086825874131665)
    (1, 0, 0, '|', 0.24702481160780954, 0.28854184625132873, 0.48254059157599288)
    (0, 1, 0, '|', 0.067611833393861551, 0.32634122528723541, 0.3603539410680292)
    (0, 0, 1, '|', 0.00099120581427150265, 0.81524464854110268, 0.7766019419916379)
    (1, 0, 0, '|', 0.13834511096615415, 0.34841501467476405, 0.51836071561008101)
    (0, 1, 0, '|', 0.016372642284765565, 0.41238357375056611, 0.43634513942251596)
    (0, 0, 1, '|', 0.0037691886604294066, 0.82681289289909587, 0.53737500968507868)
    (1, 0, 0, '|', 0.059341728494750418, 0.42848687746147546, 0.53681889199480559)
    (0, 1, 0, '|', 0.051878819551578907, 0.35218348877384287, 0.30540719698913371)
    (0, 0, 1, '|', 0.00034253831562335317, 0.93797579224885708, 0.83927835489768177)
    (1, 0, 0, '|', 0.20516157317956676, 0.25286042218720262, 0.3798919502414338)
    (0, 1, 0, '|', 0.073401710772639572, 0.32450860879159288, 0.2708588316538208)
    (0, 0, 1, '|', 0.00042964698225541665, 0.92740036497847211, 0.52900410009148058)
    (1, 0, 0, '|', 0.15325102694702047, 0.30998102323206955, 0.44893861394894907)
    (0, 1, 0, '|', 0.057363139775989093, 0.31757937567479078, 0.35525022552786251)
    (0, 0, 1, '|', 0.00033261259062855749, 0.91417961946778514, 0.86923241610683466)
    (1, 0, 0, '|', 0.15230379635998487, 0.31716841595258294, 0.51733437903024415)
    (0, 1, 0, '|', 0.045754204425046331, 0.32370385894653458, 0.31807177269741344)
    (0, 0, 1, '|', 0.00092714191547371065, 0.87547858215552576, 0.74459295337796105)
    (1, 0, 0, '|', 0.099402458891441253, 0.35060088480068058, 0.52304472604311458)
    (0, 1, 0, '|', 0.11559501743016121, 0.33803365499132965, 0.23396936463220522)
    (0, 0, 1, '|', 9.128983121034998e-05, 0.94092470493997893, 0.9103801765100471)
    (1, 0, 0, '|', 0.061805882207481622, 0.40278473896189776, 0.57864600370384423)
    (0, 1, 0, '|', 0.018304833065591448, 0.40782976513858132, 0.45141874966477674)
    (0, 0, 1, '|', 0.00035972550886225867, 0.91130550531850452, 0.80884012618909418)
    (1, 0, 0, '|', 0.03775990499540717, 0.47518327832905521, 0.57446964613964657)
    (0, 1, 0, '|', 0.070561601645551197, 0.32399072774541382, 0.2706714670972683)
    (0, 0, 1, '|', 0.00065266826751631164, 0.8816172275826325, 0.46371534339253573)
    (1, 0, 0, '|', 0.27125425221592775, 0.26802313595417215, 0.3872349300373964)
    (0, 1, 0, '|', 0.0020319383507893551, 0.63378194351825157, 0.6967276260361055)
    (0, 0, 1, '|', 0.0003631837840902233, 0.89652956933590577, 0.70506197805501525)
    (1, 0, 0, '|', 0.15890163144334002, 0.33002453149190297, 0.48227531912242205)
    (0, 1, 0, '|', 0.020310563792990906, 0.40195285493893024, 0.42126682676269861)
    (0, 0, 1, '|', 0.00059818289245195256, 0.86627910481133619, 0.74154411411286836)
    stop



```
def Sigmoid(x): # a is used to adjust the slope of the function 
    return (1/(1+np.exp(-x)))

def Pool(I,W):
    PoolImg=np.zeros((len(I)/len(W),len(I)/len(W))) # W must fit an integer times into I.
    for i in range(0,len(PoolImg)):
        for j in range(0,len(PoolImg)):
            SelAr=I[i*len(W):(i+1)*len(W),j*len(W):(j+1)*len(W)]
            PoolImg[i,j]=np.inner(SelAr.flatten(),W.flatten()) # Now this is just an inner product since we have vectors
    return PoolImg

scale=1*10**-3 # at last output-layer
scale_WP=1*10**-4 #to reduce the end-values those can become very high!
WC11=np.array([
    [1,1,1,1,1],
    [1,2,2,2,1],
    [1,2,4,2,1],
    [1,2,2,2,1],
    [1,1,1,1,1],    
    ])*(1/25)

WC12=np.array([
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    [0,0,0.5,1,1],
    ])*(1/25)

WC13=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [0.5,0.5,0.5,0.5,0.5],
    [0,0,0,0,0],
    [0,0,0,0,0]
    ])*(1/25)

WC14=np.array([
    [1,1,1,1,1],
    [1,1,1,0.5],
    [0,0,0.5,0,0
    [0,0.5,0,0.5,1],
    [0.5,0,0,0,0.5]
    ])*(1/25)

WC15=np.array([
    [0.5,0,0,0,0],
    [1,0.5,0,0,0],
    [1,1,0.5,0,0],
    [1,1,1,0.5,0],
    [1,1,1,1,0.5]
    ])*(1/25)

WC16=np.array([
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1],
    [1,0,1,0,1]
    ])*(1/25)

# averaging kernel in L2
WP=np.array([ 
    [1,1],
    [1,1],
    ])

WC31=np.array([ 
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    [1,1,1,1],
    ])*(1/25)

WC32=np.array([ 
    [1,0,0,0],
    [1,1,0,0],
    [1,1,1,0],
    [1,1,1,1],
    ])*(1/25)



WP4=np.array([
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    [1,1,1,1,1],
    ])*scale_WP

# we initialize random weight to prevent oscillations while training
W1_1=np.random.rand(1,16)
W1_2=np.random.rand(1,16)
W1_3=np.random.rand(1,16)
W1_4=np.random.rand(1,16)
W1_5=np.random.rand(1,16)
W1_6=np.random.rand(1,16)
W1_7=np.random.rand(1,16)
W1_8=np.random.rand(1,16)

W2_1=np.random.rand(1,16)
W2_2=np.random.rand(1,16)
W2_3=np.random.rand(1,16)
W2_4=np.random.rand(1,16)
W2_5=np.random.rand(1,16)
W2_6=np.random.rand(1,16)
W2_7=np.random.rand(1,16)
W2_8=np.random.rand(1,16)

W3_1=np.random.rand(1,16)
W3_2=np.random.rand(1,16)
W3_3=np.random.rand(1,16)
W3_4=np.random.rand(1,16)
W3_5=np.random.rand(1,16)
W3_6=np.random.rand(1,16)
W3_7=np.random.rand(1,16)
W3_8=np.random.rand(1,16)

b1=0
b2=0
b3=0




data_list=([[dataPatchedF1,dataPatchedF2,dataPatchedF3,dataPatchedF4,dataPatchedF5],
           [dataPatchedC1,dataPatchedC2,dataPatchedC3,dataPatchedC4,dataPatchedC5],
           [dataPatchedW1,dataPatchedW2,dataPatchedW3,dataPatchedW4,dataPatchedW5]
            ])

class_list=[[1,0,0],
            [0,1,0],
            [0,0,1]
            ]

i=0
j=0
d1=0
d2=0

for u in range(0,4000):
    if(d1==3): 
        d1=0 
        d2=d2+1 
    if(d2==5): 
        d2=0
        if(j==24 and i==9):
            print('stop')
            break # Searchspace of 2000x2000 is achieved and the loop will stop
        if(i==9): # i must increase to cover all patches over y-direction 
            j=j+1 # selects new patch in x direction
            i=0
        i=i+1
    inputPatch=data_list[d1][d2][i,j,:,:]  
    t1,t2,t3=class_list[d1]
#     print(t1,t2)
    d1=d1+1
        
    #L1
    ConvImg1=sig.convolve(inputPatch,WC11,mode='valid')
    ConvImg2=sig.convolve(inputPatch,WC12,mode='valid')
    ConvImg3=sig.convolve(inputPatch,WC13,mode='valid')
    ConvImg4=sig.convolve(inputPatch,WC14,mode='valid')
    ConvImg5=sig.convolve(inputPatch,WC15,mode='valid')
    ConvImg6=sig.convolve(inputPatch,WC16,mode='valid')


    #L2
    PoolImg1=Pool(ConvImg1,WP) # reducing dimensionality by 2.
    PoolImg2=Pool(ConvImg2,WP)
    PoolImg3=Pool(ConvImg3,WP)
    PoolImg4=Pool(ConvImg4,WP)
    PoolImg5=Pool(ConvImg5,WP)
    PoolImg6=Pool(ConvImg6,WP)
    
    #L3
    ConvImg1_31=sig.convolve(PoolImg1,WC31,mode='valid') 
    ConvImg1_32=sig.convolve(PoolImg1,WC32,mode='valid') 
    ConvImg2_31=sig.convolve(PoolImg2,WC31,mode='valid')  
    ConvImg2_32=sig.convolve(PoolImg2,WC32,mode='valid')
    ConvImg3_31=sig.convolve(PoolImg3,WC31,mode='valid')
    ConvImg3_32=sig.convolve(PoolImg3,WC32,mode='valid')
    ConvImg4_31=sig.convolve(PoolImg4,WC31,mode='valid')
    ConvImg4_32=sig.convolve(PoolImg4,WC32,mode='valid')
    ConvImg5_31=sig.convolve(PoolImg5,WC31,mode='valid')
    ConvImg5_32=sig.convolve(PoolImg5,WC32,mode='valid')
    ConvImg6_31=sig.convolve(PoolImg6,WC31,mode='valid')
    ConvImg6_32=sig.convolve(PoolImg6,WC32,mode='valid')

    #L4
    # reducing dimensionality by 2.
    PoolImg1_41=Pool(ConvImg1_31,WP4)*scale
    PoolImg1_42=Pool(ConvImg1_32,WP4)*scale 
    PoolImg2_41=Pool(ConvImg2_31,WP4)*scale
    PoolImg2_42=Pool(ConvImg2_32,WP4)*scale
    PoolImg3_41=Pool(ConvImg3_31,WP4)*scale
    PoolImg3_42=Pool(ConvImg3_32,WP4)*scale 
    PoolImg4_41=Pool(ConvImg4_31,WP4)*scale
    PoolImg4_42=Pool(ConvImg4_32,WP4)*scale 
    PoolImg5_41=Pool(ConvImg5_31,WP4)*scale
    PoolImg5_42=Pool(ConvImg5_32,WP4)*scale
    PoolImg6_41=Pool(ConvImg6_31,WP4)*scale
    PoolImg6_42=Pool(ConvImg6_32,WP4)*scale 
    

#     plt.figure()
#     plt.imshow(PoolImg6_1)
#     plt.xlabel(classType)
    
    n1=2*10**-1
    n2=1*10**0
    
    # Now we have the outputs
    x1=(
        np.inner(PoolImg1_1.flatten(),W1_1)+ 
        np.inner(PoolImg2_1.flatten(),W1_2)+
        np.inner(PoolImg3_1.flatten(),W1_3)+
        np.inner(PoolImg4_1.flatten(),W1_4)+
        np.inner(PoolImg5_1.flatten(),W1_5)+
        np.inner(PoolImg6_1.flatten(),W1_6)
        )
    x2=(
        np.inner(PoolImg1_1.flatten(),W2_1)+ 
        np.inner(PoolImg2_1.flatten(),W2_2)+
        np.inner(PoolImg3_1.flatten(),W2_3)+
        np.inner(PoolImg4_1.flatten(),W2_4)+
        np.inner(PoolImg5_1.flatten(),W2_5)+
        np.inner(PoolImg6_1.flatten(),W2_6)
        )
    x3=(
        np.inner(PoolImg1_1.flatten(),W3_1)+ 
        np.inner(PoolImg2_1.flatten(),W3_2)+
        np.inner(PoolImg3_1.flatten(),W3_3)+
        np.inner(PoolImg4_1.flatten(),W3_4)+
        np.inner(PoolImg5_1.flatten(),W3_5)+
        np.inner(PoolImg6_1.flatten(),W3_6)
        )
    x1=x1[0]
    x2=x2[0]
    x3=x3[0]
    f1=Sigmoid(x1-b1)
    f2=Sigmoid(x2-b2)
    f3=Sigmoid(x3-b3)
    
    W1_1=W1_1+ n1*(t1-f1)*PoolImg1_1.flatten()
    W1_2=W1_2+ n1*(t1-f1)*PoolImg2_1.flatten()
    W1_3=W1_3+ n1*(t1-f1)*PoolImg3_1.flatten()
    W1_4=W1_4+ n1*(t1-f1)*PoolImg4_1.flatten()
    W1_5=W1_5+ n1*(t1-f1)*PoolImg5_1.flatten()
    W1_6=W1_6+ n1*(t1-f1)*PoolImg6_1.flatten()
    
    W2_1=W2_1+ n1*(t2-f2)*PoolImg1_1.flatten()
    W2_2=W2_2+ n1*(t2-f2)*PoolImg2_1.flatten()
    W2_3=W2_3+ n1*(t2-f2)*PoolImg3_1.flatten()
    W2_4=W2_4+ n1*(t2-f2)*PoolImg4_1.flatten()
    W2_5=W2_5+ n1*(t2-f2)*PoolImg5_1.flatten()
    W2_6=W2_6+ n1*(t2-f2)*PoolImg6_1.flatten()
    
    W3_1=W3_1+ n1*(t3-f3)*PoolImg1_1.flatten()
    W3_2=W3_2+ n1*(t3-f3)*PoolImg2_1.flatten()
    W3_3=W3_3+ n1*(t3-f3)*PoolImg3_1.flatten()
    W3_4=W3_4+ n1*(t3-f3)*PoolImg4_1.flatten()
    W3_5=W3_5+ n1*(t3-f3)*PoolImg5_1.flatten()
    W3_6=W3_6+ n1*(t3-f3)*PoolImg6_1.flatten()
    
    
    b1=b1-n2*(t1-f1)
    b2=b2-n2*(t2-f2)
    b3=b2-n2*(t3-f3)

    if(u%10==0): print(t1,t2,t3,'|',f1,f2,f3)
# 
```


```
ConvImg5.shape
```




    (46L, 46L)




```

```

ea669e83bd6453818eaae596004412b965b507c4
ipynb
Jupyter Notebook
code/CNN/.ipynb_checkpoints/CNN-checkpoint.ipynb
ttphan/The-Cupcakes-3000
44b9e2b433ff945b1b6d57354dc1d22afbd0eddd
Unlicense MIT
code/CNN/.ipynb_checkpoints/CNN-checkpoint.ipynb
ttphan/The-Cupcakes-3000
44b9e2b433ff945b1b6d57354dc1d22afbd0eddd
Unlicense MIT
code/CNN/.ipynb_checkpoints/CNN-checkpoint.ipynb
ttphan/The-Cupcakes-3000
44b9e2b433ff945b1b6d57354dc1d22afbd0eddd
Unlicense MIT
2017-06-14T02:37:48.000Z
2017-06-14T02:37:48.000Z
Qwen/Qwen-72B
1. YES
2. YES