# Neural Nets t2


```python
%matplotlib widget
#%matplotlib inline
%load_ext autoreload
%autoreload 2
```


```python
# import Importing_Notebooks
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt
import dill
```

A network built of components which:
1. accept an ordered set of reals (we'll use `numpy.array`, and  call them vectors) at the input port and produce another at the output port - this is forward propagation. ${\displaystyle f\colon \mathbf {R} ^{n}\to \mathbf {R} ^{m}}$
1. accept an ordered set of reals at the output port, representing the gradient of the loss function at the output, and produce the gradient of the loss function at the input port - this is back propagation, aka backprop. ${\displaystyle b\colon \mathbf {R} ^{m}\to \mathbf {R} ^{n}}$
1. from the gradient of the loss function at the output, calculate the partial of the loss function w.r.t the internal parameters ${\displaystyle \frac{\partial E}{\partial w} }$
1. accept a scalar $\alpha$ to control the adjustment of internal parameters. _Or is this effected by scaling the loss gradient before passing??_
1. update internal parameters ${\displaystyle w \leftarrow w - \alpha \frac{\partial E}{\partial w} }$



```python
class Layer:
    def __init__(self):
        pass
    
    def __call__(self, x):
        """Computes response to input"""
        raise NotImplementedError
        
    def backprop(self, output_delE):
        """Uses output error gradient to adjust internal parameters, and returns gradient of error at input"""
        raise NotImplementedError
```

A network built of a cascade of layers:


```python
class Network:
    def __init__(self):
        self.layers = []
        self.alpha = 0.1 #FIXME
        
    def extend(self, net):
        self.layers.append(net)
        
    def __call__(self, input):
        v = input
        for net in self.layers:
            v = net(v)
        return v
    
    def learn(self, facts):
        for (x, expected) in facts:
            y = self(x)
            e = y - expected
            loss = e.dot(e)/2.0
            agrad = e * self.alpha
            for net in reversed(self.layers):
                agrad = net.backprop(agrad)
        return loss
```

## Useful Layers

### Identify


```python
class IdentityLayer(Layer):
    def __call__(self, x):
        return x
    
    def backprop(self, output_delE):
        return output_delE
```

### Affine
A layer that does an [affine transformation](https://mathworld.wolfram.com/AffineTransformation.html) aka affinity, which is the classic fully-connected layer with output offsets.

$$ \mathbf{M} \mathbf{x} + \mathbf{b} = \mathbf{y} $$
where
$$
\mathbf{x} = \sum_{j=1}^{n} x_j \mathbf{\hat{x}}_j \\
\mathbf{b} = \sum_{i=1}^{m} b_i \mathbf{\hat{y}}_i \\
\mathbf{y} = \sum_{i=1}^{m} y_i \mathbf{\hat{y}}_i
$$
and $\mathbf{M}$ can be written
$$
\begin{bmatrix}
    m_{1,1} & \dots & m_{1,n} \\
    \vdots & \ddots & \vdots \\
    m_{m,1} & \dots & m_{m,n}
\end{bmatrix} \\
$$

#### Error gradient back-propagation
$$ 
\begin{align}
 \frac{\partial loss}{\partial\mathbf{x}}
  = \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{x}}
  = \mathbf{M}\frac{\partial loss}{\partial\mathbf{y}}
\end{align}
$$
_SOLVE: Left-multiply or right-multiply?_

#### Parameter adjustment
$$
 \frac{\partial loss}{\partial\mathbf{M}}
 = \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{M}}
 = \frac{\partial loss}{\partial\mathbf{y}} \mathbf{x} \\
 \frac{\partial loss}{\partial\mathbf{b}}
 = \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{b}}
 = \frac{\partial loss}{\partial\mathbf{y}}
$$


```python
class AffinityLayer(Layer):
    """An affine transformation, which is the classic fully-connected layer with offsets"""
    def __init__(self, n, m):
        self.M = np.empty((m, n))
        self.b = np.empty(m)
        self.randomize()
        
    def randomize(self):
        self.M[:] = np.random.randn(*self.M.shape)
        self.b[:] = np.random.randn(*self.b.shape)
        
    def __call__(self, x):
        self.input = x
        self.output = self.M @ x + self.b
        return self.output
    
    def backprop(self, output_delE):
        input_delE = self.M @ output_delE
        self.M -= np.einsum('i,j', output_delE, self.input) # use np.outer?
        self.b -= output_delE
        return input_delE
```

### Map
Maps a scalar function on the inputs, for e.g. activation layers.


```python
class MapLayer(Layer):
    """Map a scalar function on the input taken element-wise"""
    def __init__(self, fun, dfundx):
        self.vfun = np.vectorize(fun)
        self.vdfundx = np.vectorize(dfundx)

        
    def __call__(self, x):
        self.input = x
        return self.vfun(x)
    
    def backprop(self, output_delE):
        input_delE = self.vdfundx(self.input) * output_delE
        return input_delE
```

___

## Tests

### One identity layer
See if the wheels turn:


```python
net = Network()
net.extend(IdentityLayer())
all(net(np.arange(3)) == np.arange(3))
```




    True



It does not learn, as expected:


```python
facts = [(np.arange(2*n, 2*n+2), np.arange(2*n+1, 2*n-1, -1)) for n in range(3)]
net.learn(facts)
```




    1.0




```python
net(np.arange(2,4))
```




    array([2, 3])



### One map layer


```python
net = Network()
net.extend(MapLayer(lambda x: x+1, lambda d: 1))
all(net(np.arange(3)) == np.arange(3)+1)
```




    True



It does not learn, as expected:


```python
net.learn(facts), all(net(np.arange(5)) == np.arange(5)+1), net(np.arange(2,4))
```




    (2.0, True, array([3, 4]))



### One affine layer


```python
net = Network()
net.extend(AffinityLayer(2,2))
```


```python
t = net.layers[0]
t.M, t.b
```




    (array([[-0.21533062,  0.28402263],
            [-0.28379346, -1.35249073]]),
     array([ 0.61141609, -1.15769416]))



Can it learn the identity transformation?


```python
# from nnbench import NNBench
from matplotlib.widgets import Slider, Button, RadioButtons
```


```python
class NNBench:
    def __init__(self, net, ideal=lambda x:x):
        self.net = net
        self.ideal = ideal
        self.gc_protect = []
        self.seed = 3
    
    def checkpoint_net(self):
        self.net_checkpoint = dill.dumps(self.net)
        
    def rollback_net(self):
        self.net = dill.loads(self.net_checkpoint)
        
    def training_data_gen(self, n):
        """Generate n instances of labelled training data"""
        np.random.seed(self.seed)
        for i in range(n):
            v = np.random.randn(2)
            yield (v, self.ideal(v))
            
    def learn(self, n=100):
        return [self.net.learn([fact]) for fact in self.training_data_gen(n)]
            
    def learning_potential(self, n=100, alpha=None):
        stash = dill.dumps(self.net)
        if alpha is not None: # only change the net's alpha if a value was passed to us
            self.net.alpha = alpha
        loss = self.net.learn(fact for fact in self.training_data_gen(n))
        self.net = dill.loads(stash)
        return -np.log(loss)
        
    def plot_learning(self, n):
        from matplotlib import pyplot as plt
        # self.losses = losses = [self.net.learn(fact for fact in self.training_data_gen(n))]
        losses = self.learn(n)
        plt.yscale('log')
        plt.plot(range(len(losses)),losses)
        plt.show(block=0)
        
    def knobs_plot_learning(self, n):
        # from matplotlib import pyplot as plt
        fig, ax = plt.subplots()
        plt.subplots_adjust(left=0.25, bottom=0.25)
        a0 = 5
        f0 = 3
        
        ###
        losses = [self.net.learn([fact]) for fact in self.training_data_gen(n)]
        l, = plt.plot(range(len(losses)), losses, lw=2)
        ax.margins(x=0)
        plt.yscale('log')

        axcolor = 'lightgoldenrodyellow'
        axfreq = plt.axes([0.25, 0.1, 0.65, 0.03], facecolor=axcolor)
        axamp = plt.axes([0.25, 0.15, 0.65, 0.03], facecolor=axcolor)

        sfreq = Slider(axfreq, '⍺', 0, 1, valinit=self.net.alpha)
        samp = Slider(axamp, 'Num', 1, 1000, valinit=100, valstep=1)
        
        filtfunc = [lambda x:x]
        
        
        big = max(losses)
        ax.set_title(f"maxloss:{big}")
    
        iax = plt.axes([0.025, 0.7, 0.15, 0.15])
        def make_iax_image():
            return np.concatenate([np.concatenate((l.M,np.array([l.b])),axis=0)
                                   for l in self.net.layers
                                  if hasattr(l, 'M')],axis=1)
        def update_iax(img=[iax.imshow(make_iax_image())]):
            img[0].remove()
            img[0] = iax.imshow(make_iax_image())

        
        
        def update(val,ax=ax,loc=[l]):
            n = int(samp.val)
            self.rollback_net()
            sfunc = lambda x: 2**(-1.005/(x+.005))
            self.net.alpha = sfunc(sfreq.val)
            #sfreq.set_label("2.4e"%(self.net.alpha,))
            losses = filtfunc[0]([self.net.learn([fact]) for fact in self.training_data_gen(n)])
            big = max(losses)
            ax.set_title(f"⍺={self.net.alpha},max loss:{big}")
            loc[0].remove()
            loc[0], = ax.plot(range(len(losses)), losses, lw=2,color='xkcd:blue')
            ax.set_xlim((0,len(losses)))
            ax.set_ylim((min(losses),big))
            update_iax()
            fig.canvas.draw_idle()

        sfreq.on_changed(update)
        samp.on_changed(update)

        resetax = plt.axes([0.8, 0.025, 0.1, 0.04])
        button = Button(resetax, 'Reset', color=axcolor, hovercolor='0.975')

    
        def reset(event):
            self.seed += 1
            update()
        button.on_clicked(reset)

        rax = plt.axes([0.025, 0.5, 0.15, 0.15], facecolor=axcolor)
        radio = RadioButtons(rax, ('raw', 'low pass', 'green'), active=0)

        
        def colorfunc(label):
            if label == "raw":
                filtfunc[0] = lambda x:x
            elif label == "low pass":
                filtfunc[0] = lambda x:ndimage.gaussian_filter(np.array(x),3)
            #l.set_color(label)
            #fig.canvas.draw_idle()
            update()
        radio.on_clicked(colorfunc)

        plt.show()
        #return 'gc protect:', update, reset, colorfunc,sfreq,samp, radio, button
        self.gc_protect.append((update, reset, colorfunc,sfreq,samp, radio, button))
```


```python
bench = NNBench(net)
bench.checkpoint_net()
bench.learning_potential()
```




    nan




```python
bench.plot_learning(100)
```


```python
bench.ideal = lambda v: np.array([v[1], v[0]])
bench.knobs_plot_learning(100)
```


    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …


### Learn thru a map layer

This layer doubles its input:


```python
net = Network()
net.extend(AffinityLayer(2,2))

def dtanh(x):
    v = np.tanh(x)
    return (1+v)*(1-v)

net.extend(MapLayer(lambda x:x*x/2.0, lambda d:d))
#net.extend(MapLayer(np.tanh, dtanh))
bench = NNBench(net)
bench.checkpoint_net()
```


```python
net.layers[0].M, net.layers[0].b
```




    (array([[-0.32158469,  0.15113037],
            [-0.01862772,  0.48352879]]),
     array([0.76896516, 1.36624284]))




```python
bench.ideal = lambda v: [(v[0]-v[1])**2,0]
#bench.ideal = lambda v: [(v[0]>0)*2-1,(v[0]>v[1])*2-1]
bench.learning_potential()
#bench.knobs_plot_learning(100)
```

    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:13: RuntimeWarning: overflow encountered in multiply
      del sys.path[0]
    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:19: RuntimeWarning: invalid value encountered in subtract
    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:20: RuntimeWarning: invalid value encountered in subtract





    nan




```python
bench.knobs_plot_learning(100)
```


    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …


    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:13: RuntimeWarning: overflow encountered in multiply
      del sys.path[0]
    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:19: RuntimeWarning: invalid value encountered in subtract
    /opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:20: RuntimeWarning: invalid value encountered in subtract


Look into it:


```python

```




    ['ALLOW_THREADS',
     'AxisError',
     'BUFSIZE',
     'CLIP',
     'ComplexWarning',
     'DataSource',
     'ERR_CALL',
     'ERR_DEFAULT',
     'ERR_IGNORE',
     'ERR_LOG',
     'ERR_PRINT',
     'ERR_RAISE',
     'ERR_WARN',
     'FLOATING_POINT_SUPPORT',
     'FPE_DIVIDEBYZERO',
     'FPE_INVALID',
     'FPE_OVERFLOW',
     'FPE_UNDERFLOW',
     'False_',
     'Inf',
     'Infinity',
     'LowLevelCallable',
     'MAXDIMS',
     'MAY_SHARE_BOUNDS',
     'MAY_SHARE_EXACT',
     'MachAr',
     'ModuleDeprecationWarning',
     'NAN',
     'NINF',
     'NZERO',
     'NaN',
     'PINF',
     'PZERO',
     'RAISE',
     'RankWarning',
     'SHIFT_DIVIDEBYZERO',
     'SHIFT_INVALID',
     'SHIFT_OVERFLOW',
     'SHIFT_UNDERFLOW',
     'ScalarType',
     'TooHardError',
     'True_',
     'UFUNC_BUFSIZE_DEFAULT',
     'UFUNC_PYVALS_NAME',
     'VisibleDeprecationWarning',
     'WRAP',
     '_UFUNC_API',
     '__SCIPY_SETUP__',
     '__all__',
     '__builtins__',
     '__cached__',
     '__config__',
     '__doc__',
     '__file__',
     '__loader__',
     '__name__',
     '__numpy_version__',
     '__package__',
     '__path__',
     '__spec__',
     '__version__',
     '_add_newdoc_ufunc',
     '_dep_fft',
     '_deprecated',
     '_distributor_init',
     '_fun',
     '_key',
     '_lib',
     '_msg',
     '_sci',
     'absolute',
     'absolute_import',
     'add',
     'add_docstring',
     'add_newdoc',
     'add_newdoc_ufunc',
     'alen',
     'all',
     'allclose',
     'alltrue',
     'amax',
     'amin',
     'angle',
     'any',
     'append',
     'apply_along_axis',
     'apply_over_axes',
     'arange',
     'arccos',
     'arccosh',
     'arcsin',
     'arcsinh',
     'arctan',
     'arctan2',
     'arctanh',
     'argmax',
     'argmin',
     'argpartition',
     'argsort',
     'argwhere',
     'around',
     'array',
     'array2string',
     'array_equal',
     'array_equiv',
     'array_repr',
     'array_split',
     'array_str',
     'asanyarray',
     'asarray',
     'asarray_chkfinite',
     'ascontiguousarray',
     'asfarray',
     'asfortranarray',
     'asmatrix',
     'asscalar',
     'atleast_1d',
     'atleast_2d',
     'atleast_3d',
     'average',
     'bartlett',
     'base_repr',
     'binary_repr',
     'bincount',
     'bitwise_and',
     'bitwise_not',
     'bitwise_or',
     'bitwise_xor',
     'blackman',
     'block',
     'bmat',
     'bool8',
     'bool_',
     'broadcast',
     'broadcast_arrays',
     'broadcast_to',
     'busday_count',
     'busday_offset',
     'busdaycalendar',
     'byte',
     'byte_bounds',
     'bytes0',
     'bytes_',
     'c_',
     'can_cast',
     'cast',
     'cbrt',
     'cdouble',
     'ceil',
     'cfloat',
     'char',
     'character',
     'chararray',
     'choose',
     'clip',
     'clongdouble',
     'clongfloat',
     'column_stack',
     'common_type',
     'compare_chararrays',
     'complex128',
     'complex256',
     'complex64',
     'complex_',
     'complexfloating',
     'compress',
     'concatenate',
     'conj',
     'conjugate',
     'convolve',
     'copy',
     'copysign',
     'copyto',
     'corrcoef',
     'correlate',
     'cos',
     'cosh',
     'count_nonzero',
     'cov',
     'cross',
     'csingle',
     'ctypeslib',
     'cumprod',
     'cumproduct',
     'cumsum',
     'datetime64',
     'datetime_as_string',
     'datetime_data',
     'deg2rad',
     'degrees',
     'delete',
     'deprecate',
     'deprecate_with_doc',
     'diag',
     'diag_indices',
     'diag_indices_from',
     'diagflat',
     'diagonal',
     'diff',
     'digitize',
     'disp',
     'divide',
     'division',
     'divmod',
     'dot',
     'double',
     'dsplit',
     'dstack',
     'dtype',
     'e',
     'ediff1d',
     'einsum',
     'einsum_path',
     'emath',
     'empty',
     'empty_like',
     'equal',
     'errstate',
     'euler_gamma',
     'exp',
     'exp2',
     'expand_dims',
     'expm1',
     'extract',
     'eye',
     'fabs',
     'fastCopyAndTranspose',
     'fft',
     'fft_msg',
     'fill_diagonal',
     'find_common_type',
     'finfo',
     'fix',
     'flatiter',
     'flatnonzero',
     'flexible',
     'flip',
     'fliplr',
     'flipud',
     'float128',
     'float16',
     'float32',
     'float64',
     'float_',
     'float_power',
     'floating',
     'floor',
     'floor_divide',
     'fmax',
     'fmin',
     'fmod',
     'format_float_positional',
     'format_float_scientific',
     'format_parser',
     'frexp',
     'frombuffer',
     'fromfile',
     'fromfunction',
     'fromiter',
     'frompyfunc',
     'fromregex',
     'fromstring',
     'full',
     'full_like',
     'fv',
     'gcd',
     'generic',
     'genfromtxt',
     'geomspace',
     'get_array_wrap',
     'get_include',
     'get_printoptions',
     'getbufsize',
     'geterr',
     'geterrcall',
     'geterrobj',
     'gradient',
     'greater',
     'greater_equal',
     'half',
     'hamming',
     'hanning',
     'heaviside',
     'histogram',
     'histogram2d',
     'histogram_bin_edges',
     'histogramdd',
     'hsplit',
     'hstack',
     'hypot',
     'i0',
     'identity',
     'ifft',
     'iinfo',
     'imag',
     'in1d',
     'index_exp',
     'indices',
     'inexact',
     'inf',
     'info',
     'infty',
     'inner',
     'insert',
     'int0',
     'int16',
     'int32',
     'int64',
     'int8',
     'int_',
     'intc',
     'integer',
     'interp',
     'intersect1d',
     'intp',
     'invert',
     'ipmt',
     'irr',
     'is_busday',
     'isclose',
     'iscomplex',
     'iscomplexobj',
     'isfinite',
     'isfortran',
     'isin',
     'isinf',
     'isnan',
     'isnat',
     'isneginf',
     'isposinf',
     'isreal',
     'isrealobj',
     'isscalar',
     'issctype',
     'issubclass_',
     'issubdtype',
     'issubsctype',
     'iterable',
     'ix_',
     'kaiser',
     'kron',
     'lcm',
     'ldexp',
     'left_shift',
     'less',
     'less_equal',
     'lexsort',
     'linspace',
     'little_endian',
     'load',
     'loads',
     'loadtxt',
     'log',
     'log10',
     'log1p',
     'log2',
     'logaddexp',
     'logaddexp2',
     'logical_and',
     'logical_not',
     'logical_or',
     'logical_xor',
     'logn',
     'logspace',
     'longcomplex',
     'longdouble',
     'longfloat',
     'longlong',
     'lookfor',
     'ma',
     'mafromtxt',
     'mask_indices',
     'mat',
     'math',
     'matmul',
     'matrix',
     'maximum',
     'maximum_sctype',
     'may_share_memory',
     'mean',
     'median',
     'memmap',
     'meshgrid',
     'mgrid',
     'min_scalar_type',
     'minimum',
     'mintypecode',
     'mirr',
     'mod',
     'modf',
     'moveaxis',
     'msort',
     'multiply',
     'nan',
     'nan_to_num',
     'nanargmax',
     'nanargmin',
     'nancumprod',
     'nancumsum',
     'nanmax',
     'nanmean',
     'nanmedian',
     'nanmin',
     'nanpercentile',
     'nanprod',
     'nanquantile',
     'nanstd',
     'nansum',
     'nanvar',
     'nbytes',
     'ndarray',
     'ndenumerate',
     'ndfromtxt',
     'ndim',
     'ndindex',
     'nditer',
     'negative',
     'nested_iters',
     'newaxis',
     'nextafter',
     'nonzero',
     'not_equal',
     'nper',
     'npv',
     'number',
     'obj2sctype',
     'object0',
     'object_',
     'ogrid',
     'ones',
     'ones_like',
     'outer',
     'packbits',
     'pad',
     'partition',
     'percentile',
     'pi',
     'piecewise',
     'place',
     'pmt',
     'poly',
     'poly1d',
     'polyadd',
     'polyder',
     'polydiv',
     'polyfit',
     'polyint',
     'polymul',
     'polysub',
     'polyval',
     'positive',
     'power',
     'ppmt',
     'print_function',
     'printoptions',
     'prod',
     'product',
     'promote_types',
     'ptp',
     'put',
     'put_along_axis',
     'putmask',
     'pv',
     'quantile',
     'r_',
     'rad2deg',
     'radians',
     'rand',
     'randn',
     'random',
     'rate',
     'ravel',
     'ravel_multi_index',
     'real',
     'real_if_close',
     'rec',
     'recarray',
     'recfromcsv',
     'recfromtxt',
     'reciprocal',
     'record',
     'remainder',
     'repeat',
     'require',
     'reshape',
     'resize',
     'result_type',
     'right_shift',
     'rint',
     'roll',
     'rollaxis',
     'roots',
     'rot90',
     'round_',
     'row_stack',
     's_',
     'safe_eval',
     'save',
     'savetxt',
     'savez',
     'savez_compressed',
     'sctype2char',
     'sctypeDict',
     'sctypeNA',
     'sctypes',
     'searchsorted',
     'select',
     'set_numeric_ops',
     'set_printoptions',
     'set_string_function',
     'setbufsize',
     'setdiff1d',
     'seterr',
     'seterrcall',
     'seterrobj',
     'setxor1d',
     'shape',
     'shares_memory',
     'short',
     'show_config',
     'show_numpy_config',
     'sign',
     'signbit',
     'signedinteger',
     'sin',
     'sinc',
     'single',
     'singlecomplex',
     'sinh',
     'size',
     'sometrue',
     'sort',
     'sort_complex',
     'source',
     'spacing',
     'split',
     'sqrt',
     'square',
     'squeeze',
     'stack',
     'std',
     'str0',
     'str_',
     'string_',
     'subtract',
     'sum',
     'swapaxes',
     'take',
     'take_along_axis',
     'tan',
     'tanh',
     'tensordot',
     'test',
     'tile',
     'timedelta64',
     'trace',
     'tracemalloc_domain',
     'transpose',
     'trapz',
     'tri',
     'tril',
     'tril_indices',
     'tril_indices_from',
     'trim_zeros',
     'triu',
     'triu_indices',
     'triu_indices_from',
     'true_divide',
     'trunc',
     'typeDict',
     'typeNA',
     'typecodes',
     'typename',
     'ubyte',
     'ufunc',
     'uint',
     'uint0',
     'uint16',
     'uint32',
     'uint64',
     'uint8',
     'uintc',
     'uintp',
     'ulonglong',
     'unicode_',
     'union1d',
     'unique',
     'unpackbits',
     'unravel_index',
     'unsignedinteger',
     'unwrap',
     'ushort',
     'vander',
     'var',
     'vdot',
     'vectorize',
     'version',
     'void',
     'void0',
     'vsplit',
     'vstack',
     'where',
     'who',
     'zeros',
     'zeros_like']




```python
net.layers[0].randomize()
net([3, 5])
```




    array([3.2617429 , 6.04950289])




```python
net.layers[0].M, net.layers[0].b
```




    (array([[-1.41364637,  1.16071821],
            [ 0.66063039,  0.42396354]]),
     array([ 0.06821949, -1.07695745]))



Make the affine layer the identity transform:


```python
net.layers[0].M = np.array([[1,0],[0,1]])
net.layers[0].b = np.array([0,0])
net([3,5])
```




    array([ 6, 10])




```python
bench.learning_potential()
```




    -30.508647928518727




```python
bench.knobs_plot_learning(100)
```


    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …



```python
net([7,11])
```




    array([14, 22])




```python
net.layers[0].M, net.layers[0].b
```




    (array([[1, 0],
            [0, 1]]),
     array([0, 0]))



What is learning doing to it?


```python
bench.learn(10)
```




    [4.092601131822876e-32,
     3.947771200004956e-30,
     4.125582682120508e-31,
     3.777904178910002e-30,
     1.5037661005775538e-30,
     1.633188592840376e-30,
     1.3620176566706532e-30,
     5.516826325697237e-31,
     7.703719777548943e-33,
     7.888609052210118e-31]




```python
net([7,11])
```




    array([14, 22])




```python
net.layers[0].M, net.layers[0].b
```




    (array([[1, 0],
            [0, 1]]),
     array([0, 0]))



If we take the map layer off again, how does it do?


```python
bench.rollback_net()
bench.net.layers = bench.net.layers[:1]
bench.checkpoint_net()
```


```python
bench.ideal = lambda v: v
bench.learning_potential()
#bench.knobs_plot_learning(100)
```




    17.53014188241033



It learns just fine, as expected. So we definitely have a problem.

### add a RELU


```python
bench.net.layers = bench.net.layers[:1]
leak = 0
bench.net.extend(MapLayer(lambda x: (x*(1+leak/2)+abs(x)*(1-leak/2))/2, lambda d: [leak,1][1 if d>0 else 0]))
bench.net.layers
```




    [<__main__.AffinityLayer at 0x7f0d293189d0>,
     <__main__.MapLayer at 0x7f0d292a86d0>]




```python
bench.net.layers[0].randomize()
bench.checkpoint_net()
bench.ideal = lambda v: np.array([1,1])
bench.knobs_plot_learning(100)
```


    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …



```python
%debug
```

    ERROR:root:No traceback has been produced, nothing to debug.


### XOR


```python
net = Network()
net.extend(AffinityLayer(2,2))
```


```python
t = net.layers[0]
t.M, t.b
```




    (array([[ 1.18691118,  0.10949354],
            [ 1.40113726, -0.73322905]]),
     array([-0.96443749, -0.11239461]))




```python

```

918e7bebfea9a42a73e5e9384a7f155e38b0042f
ipynb
Jupyter Notebook
nbs/OLD/nnt2.ipynb
pramasoul/aix
98333b875f6c6cda6dee86e6eab02c5ddc622543
MIT
nbs/OLD/nnt2.ipynb
pramasoul/aix
98333b875f6c6cda6dee86e6eab02c5ddc622543
MIT
2021-11-29T03:44:00.000Z
2021-12-19T05:34:04.000Z
nbs/OLD/nnt2.ipynb
pramasoul/aix
98333b875f6c6cda6dee86e6eab02c5ddc622543
MIT
Qwen/Qwen-72B
1. YES
2. YES