# Long Short-Term Memory (LSTM)

This page attempts to explain why LSTM was first proposed, and what are the core features together with some examples.

This is based on the paper [Hochreiter and Schmidhuber. 1997. Long Short-Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf)

## Constant-Error Carrousel

The core feature of an LSTM unit as first proposed is the constant-error carrousel (CEC) which solves the vanishing gradient problem with standard RNN.

A CEC is neural network unit which consists of a single neuron with self-loop with weight fixed to 1.0 to ensure constant error flow when doing backpropagation.

<div style="text-align:center">

<caption>Fig 1. Diagram of a single CEC unit</caption>
</div>

Now let's see an example of CEC at work. We will use CEC to do very simple task: **recognizing whether current character is inside a bracketed expression**, with the opening bracket considered to be inside, and the closing bracket considered to be outside, for simplicity. This is solvable only using network that can store memory, since to recognize whether a character is inside a bracketed expression, we need to have the knowledge that there is an opening bracket to the left of current character which does not have the corresponding closing bracket.

The input alphabets are coming from the set: $\{`a`, `b`, `(`, `)`\}$ with the following 2-dimensional embedding:

$$
\begin{eqnarray}
emb(`a`) &=& (1, 1) \nonumber\\
emb(`b`) &=& (-1, -1) \nonumber\\
emb(`(`) &=& (1, 0) \nonumber\\
emb(`)`) &=& (0, -1) \nonumber
\end{eqnarray}
$$

For this task, we define a very simple network with two input units, one CEC unit, and one output unit with sigmoid activation ($\sigma(x) = \frac{1}{1 + e^{-x}}$), as follows:

<div style="text-align:center">

<caption>Fig 2. Network used for the bracketed expression recognition</caption>
</div>

For this task, we define the loss function as the cross-entropy (CE) between the predicted and the true one:

$$
\begin{eqnarray}
\mathrm{CE}(x, y) = - (x\log(y) + (1-x)\log(1-y)) \nonumber\\
\mathrm{Loss}(\hat{o}_t, o_t) = \mathrm{CE}(\hat{o}_t, o_t) - \mathrm{CE}(\hat{o}_t, \hat{o}_t)
\end{eqnarray}
$$

with $\hat{o}_t$ and $o_t$ represent the target value (gold standard) and output value (network prediction), respectively, at time step $t$. The first term is the cross-entropy between the target value and the output value, and the second term is the entropy of the target value itself. Note that the second term is a constant, and serves just to make the minimum achievable loss to be 0 (perfect output).

More specifically, we have:

$$
\begin{equation}
o_t = \sigma(w_3*s_t)
\end{equation}
$$

where $s_t$ is the output of the CEC unit (a.k.a. the memory), which depends on the previous value of the memory $s_{t-1}$, and the input $x_{t,1}$ and $x_{t,2}$ (representing the first and second dimension of the input at time step $t$):

$$
\begin{equation}
s_t = \underbrace{w_s * s_{t-1}}_\text{previous value} + \underbrace{w_1 * x_{t,1} + w_2 * x_{t,2}}_\text{input}
\end{equation}
$$

where $w_s$ is the weight of the self-loop, which is 1.0. But for clarity of why this should be 1.0, the calculation later does not assume $w_s=1.0$.


```python
import math
from IPython.display import Markdown, display

def printmd(string):
    display(Markdown(string))

# Embedding
embedding = {}
embedding['a'] = (1.0, 1)
embedding['b'] = (-1, -1)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

# embedding['a'] = (-1, 0)
# embedding['b'] = (-0.5, 0)
# embedding['('] = (1, 1)
# embedding[')'] = (1, -1)

# Weights
w1=1.0
w2=1.0
w3=1.0
ws=1.0

memory_history = [0]
output_history = [0]

def sigmoid(x):
    return 1.0/(1+math.exp(-x))

def gold(seq):
    result = [0]
    bracket_count = 0
    for char in seq:
        if char == '(':
            bracket_count += 1
        if char == ')':
            bracket_count -= 1
        result.append(sigmoid(bracket_count))
    return result

def activate_memory(x1, x2):
    prev_memory = memory_history[-1]
    memory_history.append(ws*prev_memory + w1*x1 + w2*x2)
    return memory_history[-1]

def activate_output(h):
    output_history.append(sigmoid(w3*h))
    return output_history[-1]

def predict(seq):
    for char in seq:
        activate_output(activate_memory(*embedding[char]))
    result = output_history[:]
    return result

def reset():
    global memory_history, output_history
    memory_history = [0]
    output_history = [0]
    
def loss(gold_seq, pred_seq):
    result = 0.0
    per_position_loss = []
    for idx, (corr, pred) in enumerate(zip(gold_seq, pred_seq)):
        cur_loss  = -(corr*math.log(pred) + (1-corr)*math.log(1-pred))
        cur_loss -= -(corr*math.log(corr) + (1-corr)*math.log(1-corr))
        result += cur_loss
        per_position_loss.append(cur_loss)
    return result, per_position_loss


def print_list(lst):
    '''A convenience method to print a list of real numbers'''
    as_str = ['{:+.3f}'.format(num) for num in lst]
    print('[{}]'.format(', '.join(as_str)))
```


```python
# See typical values of sigmoid
for i in range(5):
    print('sigmoid({}) = {}'.format(i, sigmoid(i)))
```

    sigmoid(0) = 0.5
    sigmoid(1) = 0.7310585786300049
    sigmoid(2) = 0.8807970779778823
    sigmoid(3) = 0.9525741268224334
    sigmoid(4) = 0.9820137900379085


Now let's check the function calculating the target value. Basically we want it to output $\sigma(0)$ or $\sigma(1)$ when the output is outside or inside a bracketed expression, respectively.


```python
gold('a(a)a')[1:]  # The first element is dummy
```




    [0.5, 0.7310585786300049, 0.7310585786300049, 0.5, 0.5]



Which is $\sigma(0), \sigma(1), \sigma(1), \sigma(0), \sigma(0)$, which is what we expect. So far so good.
<hr>
Now let's see what our network outputs


```python
test_seq = 'ab(ab)ab'
reset()
w1 = 1.0
w2 = 1.0
w3 = 1.0
result = predict(test_seq)
correct = gold(test_seq)
print('Output: ', end='')
print_list(result[1:])
print('Target: ', end='')
print_list(correct[1:])
print('Loss  : {:.3f}'.format(loss(correct[1:], result[1:])[0]))
```

    Output: [+0.881, +0.500, +0.731, +0.953, +0.731, +0.881, +0.982, +0.881]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Loss  : 2.900


We see that the loss is still non-zero, and we see that some values are incorrectly predicted.

Next we will see the gradient calculation in progress, so that we can update the weight to reduce the loss.

### Calculating Gradients

To do the weight update, we need to calculate the partial derivative of the loss function with respect to the each weight. We have three weight parameters $w_1, w_2$, and $w_3$, so we need to compute three different partial derivatives.

For ease of notation, we denote $\mathrm{Loss}_t = \mathrm{Loss}(\hat{o}_t, o_t)$ as the loss at time step $t$ and $\mathrm{Loss} = \sum_t \mathrm{Loss}_t$ as the total loss over one sequence.

Remember that our objective is to reduce the total loss.

$$
\begin{eqnarray}
\frac{\partial\mathrm{Loss}}{\partial w_i} & = & \sum_t\frac{\partial \mathrm{Loss}_t}{\partial w_i} \\
& = & \sum_t\frac{\partial \mathrm{Loss}_t}{\partial o_t} \cdot \frac{\partial o_t}{\partial w_i} \qquad \text{(by chain rule)} \\
\end{eqnarray}
$$

for $w_3$, we can already compute the gradient here, which is:

$$
\require{cancel}
\begin{eqnarray}
\frac{\partial\mathrm{Loss}}{\partial w_3} & = & \sum_t\frac{\partial \mathrm{Loss}_t}{\partial o_t} \cdot \frac{\partial o_t}{\partial w_i} \\
& = & \sum_t\underbrace{\frac{o_t - \hat{o}_t}{\cancel{o_t(1-o_t)}}}_{=\frac{\partial \mathrm{Loss}_t}{\partial o_t}} \cdot \underbrace{s_t \cdot \cancel{o_t(1-o_t)}}_{=\frac{\partial o_t}{\partial w_i}} \\
& = & \sum_t(o_t-\hat{o}_t)s_t
\end{eqnarray}
$$

for $w_1$ and $w_2$, we have:

$$
\begin{eqnarray}
\frac{\partial\mathrm{Loss}}{\partial w_i} & = & \sum_t\frac{\partial \mathrm{Loss}_t}{\partial o_t} \cdot \frac{\partial o_t}{\partial w_i} \\
& = & \sum_t \frac{o_t - \hat{o}_t}{o_t(1-o_t)} \cdot \frac{\partial o_t}{\partial s_t} \cdot \frac{\partial s_t}{\partial w_i} \\
& = & \sum_t \frac{o_t - \hat{o}_t}{\cancel{o_t(1-o_t)}} \cdot w_3\cdot \cancel{o_t(1-o_t)} \cdot \frac{\partial s_t}{\partial w_i} \\
& = & \sum_t (o_t - \hat{o}_t)w_3 \cdot \frac{\partial s_t}{\partial w_i} \\
& = & \sum_t (o_t - \hat{o}_t)w_3 \cdot \left(w_s\cdot\frac{\partial s_{t-1}}{\partial w_i} + x_{t,i}\right) \\
& = & \sum_t (o_t - \hat{o}_t)w_3 \cdot \left({w_s}^2\cdot\frac{\partial s_{t-2}}{\partial w_i} + w_s\cdot x_{t-1,i} + x_{t,i}\right) \\
& & \ldots \\
& = & \sum_t (o_t - \hat{o}_t)w_3 \cdot \left(\sum_{t'\leq t} {w_s}^{t-t'}x_{t',i}\right) \\
\end{eqnarray}
$$

### Important Note on $w_s$!

We see that the gradient with respect to $w_1$ and $w_2$ contains the factor ${w_s}^{t-t'}$, where $t-t'$ can be as large as the input sequence length. So if $w_s \neq 1.0$, then either the gradient will vanish or blow up as the input sequence gets longer.


```python
def dLdw1(test_seq, gold_seq, pred_seq, state_seq, info):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw1 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        cur_dell *= sum(ws**(step-1)*embedding[test_seq[step-1]][0] for step in range(1, time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def dLdw2(test_seq, gold_seq, pred_seq, state_seq, info):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw2 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        cur_dell *= sum(ws**(step-1)*embedding[test_seq[step-1]][1] for step in range(1, time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def dLdw3(test_seq, gold_seq, pred_seq, state_seq, info):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw3 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * state_seq[time_step]
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result
```

## Experiment
Now we define an experiment which takes in initial values of all the weights, learning rate, and maximum number of iterations. We also want to experiment with fixing the weight $w_3$ (i.e., it is not learned).

The code below will print the total loss, the loss at each time step, the output, target, and memory at each time step, and also the gradient for each learned parameter at each time step.


```python
def experiment(test_seq, _w1=1.0, _w2=1.0, _w3=1.0, alpha=1e-1, max_iter=250, fixed_w3=True):
    global w1, w2, w3
    reset()
    w1 = _w1
    w2 = _w2
    w3 = _w3
    correct = gold(test_seq)
    print('w1={:+.3f}, w2={:+.3f}, w3={:+.3f}'.format(w1, w2, w3))
    for iter_num in range(max_iter):
        result = predict(test_seq)
        if iter_num < 15 or (iter_num % 50 == 49):
            printmd('<div style="font-weight:bold">Iteration {}</div>'.format(iter_num))
            print('Output: ', end='')
            print_list(result[1:])
            print('Target: ', end='')
            print_list(correct[1:])
            print('Memory: ', end='')
            print_list(memory_history[1:])
        total_loss, per_position_loss = loss(correct[1:], result[1:])
        info = ['', iter_num]
        info[0] = ('<div>Loss: <span style="font-weight:bold">{:.5f}</span>' +
                   '= <span style="font-family:monaco; font-size:12px">').format(total_loss)
        for idx, per_pos_loss in enumerate(per_position_loss):
            info[0] += '{}{:.3f}'.format(' + ' if idx > 0 else '', per_pos_loss)
        info[0] += '</span></div>'
        # printmd(loss_str)
        w1 -= alpha * dLdw1(test_seq, correct, result, memory_history, info)
        w2 -= alpha * dLdw2(test_seq, correct, result, memory_history, info)
        if not fixed_w3:
            w3 -= alpha * dLdw3(test_seq, correct, result, memory_history, info)
        if iter_num < 15 or (iter_num % 50 == 49):
            printmd(info[0])
            print('w1={:+.3f}, w2={:+.3f}, w3={:+.3f}'.format(w1, w2, w3))
            print()
        reset()
    return w1, w2, w3
```


```python
embedding['a'] = (1.0, 1)
embedding['b'] = (-1, -1)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

w1, w2, w3 = experiment('ab(ab)bb', _w1=1.0, _w2=1.0, max_iter=250, alpha=1e-1, fixed_w3=True)
printmd('## Test on longer sequence')
experiment('aabba(aba)bab', _w1=w1, _w2=w2, _w3=w3, alpha=1e-2, max_iter=100)
```

    w1=+1.000, w2=+1.000, w3=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.881, +0.500, +0.731, +0.953, +0.731, +0.881, +0.500, +0.119]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+2.000, +0.000, +1.000, +3.000, +1.000, +2.000, +0.000, -2.000]



<div>Loss: <span style="font-weight:bold">1.57455</span>= <span style="font-family:monaco; font-size:12px">0.434 + 0.000 + 0.000 + 0.273 + 0.000 + 0.434 + 0.000 + 0.434</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.381</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.443</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> = <span style="color:blue; text-decoration:underline">+1.585</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.381</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.222</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> = <span style="color:blue; text-decoration:underline">+1.364</span></div>


    w1=+0.841, w2=+0.864, w3=+1.000
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.846, +0.500, +0.699, +0.927, +0.699, +0.846, +0.500, +0.154]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.705, +0.000, +0.841, +2.547, +0.841, +1.705, +0.000, -1.705]



<div>Loss: <span style="font-weight:bold">1.16233</span>= <span style="font-family:monaco; font-size:12px">0.326 + 0.000 + 0.003 + 0.178 + 0.003 + 0.326 + 0.000 + 0.326</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.346</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.032</span> + <span style="color:blue">+0.393</span> + <span style="color:red">-0.032</span> + <span style="color:blue">+0.346</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.346</span> = <span style="color:blue; text-decoration:underline">+1.367</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.346</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.196</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.346</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.346</span> = <span style="color:blue; text-decoration:underline">+1.235</span></div>


    w1=+0.705, w2=+0.740, w3=+1.000
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.809, +0.500, +0.669, +0.896, +0.669, +0.809, +0.500, +0.191]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.445, +0.000, +0.705, +2.150, +0.705, +1.445, +0.000, -1.445]



<div>Loss: <span style="font-weight:bold">0.84706</span>= <span style="font-family:monaco; font-size:12px">0.241 + 0.000 + 0.009 + 0.106 + 0.009 + 0.241 + 0.000 + 0.241</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.309</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.062</span> + <span style="color:blue">+0.329</span> + <span style="color:red">-0.062</span> + <span style="color:blue">+0.309</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.309</span> = <span style="color:blue; text-decoration:underline">+1.133</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.309</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.165</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.309</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.309</span> = <span style="color:blue; text-decoration:underline">+1.092</span></div>


    w1=+0.591, w2=+0.631, w3=+1.000
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.772, +0.500, +0.644, +0.860, +0.644, +0.772, +0.500, +0.228]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.222, -0.000, +0.591, +1.814, +0.591, +1.222, -0.000, -1.222]



<div>Loss: <span style="font-weight:bold">0.61998</span>= <span style="font-family:monaco; font-size:12px">0.176 + 0.000 + 0.017 + 0.057 + 0.017 + 0.176 + 0.000 + 0.176</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.272</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.258</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.272</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.272</span> = <span style="color:blue; text-decoration:underline">+0.900</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.272</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.272</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.272</span> = <span style="color:blue; text-decoration:underline">+0.946</span></div>


    w1=+0.501, w2=+0.536, w3=+1.000
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.738, +0.500, +0.623, +0.823, +0.623, +0.738, +0.500, +0.262]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.038, +0.000, +0.501, +1.539, +0.501, +1.038, +0.000, -1.038]



<div>Loss: <span style="font-weight:bold">0.46541</span>= <span style="font-family:monaco; font-size:12px">0.129 + 0.000 + 0.026 + 0.026 + 0.026 + 0.129 + 0.000 + 0.129</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.238</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.108</span> + <span style="color:blue">+0.185</span> + <span style="color:red">-0.108</span> + <span style="color:blue">+0.238</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.238</span> = <span style="color:blue; text-decoration:underline">+0.683</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.238</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.092</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.238</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.238</span> = <span style="color:blue; text-decoration:underline">+0.808</span></div>


    w1=+0.433, w2=+0.456, w3=+1.000
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.709, +0.500, +0.607, +0.789, +0.607, +0.709, +0.500, +0.291]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.889, +0.000, +0.433, +1.322, +0.433, +0.889, -0.000, -0.889]



<div>Loss: <span style="font-weight:bold">0.36481</span>= <span style="font-family:monaco; font-size:12px">0.096 + 0.000 + 0.034 + 0.010 + 0.034 + 0.096 + 0.000 + 0.096</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.209</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.124</span> + <span style="color:blue">+0.117</span> + <span style="color:red">-0.124</span> + <span style="color:blue">+0.209</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.209</span> = <span style="color:blue; text-decoration:underline">+0.494</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.209</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.058</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.209</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.209</span> = <span style="color:blue; text-decoration:underline">+0.684</span></div>


    w1=+0.384, w2=+0.387, w3=+1.000
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.684, +0.500, +0.595, +0.760, +0.595, +0.684, +0.500, +0.316]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.771, +0.000, +0.384, +1.155, +0.384, +0.771, +0.000, -0.771]



<div>Loss: <span style="font-weight:bold">0.30095</span>= <span style="font-family:monaco; font-size:12px">0.073 + 0.000 + 0.041 + 0.002 + 0.041 + 0.073 + 0.000 + 0.073</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.184</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.136</span> + <span style="color:blue">+0.059</span> + <span style="color:red">-0.136</span> + <span style="color:blue">+0.184</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.184</span> = <span style="color:blue; text-decoration:underline">+0.337</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.184</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.184</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.184</span> = <span style="color:blue; text-decoration:underline">+0.580</span></div>


    w1=+0.350, w2=+0.329, w3=+1.000
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.664, +0.500, +0.587, +0.737, +0.587, +0.664, +0.500, +0.336]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.679, +0.000, +0.350, +1.029, +0.350, +0.679, -0.000, -0.679]



<div>Loss: <span style="font-weight:bold">0.26040</span>= <span style="font-family:monaco; font-size:12px">0.057 + 0.000 + 0.045 + 0.000 + 0.045 + 0.057 + 0.000 + 0.057</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.164</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.144</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.144</span> + <span style="color:blue">+0.164</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.164</span> = <span style="color:blue; text-decoration:underline">+0.213</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.164</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.164</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.164</span> = <span style="color:blue; text-decoration:underline">+0.496</span></div>


    w1=+0.329, w2=+0.279, w3=+1.000
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.648, +0.500, +0.581, +0.718, +0.581, +0.648, +0.500, +0.352]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.608, +0.000, +0.329, +0.937, +0.329, +0.608, +0.000, -0.608]



<div>Loss: <span style="font-weight:bold">0.23388</span>= <span style="font-family:monaco; font-size:12px">0.046 + 0.000 + 0.048 + 0.000 + 0.048 + 0.046 + 0.000 + 0.046</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.148</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.150</span> + <span style="color:red">-0.025</span> + <span style="color:red">-0.150</span> + <span style="color:blue">+0.148</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.148</span> = <span style="color:blue; text-decoration:underline">+0.118</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.148</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.013</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.148</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.148</span> = <span style="color:blue; text-decoration:underline">+0.430</span></div>


    w1=+0.317, w2=+0.236, w3=+1.000
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.635, +0.500, +0.579, +0.705, +0.579, +0.635, +0.500, +0.365]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.553, -0.000, +0.317, +0.870, +0.317, +0.553, -0.000, -0.553]



<div>Loss: <span style="font-weight:bold">0.21552</span>= <span style="font-family:monaco; font-size:12px">0.038 + 0.000 + 0.050 + 0.002 + 0.050 + 0.038 + 0.000 + 0.038</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.135</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.152</span> + <span style="color:red">-0.053</span> + <span style="color:red">-0.152</span> + <span style="color:blue">+0.135</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.135</span> = <span style="color:blue; text-decoration:underline">+0.047</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.135</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.026</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.135</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.135</span> = <span style="color:blue; text-decoration:underline">+0.378</span></div>


    w1=+0.312, w2=+0.199, w3=+1.000
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.625, +0.500, +0.577, +0.695, +0.577, +0.625, +0.500, +0.375]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.511, -0.000, +0.312, +0.823, +0.312, +0.511, -0.000, -0.511]



<div>Loss: <span style="font-weight:bold">0.20184</span>= <span style="font-family:monaco; font-size:12px">0.032 + 0.000 + 0.051 + 0.003 + 0.051 + 0.032 + 0.000 + 0.032</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.125</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.154</span> + <span style="color:red">-0.072</span> + <span style="color:red">-0.154</span> + <span style="color:blue">+0.125</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.125</span> = <span style="color:red; text-decoration:underline">-0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.125</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.036</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.125</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.125</span> = <span style="color:blue; text-decoration:underline">+0.339</span></div>


    w1=+0.313, w2=+0.165, w3=+1.000
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.617, +0.500, +0.578, +0.688, +0.578, +0.617, +0.500, +0.383]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.477, +0.000, +0.313, +0.790, +0.313, +0.477, -0.000, -0.477]



<div>Loss: <span style="font-weight:bold">0.19087</span>= <span style="font-family:monaco; font-size:12px">0.028 + 0.000 + 0.051 + 0.004 + 0.051 + 0.028 + 0.000 + 0.028</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.117</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.154</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.154</span> + <span style="color:blue">+0.117</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.117</span> = <span style="color:red; text-decoration:underline">-0.042</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.117</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.043</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.117</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.117</span> = <span style="color:blue; text-decoration:underline">+0.308</span></div>


    w1=+0.317, w2=+0.134, w3=+1.000
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.611, +0.500, +0.579, +0.683, +0.579, +0.611, +0.500, +0.389]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.451, -0.000, +0.317, +0.768, +0.317, +0.451, -0.000, -0.451]



<div>Loss: <span style="font-weight:bold">0.18151</span>= <span style="font-family:monaco; font-size:12px">0.025 + 0.000 + 0.050 + 0.005 + 0.050 + 0.025 + 0.000 + 0.025</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.111</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.152</span> + <span style="color:red">-0.096</span> + <span style="color:red">-0.152</span> + <span style="color:blue">+0.111</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.111</span> = <span style="color:red; text-decoration:underline">-0.069</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.111</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.048</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.111</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.111</span> = <span style="color:blue; text-decoration:underline">+0.284</span></div>


    w1=+0.324, w2=+0.105, w3=+1.000
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.606, +0.500, +0.580, +0.680, +0.580, +0.606, +0.500, +0.394]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.429, +0.000, +0.324, +0.753, +0.324, +0.429, +0.000, -0.429]



<div>Loss: <span style="font-weight:bold">0.17315</span>= <span style="font-family:monaco; font-size:12px">0.023 + 0.000 + 0.049 + 0.006 + 0.049 + 0.023 + 0.000 + 0.023</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.106</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.151</span> + <span style="color:red">-0.102</span> + <span style="color:red">-0.151</span> + <span style="color:blue">+0.106</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.106</span> = <span style="color:red; text-decoration:underline">-0.087</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.106</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.051</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.106</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.106</span> = <span style="color:blue; text-decoration:underline">+0.266</span></div>


    w1=+0.332, w2=+0.079, w3=+1.000
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.601, +0.500, +0.582, +0.678, +0.582, +0.601, +0.500, +0.399]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.411, -0.000, +0.332, +0.744, +0.332, +0.411, -0.000, -0.411]



<div>Loss: <span style="font-weight:bold">0.16547</span>= <span style="font-family:monaco; font-size:12px">0.021 + 0.000 + 0.048 + 0.007 + 0.048 + 0.021 + 0.000 + 0.021</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.149</span> + <span style="color:red">-0.106</span> + <span style="color:red">-0.149</span> + <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.101</span> = <span style="color:red; text-decoration:underline">-0.100</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.053</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.101</span> = <span style="color:blue; text-decoration:underline">+0.251</span></div>


    w1=+0.342, w2=+0.054, w3=+1.000
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.545, +0.500, +0.661, +0.700, +0.661, +0.545, +0.500, +0.455]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.181, +0.000, +0.667, +0.848, +0.667, +0.181, +0.000, -0.181]



<div>Loss: <span style="font-weight:bold">0.03748</span>= <span style="font-family:monaco; font-size:12px">0.004 + 0.000 + 0.011 + 0.002 + 0.011 + 0.004 + 0.000 + 0.004</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.045</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.070</span> + <span style="color:red">-0.062</span> + <span style="color:red">-0.070</span> + <span style="color:blue">+0.045</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.045</span> = <span style="color:red; text-decoration:underline">-0.067</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.045</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.031</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.045</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.045</span> = <span style="color:blue; text-decoration:underline">+0.105</span></div>


    w1=+0.674, w2=-0.496, w3=+1.000
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.516, +0.500, +0.706, +0.720, +0.706, +0.516, +0.500, +0.484]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.065, +0.000, +0.878, +0.942, +0.878, +0.065, +0.000, -0.065]



<div>Loss: <span style="font-weight:bold">0.00490</span>= <span style="font-family:monaco; font-size:12px">0.001 + 0.000 + 0.002 + 0.000 + 0.002 + 0.001 + 0.000 + 0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.016</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.025</span> + <span style="color:red">-0.023</span> + <span style="color:red">-0.025</span> + <span style="color:blue">+0.016</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.016</span> = <span style="color:red; text-decoration:underline">-0.024</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.016</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.012</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.016</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.016</span> = <span style="color:blue; text-decoration:underline">+0.037</span></div>


    w1=+0.880, w2=-0.817, w3=+1.000
    



<div style="font-weight:bold">Iteration 149</div>


    Output: [+0.506, +0.500, +0.722, +0.727, +0.722, +0.506, +0.500, +0.494]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.024, +0.000, +0.954, +0.978, +0.954, +0.024, +0.000, -0.024]



<div>Loss: <span style="font-weight:bold">0.00067</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.006</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.009</span> + <span style="color:red">-0.009</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.006</span> = <span style="color:red; text-decoration:underline">-0.009</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.006</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.004</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.006</span> = <span style="color:blue; text-decoration:underline">+0.014</span></div>


    w1=+0.955, w2=-0.932, w3=+1.000
    



<div style="font-weight:bold">Iteration 199</div>


    Output: [+0.502, +0.500, +0.728, +0.729, +0.728, +0.502, +0.500, +0.498]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.009, +0.000, +0.983, +0.992, +0.983, +0.009, +0.000, -0.009]



<div>Loss: <span style="font-weight:bold">0.00009</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div>


    w1=+0.983, w2=-0.975, w3=+1.000
    



<div style="font-weight:bold">Iteration 249</div>


    Output: [+0.501, +0.500, +0.730, +0.730, +0.730, +0.501, +0.500, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.003, +0.000, +0.994, +0.997, +0.994, +0.003, +0.000, -0.003]



<div>Loss: <span style="font-weight:bold">0.00001</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div>


    w1=+0.994, w2=-0.990, w3=+1.000
    



## Test on longer sequence


    w1=+0.994, w2=-0.990, w3=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.501, +0.502, +0.501, +0.500, +0.501, +0.730, +0.731, +0.730, +0.731, +0.502, +0.502, +0.502, +0.502]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.003, +0.007, +0.003, +0.000, +0.003, +0.997, +1.000, +0.997, +1.000, +0.010, +0.007, +0.010, +0.007]



<div>Loss: <span style="font-weight:bold">0.00005</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> = <span style="color:blue; text-decoration:underline">+0.025</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> = <span style="color:blue; text-decoration:underline">+0.026</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.501, +0.501, +0.501, +0.500, +0.501, +0.730, +0.731, +0.730, +0.731, +0.502, +0.501, +0.502, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.003, +0.006, +0.003, +0.000, +0.003, +0.996, +0.999, +0.996, +0.999, +0.008, +0.006, +0.008, +0.006]



<div>Loss: <span style="font-weight:bold">0.00003</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.003</span> = <span style="color:blue; text-decoration:underline">+0.019</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.003</span> = <span style="color:blue; text-decoration:underline">+0.020</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.501, +0.501, +0.501, +0.500, +0.501, +0.730, +0.731, +0.730, +0.731, +0.502, +0.501, +0.502, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.005, +0.002, +0.000, +0.002, +0.996, +0.998, +0.996, +0.998, +0.007, +0.005, +0.007, +0.005]



<div>Loss: <span style="font-weight:bold">0.00003</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.014</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.016</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.501, +0.501, +0.501, +0.500, +0.501, +0.730, +0.731, +0.730, +0.731, +0.502, +0.501, +0.502, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.004, +0.002, +0.000, +0.002, +0.995, +0.997, +0.995, +0.997, +0.006, +0.004, +0.006, +0.004]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.010</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.013</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.004, +0.002, +0.000, +0.002, +0.995, +0.997, +0.995, +0.997, +0.006, +0.004, +0.006, +0.004]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.007</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.011</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.003, +0.002, +0.000, +0.002, +0.995, +0.996, +0.995, +0.996, +0.005, +0.003, +0.005, +0.003]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.009</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.003, +0.002, +0.000, +0.002, +0.994, +0.996, +0.994, +0.996, +0.005, +0.003, +0.005, +0.003]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.007</span></div>


    w1=+0.993, w2=-0.991, w3=+1.000
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.003, +0.001, +0.000, +0.001, +0.994, +0.996, +0.994, +0.996, +0.004, +0.003, +0.004, +0.003]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.006</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.003, +0.001, +0.000, +0.001, +0.994, +0.996, +0.994, +0.996, +0.004, +0.003, +0.004, +0.003]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.003, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.004, +0.003, +0.004, +0.003]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.004, +0.002, +0.004, +0.002]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.004</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.004, +0.002, +0.004, +0.002]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.004</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.003, +0.002, +0.003, +0.002]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.003, +0.002, +0.003, +0.002]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.500, +0.501, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.501, +0.501, +0.501]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.003, +0.002, +0.003, +0.002]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div>


    w1=+0.993, w2=-0.992, w3=+1.000
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.500, +0.501, +0.500]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.994, +0.995, +0.994, +0.995, +0.003, +0.002, +0.003, +0.002]



<div>Loss: <span style="font-weight:bold">0.00001</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div>


    w1=+0.994, w2=-0.993, w3=+1.000
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.730, +0.730, +0.730, +0.730, +0.501, +0.500, +0.501, +0.500]
    Target: [+0.500, +0.500, +0.500, +0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500, +0.500]
    Memory: [+0.001, +0.002, +0.001, +0.000, +0.001, +0.995, +0.996, +0.995, +0.996, +0.002, +0.002, +0.002, +0.002]



<div>Loss: <span style="font-weight:bold">0.00001</span>= <span style="font-family:monaco; font-size:12px">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div>


    w1=+0.994, w2=-0.994, w3=+1.000
    





    (0.9943962680848304, -0.9935690516603158, 1.0)



# Let's Try Adding Input Gate

We saw in the experiment before that there is conflicting update (at one point of the sequence the gradient is positive, while at another point it is negative), which the original paper explains that it is caused by the weight into the memory cell needs to update the memory at one point (when we see brackets in this case) and retain information at another point (when we see any other characters).

Another core feature of LSTM that was designed to resolve this issue is that it adds gates: input gate and output gate, to control the flow of information through the memory cells.

In the following, we try adding an input gate, which the network should learn to activate (value = 1) only when it sees an opening bracket or closing bracket. So basically the input gate is telling the network which inputs are relevant and which are not.

**Note**: Below we have two versions for the input gate: linear with sigmoid, and bilinear with bias. The $w_4$ and $w_5$ have different interpretation depending on the input gate chosen. The bilinear gate was added because the input doesn't allow the linear gate to be useful.


```python
w4 = 1.0
w5 = 1.0
input_history = [0]
gate_history = [0]

def reset_gated():
    global memory_history, output_history, input_history, gate_history
    memory_history = [0]
    output_history = [0]
    input_history = [0]
    gate_history = [0]

def activate_input(x1, x2):
    result = (w1*x1+w2*x2)
    input_history.append(result)
    return result

def activate_gate(x1, x2, bilinear_gate=True):
    if bilinear_gate:
        result = w4 + w5*x1*x2  # Bilinear gate
    else:
        result = sigmoid(w4*x1+w5*x2)  # The true linear gate
    gate_history.append(result)
    return result

def dLdw1_gated(test_seq, gold_seq, pred_seq, state_seq, input_seq, gate_seq, info, bilinear_gate=True):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw1 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        cur_dell *= sum(embedding[test_seq[step-1]][0]*gate_seq[step] for step in range(1, time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def dLdw2_gated(test_seq, gold_seq, pred_seq, state_seq, input_seq, gate_seq, info, bilinear_gate=True):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw2 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        cur_dell *= sum(embedding[test_seq[step-1]][1]*gate_seq[step] for step in range(1, time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def dLdw4_gated(test_seq, gold_seq, pred_seq, state_seq, input_seq, gate_seq, info, bilinear_gate=True):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw4 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        if bilinear_gate:
            cur_dell *= sum(input_seq[step] for step in range(1, time_step+1))
        else:
            cur_dell *= sum(embedding[test_seq[step-1]][0]*gate_seq[step]*input_seq[step]*(1-gate_seq[step])
                            for step in range(1,time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                    'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def dLdw5_gated(test_seq, gold_seq, pred_seq, state_seq, input_seq, gate_seq, info, bilinear_gate=True):
    result = 0.0
    grad_str = '<div style="font-family:monaco; font-size:12px">dL/dw5 = '
    for time_step in range(1, len(gold_seq)):
        cur_dell = (pred_seq[time_step] - gold_seq[time_step]) * w3
        if bilinear_gate:
            cur_dell *= sum(embedding[test_seq[step-1]][0]*embedding[test_seq[step-1]][1]*input_seq[step]
                            for step in range(1, time_step+1))
        else:
            cur_dell *= sum(embedding[test_seq[step-1]][1]*gate_seq[step]*input_seq[step]*(1-gate_seq[step])
                            for step in range(1,time_step+1))
        if cur_dell < 0:
            color = 'red'
        else:
            color = 'blue'
        grad_str += '{}<span style="color:{}">{:+.3f}</span>'.format(' + ' if time_step > 1 else '', color, cur_dell)
        result += cur_dell
    grad_str += ' = <span style="color:{}; text-decoration:underline">{:+.3f}</span></div>'.format(
                    'red' if result < 0 else 'blue', result)
    # printmd(grad_str)
    info[0] += grad_str
    return result

def activate_memory_gated():
    memory_history.append(ws*memory_history[-1] + input_history[-1]*gate_history[-1])
    return memory_history[-1]

def predict_gated(seq):
    for char in seq:
        activate_input(*embedding[char])
        activate_gate(*embedding[char])
        activate_output(activate_memory_gated())
    result = output_history[:]
    return result

def experiment_gated(test_seq, _w1=1.0, _w2=1.0, _w3=1.0, _w4=1.0, _w5=1.0, alpha=1e-1, max_iter=750,
                     bilinear_gate=True, fixed_w3=True, fixed_w4=False, fixed_w5=False):
    global w1, w2, w3, w4, w5
    reset_gated()
    w1 = _w1
    w2 = _w2
    w3 = _w3
    w4 = _w4
    w5 = _w5
    correct = gold(test_seq)
    print('w1={:+.3f}, w2={:+.3f}, w3={:+.3f}, w4={:+.3f}, w5={:+.3f}'.format(w1, w2, w3, w4, w5))
    for iter_num in range(max_iter):
        result = predict_gated(test_seq)
        if iter_num < 15 or (iter_num % 50 == 49):
            printmd('<div style="font-weight:bold">Iteration {}</div>'.format(iter_num))
            print('Output: ', end='')
            print_list(result[1:])
            print('Target: ', end='')
            print_list(correct[1:])
            print('Memory: ', end='')
            print_list(memory_history[1:])
            print('Input : ', end='')
            print_list(input_history[1:])
            print('Gate  : ', end='')
            print_list(gate_history[1:])
        total_loss, per_position_loss = loss(correct[1:], result[1:])
        info = ['', iter_num]
        info[0] = ('<div>Loss: <span style="font-weight:bold">{:.5f}</span>' +
                   '= <span style="font-family:monaco">').format(total_loss)
        for idx, per_pos_loss in enumerate(per_position_loss):
            info[0] += '{}{:.3f}'.format(' + ' if idx > 0 else '', per_pos_loss)
        info[0] += '</span></div>'
        # printmd(loss_str)
        w1 -= alpha * dLdw1_gated(test_seq, correct, result, memory_history, input_history, gate_history,
                                  info, bilinear_gate)
        w2 -= alpha * dLdw2_gated(test_seq, correct, result, memory_history, input_history, gate_history,
                                  info, bilinear_gate)
        if not fixed_w3:
            w3 -= alpha * dLdw3(test_seq, correct, result, memory_history, info, bilinear_gate)
        if not fixed_w4:
            w4 -= alpha * dLdw4_gated(test_seq, correct, result, memory_history, input_history, gate_history,
                                      info, bilinear_gate)
        if not fixed_w5:
            w5 -= alpha * dLdw5_gated(test_seq, correct, result, memory_history, input_history, gate_history,
                                      info, bilinear_gate)
        if iter_num < 15 or (iter_num % 50 == 49):
            printmd(info[0])
            print('w1={:+.3f}, w2={:+.3f}, w3={:+.3f}, w4={:+.3f}, w5={:+.3f}'.format(w1, w2, w3, w4, w5))
            print()
        reset_gated()
    return w1, w2, w3, w4, w5
```


```python
embedding['a'] = (1.0, 1)
embedding['b'] = (-1, -1)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

experiment_gated('ab(ab)bb', _w1=1.0, _w2=1.0, _w4=1.0, _w5=1.0, alpha=1e-1, max_iter=250, fixed_w3=True)
```

    w1=+1.000, w2=+1.000, w3=+1.000, w4=+1.000, w5=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.982, +0.500, +0.731, +0.993, +0.731, +0.881, +0.119, +0.002]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+4.000, +0.000, +1.000, +5.000, +1.000, +2.000, -2.000, -6.000]
    Input : [+2.000, -2.000, +1.000, +2.000, -2.000, +1.000, -2.000, -2.000]
    Gate  : [+2.000, +2.000, +1.000, +2.000, +2.000, +1.000, +2.000, +2.000]



<div>Loss: <span style="font-weight:bold">5.27111</span>= <span style="font-family:monaco">1.325 + 0.000 + 0.000 + 0.769 + 0.000 + 0.434 + 0.434 + 2.309</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.964</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.787</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+1.493</span> = <span style="color:blue; text-decoration:underline">+4.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.964</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.524</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+0.381</span> + <span style="color:blue">+1.493</span> = <span style="color:blue; text-decoration:underline">+3.743</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.964</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.787</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.762</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.995</span> = <span style="color:blue; text-decoration:underline">+3.507</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.964</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.524</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.762</span> + <span style="color:blue">+1.990</span> = <span style="color:blue; text-decoration:underline">+4.240</span></div>


    w1=+0.600, w2=+0.626, w3=+1.000, w4=+0.649, w5=+0.576
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.818, +0.500, +0.596, +0.869, +0.596, +0.689, +0.331, +0.099]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.501, +0.000, +0.389, +1.890, +0.389, +0.795, -0.706, -2.207]
    Input : [+1.225, -1.225, +0.600, +1.225, -1.225, +0.626, -1.225, -1.225]
    Gate  : [+1.225, +1.225, +0.649, +1.225, +1.225, +0.649, +1.225, +1.225]



<div>Loss: <span style="font-weight:bold">1.05796</span>= <span style="font-family:monaco">0.259 + 0.000 + 0.040 + 0.067 + 0.040 + 0.077 + 0.061 + 0.515</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.088</span> + <span style="color:blue">+0.258</span> + <span style="color:red">-0.088</span> + <span style="color:blue">+0.123</span> + <span style="color:blue">+0.098</span> + <span style="color:blue">+0.722</span> = <span style="color:blue; text-decoration:underline">+1.415</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.169</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.123</span> + <span style="color:blue">+0.098</span> + <span style="color:blue">+0.722</span> = <span style="color:blue; text-decoration:underline">+1.500</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.081</span> + <span style="color:blue">+0.251</span> + <span style="color:red">-0.081</span> + <span style="color:blue">+0.232</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.491</span> = <span style="color:blue; text-decoration:underline">+1.202</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.169</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.208</span> + <span style="color:blue">+0.982</span> = <span style="color:blue; text-decoration:underline">+1.748</span></div>


    w1=+0.458, w2=+0.476, w3=+1.000, w4=+0.529, w5=+0.401
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.704, +0.500, +0.560, +0.752, +0.560, +0.621, +0.407, +0.224]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.869, +0.000, +0.242, +1.111, +0.242, +0.494, -0.375, -1.243]
    Input : [+0.934, -0.934, +0.458, +0.934, -0.934, +0.476, -0.934, -0.934]
    Gate  : [+0.930, +0.930, +0.529, +0.930, +0.930, +0.529, +0.930, +0.930]



<div>Loss: <span style="font-weight:bold">0.44676</span>= <span style="font-family:monaco">0.091 + 0.000 + 0.062 + 0.001 + 0.062 + 0.030 + 0.017 + 0.182</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.190</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.090</span> + <span style="color:blue">+0.031</span> + <span style="color:red">-0.090</span> + <span style="color:blue">+0.064</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.368</span> = <span style="color:blue; text-decoration:underline">+0.509</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.190</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.020</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.064</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.368</span> = <span style="color:blue; text-decoration:underline">+0.679</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.191</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.078</span> + <span style="color:blue">+0.030</span> + <span style="color:red">-0.078</span> + <span style="color:blue">+0.113</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.258</span> = <span style="color:blue; text-decoration:underline">+0.435</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.191</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.020</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.086</span> + <span style="color:blue">+0.516</span> = <span style="color:blue; text-decoration:underline">+0.813</span></div>


    w1=+0.407, w2=+0.408, w3=+1.000, w4=+0.486, w5=+0.320
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.658, +0.500, +0.549, +0.701, +0.549, +0.598, +0.435, +0.286]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.656, +0.000, +0.198, +0.854, +0.198, +0.396, -0.261, -0.917]
    Input : [+0.815, -0.815, +0.407, +0.815, -0.815, +0.408, -0.815, -0.815]
    Gate  : [+0.806, +0.806, +0.486, +0.806, +0.806, +0.486, +0.806, +0.806]



<div>Loss: <span style="font-weight:bold">0.32491</span>= <span style="font-family:monaco">0.053 + 0.000 + 0.070 + 0.002 + 0.070 + 0.019 + 0.008 + 0.102</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.128</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.038</span> + <span style="color:red">-0.088</span> + <span style="color:blue">+0.047</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.241</span> = <span style="color:blue; text-decoration:underline">+0.222</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.128</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.024</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.047</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.241</span> = <span style="color:blue; text-decoration:underline">+0.413</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.129</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.074</span> + <span style="color:red">-0.036</span> + <span style="color:red">-0.074</span> + <span style="color:blue">+0.080</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.175</span> = <span style="color:blue; text-decoration:underline">+0.199</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.129</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.024</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.053</span> + <span style="color:blue">+0.350</span> = <span style="color:blue; text-decoration:underline">+0.507</span></div>


    w1=+0.385, w2=+0.366, w3=+1.000, w4=+0.466, w5=+0.269
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.635, +0.500, +0.545, +0.675, +0.545, +0.587, +0.450, +0.320]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.552, +0.000, +0.179, +0.731, +0.179, +0.350, -0.202, -0.754]
    Input : [+0.751, -0.751, +0.385, +0.751, -0.751, +0.366, -0.751, -0.751]
    Gate  : [+0.735, +0.735, +0.466, +0.735, +0.735, +0.466, +0.735, +0.735]



<div>Loss: <span style="font-weight:bold">0.28193</span>= <span style="font-family:monaco">0.038 + 0.000 + 0.074 + 0.007 + 0.074 + 0.015 + 0.005 + 0.070</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.099</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.067</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.040</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.181</span> = <span style="color:blue; text-decoration:underline">+0.093</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.099</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.041</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.040</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.181</span> = <span style="color:blue; text-decoration:underline">+0.293</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.072</span> + <span style="color:red">-0.064</span> + <span style="color:red">-0.072</span> + <span style="color:blue">+0.065</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.135</span> = <span style="color:blue; text-decoration:underline">+0.095</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.101</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.042</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.038</span> + <span style="color:blue">+0.271</span> = <span style="color:blue; text-decoration:underline">+0.368</span></div>


    w1=+0.376, w2=+0.337, w3=+1.000, w4=+0.456, w5=+0.232
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.620, +0.500, +0.543, +0.660, +0.543, +0.581, +0.459, +0.342]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.491, +0.000, +0.171, +0.662, +0.171, +0.325, -0.166, -0.657]
    Input : [+0.713, -0.713, +0.376, +0.713, -0.713, +0.337, -0.713, -0.713]
    Gate  : [+0.689, +0.689, +0.456, +0.689, +0.689, +0.456, +0.689, +0.689]



<div>Loss: <span style="font-weight:bold">0.26115</span>= <span style="font-family:monaco">0.030 + 0.000 + 0.075 + 0.012 + 0.075 + 0.013 + 0.003 + 0.053</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.083</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.146</span> = <span style="color:blue; text-decoration:underline">+0.022</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.083</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.049</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.146</span> = <span style="color:blue; text-decoration:underline">+0.226</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.086</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.071</span> + <span style="color:red">-0.078</span> + <span style="color:red">-0.071</span> + <span style="color:blue">+0.057</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.113</span> = <span style="color:blue; text-decoration:underline">+0.037</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.086</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.051</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.226</span> = <span style="color:blue; text-decoration:underline">+0.290</span></div>


    w1=+0.373, w2=+0.315, w3=+1.000, w4=+0.453, w5=+0.203
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.611, +0.500, +0.542, +0.650, +0.542, +0.577, +0.465, +0.356]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.451, +0.000, +0.169, +0.620, +0.169, +0.311, -0.140, -0.591]
    Input : [+0.688, -0.688, +0.373, +0.688, -0.688, +0.315, -0.688, -0.688]
    Gate  : [+0.656, +0.656, +0.453, +0.656, +0.656, +0.453, +0.656, +0.656]



<div>Loss: <span style="font-weight:bold">0.24872</span>= <span style="font-family:monaco">0.025 + 0.000 + 0.075 + 0.015 + 0.075 + 0.012 + 0.002 + 0.043</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.073</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.090</span> + <span style="color:red">-0.085</span> + <span style="color:blue">+0.035</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.123</span> = <span style="color:red; text-decoration:underline">-0.022</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.073</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.053</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.035</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.123</span> = <span style="color:blue; text-decoration:underline">+0.185</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.076</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.071</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.071</span> + <span style="color:blue">+0.053</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.099</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.076</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.056</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.024</span> + <span style="color:blue">+0.198</span> = <span style="color:blue; text-decoration:underline">+0.242</span></div>


    w1=+0.376, w2=+0.296, w3=+1.000, w4=+0.452, w5=+0.179
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.604, +0.500, +0.542, +0.644, +0.542, +0.575, +0.470, +0.367]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.424, +0.000, +0.170, +0.594, +0.170, +0.304, -0.120, -0.545]
    Input : [+0.672, -0.672, +0.376, +0.672, -0.672, +0.296, -0.672, -0.672]
    Gate  : [+0.631, +0.631, +0.452, +0.631, +0.631, +0.452, +0.631, +0.631]



<div>Loss: <span style="font-weight:bold">0.23997</span>= <span style="font-family:monaco">0.022 + 0.000 + 0.075 + 0.017 + 0.075 + 0.011 + 0.002 + 0.037</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.066</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.094</span> + <span style="color:red">-0.085</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.108</span> = <span style="color:red; text-decoration:underline">-0.052</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.066</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.055</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.108</span> = <span style="color:blue; text-decoration:underline">+0.158</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.070</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.071</span> + <span style="color:red">-0.091</span> + <span style="color:red">-0.071</span> + <span style="color:blue">+0.051</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.089</span> = <span style="color:red; text-decoration:underline">-0.023</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.070</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.058</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.020</span> + <span style="color:blue">+0.178</span> = <span style="color:blue; text-decoration:underline">+0.211</span></div>


    w1=+0.381, w2=+0.280, w3=+1.000, w4=+0.455, w5=+0.158
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.600, +0.500, +0.543, +0.641, +0.543, +0.575, +0.474, +0.375]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.405, +0.000, +0.173, +0.578, +0.173, +0.301, -0.104, -0.510]
    Input : [+0.661, -0.661, +0.381, +0.661, -0.661, +0.280, -0.661, -0.661]
    Gate  : [+0.613, +0.613, +0.455, +0.613, +0.613, +0.455, +0.613, +0.613]



<div>Loss: <span style="font-weight:bold">0.23302</span>= <span style="font-family:monaco">0.020 + 0.000 + 0.075 + 0.019 + 0.075 + 0.011 + 0.001 + 0.032</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.097</span> + <span style="color:red">-0.085</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.096</span> = <span style="color:red; text-decoration:underline">-0.072</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.055</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.096</span> = <span style="color:blue; text-decoration:underline">+0.140</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.066</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.072</span> + <span style="color:red">-0.094</span> + <span style="color:red">-0.072</span> + <span style="color:blue">+0.049</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.082</span> = <span style="color:red; text-decoration:underline">-0.040</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.066</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.060</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.017</span> + <span style="color:blue">+0.165</span> = <span style="color:blue; text-decoration:underline">+0.188</span></div>


    w1=+0.388, w2=+0.266, w3=+1.000, w4=+0.459, w5=+0.139
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.597, +0.500, +0.544, +0.639, +0.544, +0.574, +0.477, +0.382]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.391, +0.000, +0.178, +0.569, +0.178, +0.300, -0.091, -0.482]
    Input : [+0.654, -0.654, +0.388, +0.654, -0.654, +0.266, -0.654, -0.654]
    Gate  : [+0.598, +0.598, +0.459, +0.598, +0.598, +0.459, +0.598, +0.598]



<div>Loss: <span style="font-weight:bold">0.22701</span>= <span style="font-family:monaco">0.019 + 0.000 + 0.074 + 0.019 + 0.074 + 0.011 + 0.001 + 0.029</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.087</span> = <span style="color:red; text-decoration:underline">-0.087</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.055</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.087</span> = <span style="color:blue; text-decoration:underline">+0.127</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.063</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.072</span> + <span style="color:red">-0.096</span> + <span style="color:red">-0.072</span> + <span style="color:blue">+0.049</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.077</span> = <span style="color:red; text-decoration:underline">-0.052</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.063</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.061</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.015</span> + <span style="color:blue">+0.155</span> = <span style="color:blue; text-decoration:underline">+0.172</span></div>


    w1=+0.397, w2=+0.254, w3=+1.000, w4=+0.464, w5=+0.122
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.594, +0.500, +0.546, +0.638, +0.546, +0.575, +0.480, +0.387]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.381, +0.000, +0.184, +0.565, +0.184, +0.302, -0.079, -0.460]
    Input : [+0.650, -0.650, +0.397, +0.650, -0.650, +0.254, -0.650, -0.650]
    Gate  : [+0.586, +0.586, +0.464, +0.586, +0.586, +0.464, +0.586, +0.586]



<div>Loss: <span style="font-weight:bold">0.22150</span>= <span style="font-family:monaco">0.018 + 0.000 + 0.073 + 0.020 + 0.073 + 0.011 + 0.001 + 0.026</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.055</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.035</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.080</span> = <span style="color:red; text-decoration:underline">-0.098</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.055</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.055</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.035</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.080</span> = <span style="color:blue; text-decoration:underline">+0.118</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.073</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.073</span> + <span style="color:blue">+0.049</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.074</span> = <span style="color:red; text-decoration:underline">-0.061</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.061</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.013</span> + <span style="color:blue">+0.147</span> = <span style="color:blue; text-decoration:underline">+0.160</span></div>


    w1=+0.406, w2=+0.242, w3=+1.000, w4=+0.470, w5=+0.106
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.592, +0.500, +0.548, +0.637, +0.548, +0.576, +0.483, +0.391]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.373, +0.000, +0.191, +0.564, +0.191, +0.305, -0.069, -0.442]
    Input : [+0.648, -0.648, +0.406, +0.648, -0.648, +0.242, -0.648, -0.648]
    Gate  : [+0.576, +0.576, +0.470, +0.576, +0.576, +0.470, +0.576, +0.576]



<div>Loss: <span style="font-weight:bold">0.21627</span>= <span style="font-family:monaco">0.017 + 0.000 + 0.071 + 0.020 + 0.071 + 0.012 + 0.001 + 0.024</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.053</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.036</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.074</span> = <span style="color:red; text-decoration:underline">-0.106</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.053</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.054</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.036</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.074</span> = <span style="color:blue; text-decoration:underline">+0.111</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.060</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.075</span> + <span style="color:red">-0.099</span> + <span style="color:red">-0.075</span> + <span style="color:blue">+0.049</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.070</span> = <span style="color:red; text-decoration:underline">-0.069</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.060</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.061</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.141</span> = <span style="color:blue; text-decoration:underline">+0.151</span></div>


    w1=+0.417, w2=+0.231, w3=+1.000, w4=+0.477, w5=+0.091
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.591, +0.500, +0.550, +0.638, +0.550, +0.577, +0.485, +0.395]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.368, +0.000, +0.199, +0.567, +0.199, +0.309, -0.059, -0.427]
    Input : [+0.648, -0.648, +0.417, +0.648, -0.648, +0.231, -0.648, -0.648]
    Gate  : [+0.568, +0.568, +0.477, +0.568, +0.568, +0.477, +0.568, +0.568]



<div>Loss: <span style="font-weight:bold">0.21120</span>= <span style="font-family:monaco">0.017 + 0.000 + 0.070 + 0.020 + 0.070 + 0.012 + 0.000 + 0.023</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.052</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.097</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.069</span> = <span style="color:red; text-decoration:underline">-0.112</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.052</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.053</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.037</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.069</span> = <span style="color:blue; text-decoration:underline">+0.106</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.059</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.076</span> + <span style="color:red">-0.099</span> + <span style="color:red">-0.076</span> + <span style="color:blue">+0.050</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.068</span> = <span style="color:red; text-decoration:underline">-0.074</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.059</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.060</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.136</span> = <span style="color:blue; text-decoration:underline">+0.144</span></div>


    w1=+0.428, w2=+0.220, w3=+1.000, w4=+0.484, w5=+0.076
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.590, +0.500, +0.552, +0.639, +0.552, +0.578, +0.488, +0.398]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.363, +0.000, +0.207, +0.571, +0.207, +0.314, -0.050, -0.413]
    Input : [+0.648, -0.648, +0.428, +0.648, -0.648, +0.220, -0.648, -0.648]
    Gate  : [+0.561, +0.561, +0.484, +0.561, +0.561, +0.484, +0.561, +0.561]



<div>Loss: <span style="font-weight:bold">0.20622</span>= <span style="font-family:monaco">0.016 + 0.000 + 0.068 + 0.019 + 0.068 + 0.012 + 0.000 + 0.021</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.050</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.096</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.038</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.065</span> = <span style="color:red; text-decoration:underline">-0.116</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.050</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.052</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.038</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.065</span> = <span style="color:blue; text-decoration:underline">+0.102</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.077</span> + <span style="color:red">-0.099</span> + <span style="color:red">-0.077</span> + <span style="color:blue">+0.050</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.066</span> = <span style="color:red; text-decoration:underline">-0.078</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.060</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.132</span> = <span style="color:blue; text-decoration:underline">+0.139</span></div>


    w1=+0.440, w2=+0.210, w3=+1.000, w4=+0.492, w5=+0.063
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.589, +0.500, +0.554, +0.640, +0.554, +0.579, +0.490, +0.401]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.360, +0.000, +0.216, +0.577, +0.216, +0.320, -0.041, -0.401]
    Input : [+0.650, -0.650, +0.440, +0.650, -0.650, +0.210, -0.650, -0.650]
    Gate  : [+0.555, +0.555, +0.492, +0.555, +0.555, +0.492, +0.555, +0.555]



<div>Loss: <span style="font-weight:bold">0.20131</span>= <span style="font-family:monaco">0.016 + 0.000 + 0.067 + 0.019 + 0.067 + 0.013 + 0.000 + 0.020</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.049</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.095</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.039</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.061</span> = <span style="color:red; text-decoration:underline">-0.119</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.049</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.050</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.039</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.061</span> = <span style="color:blue; text-decoration:underline">+0.100</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.078</span> + <span style="color:red">-0.099</span> + <span style="color:red">-0.078</span> + <span style="color:blue">+0.051</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.064</span> = <span style="color:red; text-decoration:underline">-0.081</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.059</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.129</span> = <span style="color:blue; text-decoration:underline">+0.134</span></div>


    w1=+0.452, w2=+0.200, w3=+1.000, w4=+0.500, w5=+0.049
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.572, +0.500, +0.637, +0.701, +0.637, +0.608, +0.537, +0.464]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.291, +0.000, +0.561, +0.852, +0.561, +0.440, +0.149, -0.143]
    Input : [+0.595, -0.595, +0.759, +0.595, -0.595, -0.164, -0.595, -0.595]
    Gate  : [+0.490, +0.490, +0.739, +0.490, +0.490, +0.739, +0.490, +0.490]



<div>Loss: <span style="font-weight:bold">0.08241</span>= <span style="font-family:monaco">0.011 + 0.000 + 0.020 + 0.002 + 0.020 + 0.024 + 0.003 + 0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.035</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.070</span> + <span style="color:red">-0.037</span> + <span style="color:red">-0.070</span> + <span style="color:blue">+0.080</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.009</span> = <span style="color:red; text-decoration:underline">-0.043</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.035</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.015</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.080</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.009</span> = <span style="color:blue; text-decoration:underline">+0.119</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.043</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.072</span> + <span style="color:red">-0.041</span> + <span style="color:red">-0.072</span> + <span style="color:blue">+0.064</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.021</span> = <span style="color:red; text-decoration:underline">-0.055</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.043</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.018</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.022</span> + <span style="color:blue">+0.042</span> = <span style="color:blue; text-decoration:underline">+0.046</span></div>


    w1=+0.763, w2=-0.175, w3=+1.000, w4=+0.745, w5=-0.254
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.528, +0.500, +0.700, +0.723, +0.700, +0.543, +0.515, +0.487]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.111, +0.000, +0.848, +0.960, +0.848, +0.171, +0.059, -0.052]
    Input : [+0.168, -0.168, +0.837, +0.168, -0.168, -0.669, -0.168, -0.168]
    Gate  : [+0.661, +0.661, +1.013, +0.661, +0.661, +1.013, +0.661, +0.661]



<div>Loss: <span style="font-weight:bold">0.01076</span>= <span style="font-family:monaco">0.002 + 0.000 + 0.002 + 0.000 + 0.002 + 0.004 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.018</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.031</span> + <span style="color:red">-0.013</span> + <span style="color:red">-0.031</span> + <span style="color:blue">+0.043</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.004</span> = <span style="color:red; text-decoration:underline">-0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.018</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.005</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.043</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.004</span> = <span style="color:blue; text-decoration:underline">+0.065</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.005</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.026</span> + <span style="color:red">-0.008</span> + <span style="color:red">-0.026</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.046</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.005</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.004</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div>


    w1=+0.838, w2=-0.675, w3=+1.000, w4=+1.018, w5=-0.353
    



<div style="font-weight:bold">Iteration 149</div>


    Output: [+0.504, +0.500, +0.726, +0.730, +0.726, +0.506, +0.502, +0.498]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.017, +0.000, +0.975, +0.993, +0.975, +0.025, +0.008, -0.009]
    Input : [+0.022, -0.022, +0.856, +0.022, -0.022, -0.834, -0.022, -0.022]
    Gate  : [+0.780, +0.780, +1.140, +0.780, +0.780, +1.140, +0.780, +0.780]



<div>Loss: <span style="font-weight:bold">0.00026</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.003</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.003</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.011</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.004</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.009</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.856, w2=-0.835, w3=+1.000, w4=+1.141, w5=-0.360
    



<div style="font-weight:bold">Iteration 199</div>


    Output: [+0.500, +0.500, +0.730, +0.731, +0.730, +0.501, +0.500, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.002, +0.000, +0.997, +0.999, +0.997, +0.003, +0.001, -0.001]
    Input : [+0.002, -0.002, +0.859, +0.002, -0.002, -0.857, -0.002, -0.002]
    Gate  : [+0.800, +0.800, +1.160, +0.800, +0.800, +1.160, +0.800, +0.800]



<div>Loss: <span style="font-weight:bold">0.00000</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.860, w2=-0.857, w3=+1.000, w4=+1.160, w5=-0.360
    



<div style="font-weight:bold">Iteration 249</div>


    Output: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.000, +0.000, +1.000, +1.000, +1.000, +0.000, +0.000, -0.000]
    Input : [+0.000, -0.000, +0.860, +0.000, -0.000, -0.860, -0.000, -0.000]
    Gate  : [+0.803, +0.803, +1.162, +0.803, +0.803, +1.162, +0.803, +0.803]



<div>Loss: <span style="font-weight:bold">0.00000</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.860, w2=-0.860, w3=+1.000, w4=+1.162, w5=-0.360
    





    (0.8599643112688363,
     -0.859699702729852,
     1.0,
     1.162480041288255,
     -0.3597956247833731)



## Discussion

We see that after adding input gate (assuming it is possible for the input gate to exhibit the same properties as the _true_ input gate, manifested by using bilinear gate here), can reach the optimal (loss = 0.0) faster (after iteration 199) compared to the one without input gate (only after iteration 249), although there are more parameters to learn with the input gate (two more: $w_4$ and $w_5$) and that the initial loss is higher with input gate (due to the incorrect gate value initially).

Also we see that the gate learned is actually not the true gate that we want. This is because the input is already separable even without input gate.

## Noisy embedding

In previous experiment, the input gate learned is not the true gate that we want, but that's because the input embedding is ideal, i.e., it allows separation even without input gate.

Now let's experiment with noisy embedding, in which the true function cannot be obtained without input gate.


```python
import random
a_1 = 1.0 + 0.2*(random.random()-0.5)
a_2 = 1.0/a_1
b_1 = -1.0 + 0.2*(random.random()-0.5)
b_2 = 1.0/b_1
embedding['a'] = (a_1, a_2)
embedding['b'] = (b_1, b_2)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

from pprint import pprint
pprint(embedding)
```

    {'(': (1, 0),
     ')': (0, 1),
     'a': (1.0367606488244225, 0.9645427815319715),
     'b': (-0.9331636393562619, -1.0716234086122824)}


Here we make the input embedding such that the other characters have noise which should be ignored.

Let's see how the two models perform in this case.


```python
embedding['a'] = (a_1, a_2)
embedding['b'] = (b_1, b_2)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

experiment('ab(ab)bb', _w1=1.0, _w2=1.0, alpha=1e-1, max_iter=250, fixed_w3=True)
```

    w1=+1.000, w2=+1.000, w3=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.881, +0.499, +0.730, +0.952, +0.730, +0.880, +0.497, +0.117]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+2.001, -0.003, +0.997, +2.998, +0.993, +1.993, -0.012, -2.017]



<div>Loss: <span style="font-weight:bold">1.57825</span>= <span style="font-family:monaco; font-size:12px">0.434 + 0.000 + 0.000 + 0.273 + 0.000 + 0.431 + 0.000 + 0.440</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.395</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.474</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.459</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.252</span> = <span style="color:blue; text-decoration:underline">+1.576</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.367</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.190</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.299</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.519</span> = <span style="color:blue; text-decoration:underline">+1.376</span></div>


    w1=+0.842, w2=+0.862, w3=+1.000
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.846, +0.499, +0.698, +0.927, +0.697, +0.845, +0.496, +0.151]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.705, -0.005, +0.837, +2.542, +0.832, +1.695, -0.016, -1.726]



<div>Loss: <span style="font-weight:bold">1.16557</span>= <span style="font-family:monaco; font-size:12px">0.326 + 0.000 + 0.003 + 0.177 + 0.003 + 0.323 + 0.000 + 0.334</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.359</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.037</span> + <span style="color:blue">+0.420</span> + <span style="color:red">-0.041</span> + <span style="color:blue">+0.416</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.230</span> = <span style="color:blue; text-decoration:underline">+1.346</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.334</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.168</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.271</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.474</span> = <span style="color:blue; text-decoration:underline">+1.259</span></div>


    w1=+0.708, w2=+0.736, w3=+1.000
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.809, +0.499, +0.669, +0.895, +0.667, +0.807, +0.496, +0.188]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.444, -0.006, +0.702, +2.146, +0.697, +1.433, -0.017, -1.466]



<div>Loss: <span style="font-weight:bold">0.85001</span>= <span style="font-family:monaco; font-size:12px">0.241 + 0.000 + 0.009 + 0.106 + 0.009 + 0.237 + 0.000 + 0.248</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.320</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.069</span> + <span style="color:blue">+0.352</span> + <span style="color:red">-0.077</span> + <span style="color:blue">+0.371</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.206</span> = <span style="color:blue; text-decoration:underline">+1.102</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.298</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.141</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.242</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.424</span> = <span style="color:blue; text-decoration:underline">+1.126</span></div>


    w1=+0.598, w2=+0.624, w3=+1.000
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.772, +0.499, +0.644, +0.860, +0.643, +0.771, +0.496, +0.224]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.221, -0.005, +0.593, +1.814, +0.588, +1.212, -0.015, -1.241]



<div>Loss: <span style="font-weight:bold">0.62204</span>= <span style="font-family:monaco; font-size:12px">0.176 + 0.000 + 0.017 + 0.057 + 0.018 + 0.173 + 0.000 + 0.181</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.282</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.096</span> + <span style="color:blue">+0.276</span> + <span style="color:red">-0.106</span> + <span style="color:blue">+0.327</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.182</span> = <span style="color:blue; text-decoration:underline">+0.863</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.263</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.110</span> + <span style="color:blue">+0.019</span> + <span style="color:blue">+0.213</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.374</span> = <span style="color:blue; text-decoration:underline">+0.989</span></div>


    w1=+0.511, w2=+0.525, w3=+1.000
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.738, +0.499, +0.624, +0.824, +0.624, +0.737, +0.498, +0.259]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.036, -0.003, +0.508, +1.544, +0.505, +1.030, -0.010, -1.050]



<div>Loss: <span style="font-weight:bold">0.46547</span>= <span style="font-family:monaco; font-size:12px">0.129 + 0.000 + 0.025 + 0.027 + 0.026 + 0.127 + 0.000 + 0.132</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.247</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.118</span> + <span style="color:blue">+0.199</span> + <span style="color:red">-0.130</span> + <span style="color:blue">+0.286</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.159</span> = <span style="color:blue; text-decoration:underline">+0.642</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.230</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.080</span> + <span style="color:blue">+0.023</span> + <span style="color:blue">+0.186</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.327</span> = <span style="color:blue; text-decoration:underline">+0.858</span></div>


    w1=+0.447, w2=+0.439, w3=+1.000
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.708, +0.500, +0.610, +0.791, +0.610, +0.708, +0.499, +0.291]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.887, -0.001, +0.446, +1.333, +0.446, +0.885, -0.003, -0.891]



<div>Loss: <span style="font-weight:bold">0.36166</span>= <span style="font-family:monaco; font-size:12px">0.095 + 0.000 + 0.033 + 0.010 + 0.033 + 0.095 + 0.000 + 0.096</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.216</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.134</span> + <span style="color:blue">+0.129</span> + <span style="color:red">-0.147</span> + <span style="color:blue">+0.251</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.138</span> = <span style="color:blue; text-decoration:underline">+0.453</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.201</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.013</span> + <span style="color:blue">+0.052</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.163</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.284</span> = <span style="color:blue; text-decoration:underline">+0.739</span></div>


    w1=+0.402, w2=+0.365, w3=+1.000
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.683, +0.501, +0.600, +0.764, +0.600, +0.684, +0.501, +0.319]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.769, +0.003, +0.404, +1.173, +0.407, +0.772, +0.006, -0.761]



<div>Loss: <span style="font-weight:bold">0.29370</span>= <span style="font-family:monaco; font-size:12px">0.072 + 0.000 + 0.038 + 0.003 + 0.037 + 0.073 + 0.000 + 0.071</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.190</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.145</span> + <span style="color:blue">+0.070</span> + <span style="color:red">-0.158</span> + <span style="color:blue">+0.222</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.120</span> = <span style="color:blue; text-decoration:underline">+0.299</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.177</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.028</span> + <span style="color:blue">+0.028</span> + <span style="color:blue">+0.145</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.246</span> = <span style="color:blue; text-decoration:underline">+0.637</span></div>


    w1=+0.372, w2=+0.302, w3=+1.000
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.663, +0.502, +0.593, +0.742, +0.595, +0.665, +0.504, +0.342]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.676, +0.006, +0.378, +1.054, +0.384, +0.686, +0.016, -0.654]



<div>Loss: <span style="font-weight:bold">0.24857</span>= <span style="font-family:monaco; font-size:12px">0.056 + 0.000 + 0.041 + 0.000 + 0.040 + 0.058 + 0.000 + 0.053</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.169</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.152</span> + <span style="color:blue">+0.023</span> + <span style="color:red">-0.164</span> + <span style="color:blue">+0.199</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.104</span> = <span style="color:blue; text-decoration:underline">+0.180</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.157</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.015</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.130</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.214</span> = <span style="color:blue; text-decoration:underline">+0.553</span></div>


    w1=+0.354, w2=+0.246, w3=+1.000
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.647, +0.503, +0.590, +0.725, +0.593, +0.650, +0.507, +0.362]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.604, +0.010, +0.364, +0.968, +0.374, +0.621, +0.027, -0.567]



<div>Loss: <span style="font-weight:bold">0.21745</span>= <span style="font-family:monaco; font-size:12px">0.045 + 0.000 + 0.043 + 0.000 + 0.042 + 0.047 + 0.000 + 0.040</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.152</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.156</span> + <span style="color:red">-0.013</span> + <span style="color:red">-0.167</span> + <span style="color:blue">+0.182</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.091</span> = <span style="color:blue; text-decoration:underline">+0.091</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.141</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.030</span> + <span style="color:blue">+0.118</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.188</span> = <span style="color:blue; text-decoration:underline">+0.484</span></div>


    w1=+0.345, w2=+0.198, w3=+1.000
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.634, +0.504, +0.589, +0.713, +0.592, +0.639, +0.509, +0.379]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.548, +0.015, +0.359, +0.908, +0.374, +0.572, +0.038, -0.496]



<div>Loss: <span style="font-weight:bold">0.19479</span>= <span style="font-family:monaco; font-size:12px">0.037 + 0.000 + 0.044 + 0.001 + 0.042 + 0.040 + 0.000 + 0.030</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.139</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.157</span> + <span style="color:red">-0.040</span> + <span style="color:red">-0.167</span> + <span style="color:blue">+0.168</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.080</span> = <span style="color:blue; text-decoration:underline">+0.026</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.129</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.030</span> + <span style="color:blue">+0.109</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.165</span> = <span style="color:blue; text-decoration:underline">+0.429</span></div>


    w1=+0.342, w2=+0.155, w3=+1.000
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.623, +0.505, +0.589, +0.704, +0.594, +0.631, +0.512, +0.393]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.504, +0.019, +0.361, +0.865, +0.380, +0.535, +0.050, -0.436]



<div>Loss: <span style="font-weight:bold">0.17732</span>= <span style="font-family:monaco; font-size:12px">0.031 + 0.000 + 0.044 + 0.002 + 0.041 + 0.035 + 0.000 + 0.024</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.128</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.156</span> + <span style="color:red">-0.058</span> + <span style="color:red">-0.166</span> + <span style="color:blue">+0.158</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.071</span> = <span style="color:red; text-decoration:underline">-0.020</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.119</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.023</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.103</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.146</span> = <span style="color:blue; text-decoration:underline">+0.384</span></div>


    w1=+0.344, w2=+0.116, w3=+1.000
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.615, +0.506, +0.591, +0.698, +0.596, +0.624, +0.515, +0.405]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.469, +0.023, +0.367, +0.837, +0.391, +0.507, +0.061, -0.385]



<div>Loss: <span style="font-weight:bold">0.16317</span>= <span style="font-family:monaco; font-size:12px">0.027 + 0.000 + 0.043 + 0.003 + 0.040 + 0.032 + 0.000 + 0.018</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.119</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.155</span> + <span style="color:red">-0.071</span> + <span style="color:red">-0.163</span> + <span style="color:blue">+0.150</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.063</span> = <span style="color:red; text-decoration:underline">-0.052</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.111</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.029</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.098</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.129</span> = <span style="color:blue; text-decoration:underline">+0.348</span></div>


    w1=+0.349, w2=+0.082, w3=+1.000
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.609, +0.507, +0.593, +0.694, +0.600, +0.619, +0.518, +0.416]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.441, +0.027, +0.377, +0.818, +0.404, +0.486, +0.072, -0.341]



<div>Loss: <span style="font-weight:bold">0.15126</span>= <span style="font-family:monaco; font-size:12px">0.024 + 0.000 + 0.042 + 0.003 + 0.038 + 0.029 + 0.001 + 0.014</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.112</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.152</span> + <span style="color:red">-0.080</span> + <span style="color:red">-0.159</span> + <span style="color:blue">+0.144</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.056</span> = <span style="color:red; text-decoration:underline">-0.073</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.105</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.032</span> + <span style="color:blue">+0.028</span> + <span style="color:blue">+0.094</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.115</span> = <span style="color:blue; text-decoration:underline">+0.318</span></div>


    w1=+0.357, w2=+0.050, w3=+1.000
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.603, +0.508, +0.596, +0.691, +0.603, +0.615, +0.521, +0.425]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.418, +0.032, +0.388, +0.806, +0.420, +0.470, +0.084, -0.303]



<div>Loss: <span style="font-weight:bold">0.14097</span>= <span style="font-family:monaco; font-size:12px">0.022 + 0.000 + 0.040 + 0.004 + 0.036 + 0.027 + 0.001 + 0.011</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.107</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.149</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.154</span> + <span style="color:blue">+0.139</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.050</span> = <span style="color:red; text-decoration:underline">-0.086</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.099</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.034</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.091</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.102</span> = <span style="color:blue; text-decoration:underline">+0.293</span></div>


    w1=+0.365, w2=+0.021, w3=+1.000
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.598, +0.509, +0.599, +0.690, +0.607, +0.612, +0.524, +0.433]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.399, +0.036, +0.401, +0.800, +0.437, +0.457, +0.094, -0.269]



<div>Loss: <span style="font-weight:bold">0.13193</span>= <span style="font-family:monaco; font-size:12px">0.020 + 0.000 + 0.038 + 0.004 + 0.034 + 0.026 + 0.001 + 0.009</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.102</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.146</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.149</span> + <span style="color:blue">+0.136</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.044</span> = <span style="color:red; text-decoration:underline">-0.094</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.095</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.035</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.088</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.091</span> = <span style="color:blue; text-decoration:underline">+0.271</span></div>


    w1=+0.375, w2=-0.007, w3=+1.000
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.552, +0.526, +0.663, +0.707, +0.685, +0.593, +0.567, +0.542]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.207, +0.102, +0.675, +0.882, +0.777, +0.376, +0.272, +0.167]



<div>Loss: <span style="font-weight:bold">0.05422</span>= <span style="font-family:monaco; font-size:12px">0.005 + 0.001 + 0.011 + 0.001 + 0.005 + 0.018 + 0.009 + 0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.053</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.075</span> + <span style="color:red">-0.051</span> + <span style="color:red">-0.055</span> + <span style="color:blue">+0.112</span> + <span style="color:blue">+0.018</span> + <span style="color:red">-0.027</span> = <span style="color:red; text-decoration:underline">-0.022</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.050</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.020</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.073</span> + <span style="color:red">-0.019</span> + <span style="color:red">-0.056</span> = <span style="color:blue; text-decoration:underline">+0.041</span></div>


    w1=+0.575, w2=-0.405, w3=+1.000
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.544, +0.529, +0.675, +0.712, +0.699, +0.591, +0.576, +0.561]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.176, +0.115, +0.730, +0.906, +0.845, +0.367, +0.305, +0.244]



<div>Loss: <span style="font-weight:bold">0.05201</span>= <span style="font-family:monaco; font-size:12px">0.004 + 0.002 + 0.007 + 0.001 + 0.002 + 0.017 + 0.012 + 0.007</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.046</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.062</span> + <span style="color:red">-0.040</span> + <span style="color:red">-0.038</span> + <span style="color:blue">+0.109</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.040</span> = <span style="color:red; text-decoration:underline">-0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.042</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.071</span> + <span style="color:red">-0.022</span> + <span style="color:red">-0.082</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div>


    w1=+0.615, w2=-0.478, w3=+1.000
    



<div style="font-weight:bold">Iteration 149</div>


    Output: [+0.543, +0.529, +0.676, +0.713, +0.701, +0.590, +0.576, +0.562]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.174, +0.116, +0.734, +0.908, +0.850, +0.366, +0.308, +0.250]



<div>Loss: <span style="font-weight:bold">0.05199</span>= <span style="font-family:monaco; font-size:12px">0.004 + 0.002 + 0.007 + 0.001 + 0.002 + 0.017 + 0.012 + 0.008</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.045</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.061</span> + <span style="color:red">-0.039</span> + <span style="color:red">-0.037</span> + <span style="color:blue">+0.109</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.041</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.042</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.071</span> + <span style="color:red">-0.022</span> + <span style="color:red">-0.084</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.618, w2=-0.484, w3=+1.000
    



<div style="font-weight:bold">Iteration 199</div>


    Output: [+0.543, +0.529, +0.676, +0.713, +0.701, +0.590, +0.576, +0.562]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.174, +0.116, +0.735, +0.908, +0.851, +0.366, +0.308, +0.250]



<div>Loss: <span style="font-weight:bold">0.05199</span>= <span style="font-family:monaco; font-size:12px">0.004 + 0.002 + 0.007 + 0.001 + 0.002 + 0.017 + 0.012 + 0.008</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.045</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.061</span> + <span style="color:red">-0.039</span> + <span style="color:red">-0.037</span> + <span style="color:blue">+0.109</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.041</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.042</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.071</span> + <span style="color:red">-0.022</span> + <span style="color:red">-0.085</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.619, w2=-0.485, w3=+1.000
    



<div style="font-weight:bold">Iteration 249</div>


    Output: [+0.543, +0.529, +0.676, +0.713, +0.701, +0.590, +0.576, +0.562]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.174, +0.116, +0.735, +0.908, +0.851, +0.366, +0.308, +0.250]



<div>Loss: <span style="font-weight:bold">0.05199</span>= <span style="font-family:monaco; font-size:12px">0.004 + 0.002 + 0.007 + 0.001 + 0.002 + 0.017 + 0.012 + 0.008</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.045</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.061</span> + <span style="color:red">-0.039</span> + <span style="color:red">-0.037</span> + <span style="color:blue">+0.109</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.041</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.042</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.071</span> + <span style="color:red">-0.022</span> + <span style="color:red">-0.085</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div>


    w1=+0.619, w2=-0.485, w3=+1.000
    





    (0.6186425813596521, -0.48489064055179726, 1.0)




```python
embedding['a'] = (a_1, a_2)
embedding['b'] = (b_1, b_2)
embedding['('] = (1, 0)
embedding[')'] = (0, 1)

experiment_gated('ab(ab)bb', _w1=1.0, _w2=1.0, _w4=1.0, _w5=1.0, alpha=1e-1, max_iter=250, fixed_w3=True)
```

    w1=+1.000, w2=+1.000, w3=+1.000, w4=+1.000, w5=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.982, +0.498, +0.730, +0.993, +0.728, +0.879, +0.117, +0.002]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+4.003, -0.007, +0.993, +4.996, +0.986, +1.986, -2.024, -6.033]
    Input : [+2.001, -2.005, +1.000, +2.001, -2.005, +1.000, -2.005, -2.005]
    Gate  : [+2.000, +2.000, +1.000, +2.000, +2.000, +1.000, +2.000, +2.000]



<div>Loss: <span style="font-weight:bold">5.29140</span>= <span style="font-family:monaco">1.326 + 0.000 + 0.000 + 0.768 + 0.000 + 0.428 + 0.443 + 2.326</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+1.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.860</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.537</span> + <span style="color:blue">+0.173</span> + <span style="color:blue">+1.154</span> = <span style="color:blue; text-decoration:underline">+3.717</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.930</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.450</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.217</span> + <span style="color:blue">+0.602</span> + <span style="color:blue">+1.849</span> = <span style="color:blue; text-decoration:underline">+4.049</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.965</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.786</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.756</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+1.003</span> = <span style="color:blue; text-decoration:underline">+3.511</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.965</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.524</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.771</span> + <span style="color:blue">+1.999</span> = <span style="color:blue; text-decoration:underline">+4.256</span></div>


    w1=+0.628, w2=+0.595, w3=+1.000, w4=+0.649, w5=+0.574
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.817, +0.500, +0.601, +0.871, +0.601, +0.689, +0.332, +0.100]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+1.499, +0.002, +0.409, +1.908, +0.411, +0.797, -0.700, -2.198]
    Input : [+1.225, -1.224, +0.628, +1.225, -1.224, +0.595, -1.224, -1.224]
    Gate  : [+1.223, +1.223, +0.649, +1.223, +1.223, +0.649, +1.223, +1.223]



<div>Loss: <span style="font-weight:bold">1.04987</span>= <span style="font-family:monaco">0.258 + 0.000 + 0.037 + 0.069 + 0.037 + 0.077 + 0.060 + 0.511</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.403</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.101</span> + <span style="color:blue">+0.286</span> + <span style="color:red">-0.117</span> + <span style="color:blue">+0.171</span> + <span style="color:blue">+0.040</span> + <span style="color:blue">+0.552</span> = <span style="color:blue; text-decoration:underline">+1.234</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.375</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.017</span> + <span style="color:blue">+0.147</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.073</span> + <span style="color:blue">+0.155</span> + <span style="color:blue">+0.894</span> = <span style="color:blue; text-decoration:underline">+1.695</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.082</span> + <span style="color:blue">+0.259</span> + <span style="color:red">-0.082</span> + <span style="color:blue">+0.232</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.489</span> = <span style="color:blue; text-decoration:underline">+1.205</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.389</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.171</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.205</span> + <span style="color:blue">+0.978</span> = <span style="color:blue; text-decoration:underline">+1.744</span></div>


    w1=+0.505, w2=+0.426, w3=+1.000, w4=+0.528, w5=+0.400
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.704, +0.502, +0.568, +0.758, +0.569, +0.623, +0.412, +0.228]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.867, +0.006, +0.273, +1.140, +0.279, +0.504, -0.357, -1.218]
    Input : [+0.934, -0.927, +0.505, +0.934, -0.927, +0.426, -0.927, -0.927]
    Gate  : [+0.928, +0.928, +0.528, +0.928, +0.928, +0.528, +0.928, +0.928]



<div>Loss: <span style="font-weight:bold">0.42849</span>= <span style="font-family:monaco">0.091 + 0.000 + 0.057 + 0.002 + 0.056 + 0.031 + 0.016 + 0.175</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.197</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.102</span> + <span style="color:blue">+0.042</span> + <span style="color:red">-0.117</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.013</span> + <span style="color:blue">+0.275</span> = <span style="color:blue; text-decoration:underline">+0.397</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.183</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.016</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.032</span> + <span style="color:blue">+0.041</span> + <span style="color:blue">+0.059</span> + <span style="color:blue">+0.451</span> = <span style="color:blue; text-decoration:underline">+0.803</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.191</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.084</span> + <span style="color:blue">+0.039</span> + <span style="color:red">-0.084</span> + <span style="color:blue">+0.117</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.247</span> = <span style="color:blue; text-decoration:underline">+0.424</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.191</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.025</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.081</span> + <span style="color:blue">+0.500</span> = <span style="color:blue; text-decoration:underline">+0.795</span></div>


    w1=+0.465, w2=+0.345, w3=+1.000, w4=+0.486, w5=+0.321
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.659, +0.502, +0.559, +0.709, +0.561, +0.602, +0.441, +0.292]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.658, +0.009, +0.235, +0.893, +0.244, +0.412, -0.237, -0.885]
    Input : [+0.815, -0.804, +0.465, +0.815, -0.804, +0.345, -0.804, -0.804]
    Gate  : [+0.807, +0.807, +0.486, +0.807, +0.807, +0.486, +0.807, +0.807]



<div>Loss: <span style="font-weight:bold">0.30269</span>= <span style="font-family:monaco">0.053 + 0.000 + 0.064 + 0.001 + 0.062 + 0.021 + 0.007 + 0.095</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.133</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.030</span> + <span style="color:red">-0.111</span> + <span style="color:blue">+0.066</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.177</span> = <span style="color:blue; text-decoration:underline">+0.142</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.123</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.015</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.032</span> + <span style="color:blue">+0.032</span> + <span style="color:blue">+0.294</span> = <span style="color:blue; text-decoration:underline">+0.511</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.129</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.028</span> + <span style="color:red">-0.083</span> + <span style="color:blue">+0.085</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.161</span> = <span style="color:blue; text-decoration:underline">+0.180</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.129</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.018</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.046</span> + <span style="color:blue">+0.330</span> = <span style="color:blue; text-decoration:underline">+0.484</span></div>


    w1=+0.451, w2=+0.294, w3=+1.000, w4=+0.468, w5=+0.272
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.636, +0.503, +0.555, +0.685, +0.558, +0.592, +0.457, +0.328]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.556, +0.011, +0.222, +0.778, +0.234, +0.371, -0.174, -0.718]
    Input : [+0.751, -0.736, +0.451, +0.751, -0.736, +0.294, -0.736, -0.736]
    Gate  : [+0.740, +0.740, +0.468, +0.740, +0.740, +0.468, +0.740, +0.740]



<div>Loss: <span style="font-weight:bold">0.25675</span>= <span style="font-family:monaco">0.038 + 0.000 + 0.066 + 0.005 + 0.064 + 0.017 + 0.004 + 0.063</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.104</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.096</span> + <span style="color:red">-0.060</span> + <span style="color:red">-0.107</span> + <span style="color:blue">+0.057</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.131</span> = <span style="color:blue; text-decoration:underline">+0.032</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.097</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.029</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.028</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.220</span> = <span style="color:blue; text-decoration:underline">+0.378</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.102</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.056</span> + <span style="color:red">-0.083</span> + <span style="color:blue">+0.071</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.120</span> = <span style="color:blue; text-decoration:underline">+0.070</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.102</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.035</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.031</span> + <span style="color:blue">+0.248</span> = <span style="color:blue; text-decoration:underline">+0.341</span></div>


    w1=+0.448, w2=+0.256, w3=+1.000, w4=+0.461, w5=+0.238
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.622, +0.503, +0.555, +0.672, +0.558, +0.587, +0.467, +0.350]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.497, +0.013, +0.220, +0.717, +0.233, +0.351, -0.133, -0.617]
    Input : [+0.712, -0.693, +0.448, +0.712, -0.693, +0.256, -0.693, -0.693]
    Gate  : [+0.699, +0.699, +0.461, +0.699, +0.699, +0.461, +0.699, +0.699]



<div>Loss: <span style="font-weight:bold">0.23339</span>= <span style="font-family:monaco">0.031 + 0.000 + 0.066 + 0.008 + 0.064 + 0.015 + 0.002 + 0.047</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.088</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.094</span> + <span style="color:red">-0.074</span> + <span style="color:red">-0.105</span> + <span style="color:blue">+0.053</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.105</span> = <span style="color:red; text-decoration:underline">-0.026</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.082</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.035</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.015</span> + <span style="color:blue">+0.178</span> = <span style="color:blue; text-decoration:underline">+0.305</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.087</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.070</span> + <span style="color:red">-0.084</span> + <span style="color:blue">+0.064</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.096</span> = <span style="color:blue; text-decoration:underline">+0.010</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.087</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.043</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.202</span> = <span style="color:blue; text-decoration:underline">+0.260</span></div>


    w1=+0.450, w2=+0.226, w3=+1.000, w4=+0.460, w5=+0.212
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.613, +0.504, +0.555, +0.664, +0.559, +0.584, +0.474, +0.366]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.460, +0.015, +0.222, +0.682, +0.237, +0.341, -0.104, -0.549]
    Input : [+0.685, -0.662, +0.450, +0.685, -0.662, +0.226, -0.662, -0.662]
    Gate  : [+0.672, +0.672, +0.460, +0.672, +0.672, +0.460, +0.672, +0.672]



<div>Loss: <span style="font-weight:bold">0.21859</span>= <span style="font-family:monaco">0.026 + 0.000 + 0.066 + 0.010 + 0.063 + 0.014 + 0.001 + 0.037</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.079</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.093</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.103</span> + <span style="color:blue">+0.051</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.088</span> = <span style="color:red; text-decoration:underline">-0.060</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.073</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.038</span> + <span style="color:blue">+0.025</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.151</span> = <span style="color:blue; text-decoration:underline">+0.260</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.077</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.083</span> + <span style="color:red">-0.077</span> + <span style="color:red">-0.085</span> + <span style="color:blue">+0.061</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.081</span> = <span style="color:red; text-decoration:underline">-0.028</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.077</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.004</span> + <span style="color:red">-0.047</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.016</span> + <span style="color:blue">+0.171</span> = <span style="color:blue; text-decoration:underline">+0.210</span></div>


    w1=+0.456, w2=+0.200, w3=+1.000, w4=+0.463, w5=+0.191
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.607, +0.504, +0.557, +0.660, +0.561, +0.584, +0.480, +0.378]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.435, +0.017, +0.228, +0.663, +0.245, +0.337, -0.081, -0.500]
    Input : [+0.666, -0.640, +0.456, +0.666, -0.640, +0.200, -0.640, -0.640]
    Gate  : [+0.654, +0.654, +0.463, +0.654, +0.654, +0.463, +0.654, +0.654]



<div>Loss: <span style="font-weight:bold">0.20766</span>= <span style="font-family:monaco">0.024 + 0.000 + 0.065 + 0.012 + 0.062 + 0.014 + 0.001 + 0.031</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.073</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.102</span> + <span style="color:blue">+0.050</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.076</span> = <span style="color:red; text-decoration:underline">-0.081</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.068</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.012</span> + <span style="color:red">-0.040</span> + <span style="color:blue">+0.024</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.132</span> = <span style="color:blue; text-decoration:underline">+0.230</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.071</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.084</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.059</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.070</span> = <span style="color:red; text-decoration:underline">-0.053</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.071</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.005</span> + <span style="color:red">-0.049</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.012</span> + <span style="color:blue">+0.150</span> = <span style="color:blue; text-decoration:underline">+0.176</span></div>


    w1=+0.464, w2=+0.177, w3=+1.000, w4=+0.468, w5=+0.174
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.603, +0.505, +0.559, +0.658, +0.563, +0.584, +0.484, +0.387]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.418, +0.019, +0.236, +0.654, +0.255, +0.338, -0.062, -0.462]
    Input : [+0.652, -0.623, +0.464, +0.652, -0.623, +0.177, -0.623, -0.623]
    Gate  : [+0.642, +0.642, +0.468, +0.642, +0.642, +0.468, +0.642, +0.642]



<div>Loss: <span style="font-weight:bold">0.19871</span>= <span style="font-family:monaco">0.022 + 0.000 + 0.063 + 0.012 + 0.060 + 0.014 + 0.000 + 0.026</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.069</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.101</span> + <span style="color:blue">+0.050</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.068</span> = <span style="color:red; text-decoration:underline">-0.094</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.064</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.012</span> + <span style="color:red">-0.040</span> + <span style="color:blue">+0.023</span> + <span style="color:blue">+0.028</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.118</span> = <span style="color:blue; text-decoration:underline">+0.210</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.067</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.084</span> + <span style="color:red">-0.088</span> + <span style="color:blue">+0.058</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.062</span> = <span style="color:red; text-decoration:underline">-0.070</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.067</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.005</span> + <span style="color:red">-0.050</span> + <span style="color:red">-0.010</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.135</span> = <span style="color:blue; text-decoration:underline">+0.151</span></div>


    w1=+0.474, w2=+0.156, w3=+1.000, w4=+0.475, w5=+0.158
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.600, +0.505, +0.561, +0.657, +0.566, +0.584, +0.489, +0.394]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.406, +0.021, +0.246, +0.652, +0.266, +0.340, -0.046, -0.432]
    Input : [+0.642, -0.609, +0.474, +0.642, -0.609, +0.156, -0.609, -0.609]
    Gate  : [+0.633, +0.633, +0.475, +0.633, +0.633, +0.475, +0.633, +0.633]



<div>Loss: <span style="font-weight:bold">0.19087</span>= <span style="font-family:monaco">0.021 + 0.000 + 0.062 + 0.013 + 0.058 + 0.014 + 0.000 + 0.023</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.066</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.100</span> + <span style="color:blue">+0.051</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.061</span> = <span style="color:red; text-decoration:underline">-0.102</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.061</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.012</span> + <span style="color:red">-0.040</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.029</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.108</span> = <span style="color:blue; text-decoration:underline">+0.195</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.064</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.086</span> + <span style="color:red">-0.084</span> + <span style="color:red">-0.089</span> + <span style="color:blue">+0.058</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.056</span> = <span style="color:red; text-decoration:underline">-0.082</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.064</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.050</span> + <span style="color:red">-0.011</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.123</span> = <span style="color:blue; text-decoration:underline">+0.133</span></div>


    w1=+0.484, w2=+0.136, w3=+1.000, w4=+0.483, w5=+0.145
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.598, +0.506, +0.564, +0.658, +0.569, +0.585, +0.492, +0.400]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.398, +0.022, +0.256, +0.654, +0.278, +0.344, -0.031, -0.407]
    Input : [+0.633, -0.598, +0.484, +0.633, -0.598, +0.136, -0.598, -0.598]
    Gate  : [+0.628, +0.628, +0.483, +0.628, +0.628, +0.483, +0.628, +0.628]



<div>Loss: <span style="font-weight:bold">0.18371</span>= <span style="font-family:monaco">0.020 + 0.000 + 0.060 + 0.012 + 0.056 + 0.015 + 0.000 + 0.021</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.064</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.099</span> + <span style="color:blue">+0.052</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.056</span> = <span style="color:red; text-decoration:underline">-0.106</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.059</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.039</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.030</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.100</span> = <span style="color:blue; text-decoration:underline">+0.185</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.062</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.084</span> + <span style="color:red">-0.090</span> + <span style="color:blue">+0.059</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.051</span> = <span style="color:red; text-decoration:underline">-0.090</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.062</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.049</span> + <span style="color:red">-0.012</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.113</span> = <span style="color:blue; text-decoration:underline">+0.119</span></div>


    w1=+0.495, w2=+0.118, w3=+1.000, w4=+0.492, w5=+0.133
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.597, +0.506, +0.566, +0.659, +0.572, +0.587, +0.496, +0.405]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.392, +0.024, +0.268, +0.659, +0.292, +0.350, -0.018, -0.386]
    Input : [+0.626, -0.588, +0.495, +0.626, -0.588, +0.118, -0.588, -0.588]
    Gate  : [+0.625, +0.625, +0.492, +0.625, +0.625, +0.492, +0.625, +0.625]



<div>Loss: <span style="font-weight:bold">0.17703</span>= <span style="font-family:monaco">0.019 + 0.000 + 0.058 + 0.012 + 0.054 + 0.015 + 0.000 + 0.018</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.063</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.087</span> + <span style="color:red">-0.099</span> + <span style="color:blue">+0.054</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.052</span> = <span style="color:red; text-decoration:underline">-0.108</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.058</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.039</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.031</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.094</span> = <span style="color:blue; text-decoration:underline">+0.178</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.083</span> + <span style="color:red">-0.091</span> + <span style="color:blue">+0.060</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.046</span> = <span style="color:red; text-decoration:underline">-0.096</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.048</span> + <span style="color:red">-0.012</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.105</span> = <span style="color:blue; text-decoration:underline">+0.108</span></div>


    w1=+0.505, w2=+0.100, w3=+1.000, w4=+0.502, w5=+0.122
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.596, +0.506, +0.569, +0.661, +0.576, +0.588, +0.499, +0.409]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.387, +0.026, +0.280, +0.667, +0.306, +0.356, -0.006, -0.367]
    Input : [+0.621, -0.579, +0.505, +0.621, -0.579, +0.100, -0.579, -0.579]
    Gate  : [+0.624, +0.624, +0.502, +0.624, +0.624, +0.502, +0.624, +0.624]



<div>Loss: <span style="font-weight:bold">0.17071</span>= <span style="font-family:monaco">0.019 + 0.000 + 0.056 + 0.011 + 0.052 + 0.016 + 0.000 + 0.017</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.062</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.098</span> + <span style="color:blue">+0.056</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.048</span> = <span style="color:red; text-decoration:underline">-0.109</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.058</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.038</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.032</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.088</span> = <span style="color:blue; text-decoration:underline">+0.172</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.059</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.091</span> + <span style="color:blue">+0.061</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.043</span> = <span style="color:red; text-decoration:underline">-0.099</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.059</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.047</span> + <span style="color:red">-0.013</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.097</span> = <span style="color:blue; text-decoration:underline">+0.099</span></div>


    w1=+0.516, w2=+0.083, w3=+1.000, w4=+0.512, w5=+0.113
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.595, +0.507, +0.572, +0.663, +0.579, +0.590, +0.502, +0.413]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.384, +0.028, +0.292, +0.676, +0.320, +0.362, +0.006, -0.350]
    Input : [+0.615, -0.571, +0.516, +0.615, -0.571, +0.083, -0.571, -0.571]
    Gate  : [+0.624, +0.624, +0.512, +0.624, +0.624, +0.512, +0.624, +0.624]



<div>Loss: <span style="font-weight:bold">0.16468</span>= <span style="font-family:monaco">0.018 + 0.000 + 0.054 + 0.011 + 0.050 + 0.016 + 0.000 + 0.015</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.091</span> + <span style="color:red">-0.083</span> + <span style="color:red">-0.097</span> + <span style="color:blue">+0.057</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.045</span> = <span style="color:red; text-decoration:underline">-0.107</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.057</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.037</span> + <span style="color:blue">+0.020</span> + <span style="color:blue">+0.034</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.083</span> = <span style="color:blue; text-decoration:underline">+0.168</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.089</span> + <span style="color:red">-0.080</span> + <span style="color:red">-0.092</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.039</span> = <span style="color:red; text-decoration:underline">-0.101</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.045</span> + <span style="color:red">-0.014</span> + <span style="color:blue">+0.008</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.091</span> = <span style="color:blue; text-decoration:underline">+0.091</span></div>


    w1=+0.527, w2=+0.066, w3=+1.000, w4=+0.522, w5=+0.103
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.594, +0.507, +0.576, +0.665, +0.583, +0.591, +0.504, +0.417]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.381, +0.030, +0.305, +0.686, +0.334, +0.369, +0.017, -0.335]
    Input : [+0.610, -0.563, +0.527, +0.610, -0.563, +0.066, -0.563, -0.563]
    Gate  : [+0.625, +0.625, +0.522, +0.625, +0.625, +0.522, +0.625, +0.625]



<div>Loss: <span style="font-weight:bold">0.15892</span>= <span style="font-family:monaco">0.018 + 0.000 + 0.052 + 0.010 + 0.048 + 0.017 + 0.000 + 0.014</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.061</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.091</span> + <span style="color:red">-0.081</span> + <span style="color:red">-0.097</span> + <span style="color:blue">+0.059</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.043</span> = <span style="color:red; text-decoration:underline">-0.105</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.057</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.035</span> + <span style="color:blue">+0.020</span> + <span style="color:blue">+0.035</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.079</span> = <span style="color:blue; text-decoration:underline">+0.164</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.089</span> + <span style="color:red">-0.078</span> + <span style="color:red">-0.092</span> + <span style="color:blue">+0.063</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.036</span> = <span style="color:red; text-decoration:underline">-0.102</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.058</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.043</span> + <span style="color:red">-0.014</span> + <span style="color:blue">+0.009</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.085</span> = <span style="color:blue; text-decoration:underline">+0.085</span></div>


    w1=+0.538, w2=+0.050, w3=+1.000, w4=+0.532, w5=+0.095
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.565, +0.522, +0.652, +0.708, +0.671, +0.601, +0.559, +0.516]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.260, +0.087, +0.628, +0.887, +0.715, +0.408, +0.236, +0.063]
    Input : [+0.336, -0.224, +0.687, +0.336, -0.224, -0.389, -0.224, -0.224]
    Gate  : [+0.772, +0.772, +0.787, +0.772, +0.772, +0.787, +0.772, +0.772]



<div>Loss: <span style="font-weight:bold">0.06145</span>= <span style="font-family:monaco">0.008 + 0.001 + 0.014 + 0.001 + 0.008 + 0.021 + 0.007 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.052</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.069</span> + <span style="color:red">-0.038</span> + <span style="color:red">-0.056</span> + <span style="color:blue">+0.095</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.008</span> = <span style="color:red; text-decoration:underline">-0.009</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.048</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.015</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.063</span> + <span style="color:red">-0.012</span> + <span style="color:red">-0.016</span> = <span style="color:blue; text-decoration:underline">+0.082</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.022</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.063</span> + <span style="color:red">-0.026</span> + <span style="color:red">-0.054</span> + <span style="color:blue">+0.053</span> + <span style="color:blue">+0.018</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.048</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.022</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.009</span> + <span style="color:red">-0.010</span> + <span style="color:red">-0.013</span> + <span style="color:blue">+0.023</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> = <span style="color:blue; text-decoration:underline">+0.011</span></div>


    w1=+0.688, w2=-0.398, w3=+1.000, w4=+0.792, w5=-0.016
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.536, +0.526, +0.678, +0.708, +0.700, +0.583, +0.573, +0.563]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.144, +0.103, +0.744, +0.888, +0.847, +0.333, +0.293, +0.252]
    Input : [+0.188, -0.053, +0.712, +0.188, -0.053, -0.571, -0.053, -0.053]
    Gate  : [+0.762, +0.762, +0.900, +0.762, +0.762, +0.900, +0.762, +0.762]



<div>Loss: <span style="font-weight:bold">0.04661</span>= <span style="font-family:monaco">0.003 + 0.001 + 0.007 + 0.001 + 0.002 + 0.014 + 0.011 + 0.008</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.028</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.052</span> + <span style="color:red">-0.040</span> + <span style="color:red">-0.033</span> + <span style="color:blue">+0.087</span> + <span style="color:blue">+0.025</span> + <span style="color:red">-0.023</span> = <span style="color:red; text-decoration:underline">-0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.026</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:red">-0.015</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.061</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.056</span> = <span style="color:blue; text-decoration:underline">+0.018</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.045</span> + <span style="color:red">-0.023</span> + <span style="color:red">-0.031</span> + <span style="color:blue">+0.034</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.019</span> = <span style="color:red; text-decoration:underline">-0.010</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.007</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.016</span> + <span style="color:blue">+0.010</span> = <span style="color:blue; text-decoration:underline">+0.036</span></div>


    w1=+0.713, w2=-0.572, w3=+1.000, w4=+0.901, w5=-0.141
    



<div style="font-weight:bold">Iteration 149</div>


    Output: [+0.518, +0.522, +0.688, +0.703, +0.707, +0.560, +0.565, +0.569]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.070, +0.089, +0.792, +0.862, +0.881, +0.241, +0.260, +0.279]
    Input : [+0.118, +0.032, +0.744, +0.118, +0.032, -0.678, +0.032, +0.032]
    Gate  : [+0.596, +0.596, +0.943, +0.596, +0.596, +0.943, +0.596, +0.596]



<div>Loss: <span style="font-weight:bold">0.03474</span>= <span style="font-family:monaco">0.001 + 0.001 + 0.004 + 0.002 + 0.001 + 0.007 + 0.008 + 0.010</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.011</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.043</span> + <span style="color:red">-0.046</span> + <span style="color:red">-0.026</span> + <span style="color:blue">+0.064</span> + <span style="color:blue">+0.033</span> + <span style="color:red">-0.003</span> = <span style="color:red; text-decoration:underline">-0.008</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.010</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.014</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.049</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.032</span> = <span style="color:blue; text-decoration:underline">+0.028</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.038</span> + <span style="color:red">-0.028</span> + <span style="color:red">-0.025</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.030</span> = <span style="color:red; text-decoration:underline">-0.009</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.006</span> + <span style="color:red">-0.008</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.018</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.025</span> = <span style="color:blue; text-decoration:underline">+0.049</span></div>


    w1=+0.745, w2=-0.681, w3=+1.000, w4=+0.944, w5=-0.352
    



<div style="font-weight:bold">Iteration 199</div>


    Output: [+0.501, +0.517, +0.703, +0.704, +0.717, +0.521, +0.537, +0.553]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [+0.004, +0.067, +0.863, +0.866, +0.930, +0.084, +0.147, +0.211]
    Input : [+0.009, +0.164, +0.800, +0.009, +0.164, -0.850, +0.164, +0.164]
    Gate  : [+0.385, +0.385, +0.995, +0.385, +0.385, +0.995, +0.385, +0.385]



<div>Loss: <span style="font-weight:bold">0.01388</span>= <span style="font-family:monaco">0.000 + 0.001 + 0.002 + 0.002 + 0.000 + 0.001 + 0.003 + 0.006</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.029</span> + <span style="color:red">-0.039</span> + <span style="color:red">-0.015</span> + <span style="color:blue">+0.023</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.019</span> = <span style="color:red; text-decoration:underline">-0.014</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.019</span> + <span style="color:blue">+0.018</span> + <span style="color:blue">+0.005</span> = <span style="color:blue; text-decoration:underline">+0.035</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.027</span> + <span style="color:red">-0.027</span> + <span style="color:red">-0.016</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.017</span> + <span style="color:blue">+0.033</span> = <span style="color:red; text-decoration:underline">-0.011</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.005</span> + <span style="color:red">-0.005</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.019</span> + <span style="color:blue">+0.036</span> = <span style="color:blue; text-decoration:underline">+0.050</span></div>


    w1=+0.802, w2=-0.854, w3=+1.000, w4=+0.996, w5=-0.614
    



<div style="font-weight:bold">Iteration 249</div>


    Output: [+0.498, +0.512, +0.720, +0.718, +0.729, +0.497, +0.510, +0.524]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.500, +0.500, +0.500]
    Memory: [-0.009, +0.047, +0.943, +0.934, +0.990, -0.014, +0.042, +0.097]
    Input : [-0.037, +0.231, +0.868, -0.037, +0.231, -0.972, +0.231, +0.231]
    Gate  : [+0.240, +0.240, +1.033, +0.240, +0.240, +1.033, +0.240, +0.240]



<div>Loss: <span style="font-weight:bold">0.00246</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.012</span> + <span style="color:red">-0.017</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.015</span> = <span style="color:red; text-decoration:underline">-0.011</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.001</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.011</span> = <span style="color:blue; text-decoration:underline">+0.012</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.012</span> + <span style="color:red">-0.013</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.018</span> = <span style="color:red; text-decoration:underline">-0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.021</span> = <span style="color:blue; text-decoration:underline">+0.023</span></div>


    w1=+0.869, w2=-0.973, w3=+1.000, w4=+1.033, w5=-0.795
    





    (0.8691986382286676,
     -0.9728195862949829,
     1.0,
     1.0330824913276644,
     -0.7952812244235943)



Now we see that the input gate is closer to the true gate: it tries to ignore irrelevant input by setting the weights of those input closer to 0. Although in this case it is still far from the true gate (the irrelevant input still gets positive score), we see that it has good impact on the loss, reaching an order of magnitude lower. And actually if we run more iterations, we see later that the gate will be learned correctly ($w_4 = 1.0, w_5=-1.0$).

Notice that in the network without input gate, at the end the overall gradient is zero, but actually the gradient at each position in the sequence is not zero, and in fact the magnitude is not quite small, meaning the network ends up at a non-optimal position, while in the gated version, we see the gradient approaches zero in all position.


```python
# Trying nested brackets
experiment_gated('ab(aaa(bab)b)')
```

    w1=+1.000, w2=+1.000, w3=+1.000, w4=+1.000, w5=+1.000



<div style="font-weight:bold">Iteration 0</div>


    Output: [+0.982, +0.498, +0.730, +0.993, +1.000, +1.000, +1.000, +1.000, +1.000, +1.000, +1.000, +0.999, +1.000]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+4.003, -0.007, +0.993, +4.996, +8.998, +13.001, +14.001, +9.991, +13.994, +9.984, +10.984, +6.975, +7.975]
    Input : [+2.001, -2.005, +1.000, +2.001, +2.001, +2.001, +1.000, -2.005, +2.001, -2.005, +1.000, -2.005, +1.000]
    Gate  : [+2.000, +2.000, +1.000, +2.000, +2.000, +2.000, +1.000, +2.000, +2.000, +2.000, +1.000, +2.000, +1.000]



<div>Loss: <span style="font-weight:bold">18.06454</span>= <span style="font-family:monaco">1.326 + 0.000 + 0.000 + 0.768 + 1.838 + 2.914 + 1.304 + 0.826 + 1.303 + 0.825 + 2.372 + 1.295 + 3.295</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+1.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.860</span> + <span style="color:blue">+1.439</span> + <span style="color:blue">+1.998</span> + <span style="color:blue">+1.005</span> + <span style="color:blue">+0.782</span> + <span style="color:blue">+1.029</span> + <span style="color:blue">+0.807</span> + <span style="color:blue">+1.820</span> + <span style="color:blue">+1.314</span> + <span style="color:blue">+2.449</span> = <span style="color:blue; text-decoration:underline">+14.501</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.930</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.450</span> + <span style="color:blue">+0.980</span> + <span style="color:blue">+1.499</span> + <span style="color:blue">+0.664</span> + <span style="color:blue">+0.409</span> + <span style="color:blue">+0.639</span> + <span style="color:blue">+0.383</span> + <span style="color:blue">+1.134</span> + <span style="color:blue">+0.555</span> + <span style="color:blue">+1.535</span> = <span style="color:blue; text-decoration:underline">+9.178</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:blue">+0.965</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.786</span> + <span style="color:blue">+1.344</span> + <span style="color:blue">+1.883</span> + <span style="color:blue">+0.954</span> + <span style="color:blue">+0.714</span> + <span style="color:blue">+0.953</span> + <span style="color:blue">+0.714</span> + <span style="color:blue">+1.880</span> + <span style="color:blue">+1.337</span> + <span style="color:blue">+2.992</span> = <span style="color:blue; text-decoration:underline">+14.520</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:blue">+0.965</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.524</span> + <span style="color:blue">+1.075</span> + <span style="color:blue">+1.614</span> + <span style="color:blue">+0.715</span> + <span style="color:blue">+0.476</span> + <span style="color:blue">+0.715</span> + <span style="color:blue">+0.476</span> + <span style="color:blue">+1.074</span> + <span style="color:blue">+0.533</span> + <span style="color:blue">+0.993</span> = <span style="color:blue; text-decoration:underline">+9.159</span></div>


    w1=-0.450, w2=+0.082, w3=+1.000, w4=-0.452, w5=+0.084
    



<div style="font-weight:bold">Iteration 1</div>


    Output: [+0.536, +0.505, +0.556, +0.591, +0.625, +0.657, +0.702, +0.675, +0.706, +0.680, +0.672, +0.644, +0.636]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.142, +0.020, +0.224, +0.366, +0.509, +0.651, +0.855, +0.733, +0.875, +0.753, +0.716, +0.594, +0.557]
    Input : [-0.387, +0.332, -0.450, -0.387, -0.387, -0.387, -0.450, +0.332, -0.387, +0.332, +0.082, +0.332, +0.082]
    Gate  : [-0.368, -0.368, -0.452, -0.368, -0.368, -0.368, -0.452, -0.368, -0.368, -0.368, -0.452, -0.368, -0.452]



<div>Loss: <span style="font-weight:bold">0.61597</span>= <span style="font-family:monaco">0.003 + 0.000 + 0.065 + 0.043 + 0.025 + 0.013 + 0.091 + 0.114 + 0.087 + 0.110 + 0.008 + 0.017 + 0.038</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.014</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.086</span> + <span style="color:blue">+0.122</span> + <span style="color:blue">+0.133</span> + <span style="color:blue">+0.121</span> + <span style="color:blue">+0.374</span> + <span style="color:blue">+0.358</span> + <span style="color:blue">+0.372</span> + <span style="color:blue">+0.358</span> + <span style="color:blue">+0.106</span> + <span style="color:blue">+0.125</span> + <span style="color:red">-0.195</span> = <span style="color:blue; text-decoration:underline">+1.946</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.013</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.044</span> + <span style="color:blue">+0.071</span> + <span style="color:blue">+0.076</span> + <span style="color:blue">+0.184</span> + <span style="color:blue">+0.130</span> + <span style="color:blue">+0.172</span> + <span style="color:blue">+0.119</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.056</span> + <span style="color:red">-0.149</span> = <span style="color:blue; text-decoration:underline">+0.746</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.014</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.125</span> + <span style="color:blue">+0.136</span> + <span style="color:blue">+0.123</span> + <span style="color:blue">+0.380</span> + <span style="color:blue">+0.367</span> + <span style="color:blue">+0.380</span> + <span style="color:blue">+0.370</span> + <span style="color:blue">+0.104</span> + <span style="color:blue">+0.124</span> + <span style="color:red">-0.182</span> = <span style="color:blue; text-decoration:underline">+2.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.014</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.088</span> + <span style="color:blue">+0.090</span> + <span style="color:blue">+0.218</span> + <span style="color:blue">+0.182</span> + <span style="color:blue">+0.223</span> + <span style="color:blue">+0.189</span> + <span style="color:blue">+0.056</span> + <span style="color:blue">+0.053</span> + <span style="color:red">-0.083</span> = <span style="color:blue; text-decoration:underline">+1.074</span></div>


    w1=-0.645, w2=+0.008, w3=+1.000, w4=-0.652, w5=-0.023
    



<div style="font-weight:bold">Iteration 2</div>


    Output: [+0.610, +0.511, +0.614, +0.714, +0.796, +0.859, +0.903, +0.861, +0.906, +0.867, +0.866, +0.812, +0.812]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.446, +0.046, +0.466, +0.913, +1.359, +1.806, +2.226, +1.825, +2.272, +1.871, +1.866, +1.465, +1.460]
    Input : [-0.661, +0.593, -0.645, -0.661, -0.661, -0.661, -0.645, +0.593, -0.661, +0.593, +0.008, +0.593, +0.008]
    Gate  : [-0.675, -0.675, -0.652, -0.675, -0.675, -0.675, -0.652, -0.675, -0.675, -0.675, -0.652, -0.675, -0.652]



<div>Loss: <span style="font-weight:bold">0.46103</span>= <span style="font-family:monaco">0.025 + 0.000 + 0.030 + 0.001 + 0.012 + 0.056 + 0.003 + 0.002 + 0.004 + 0.001 + 0.063 + 0.020 + 0.246</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.077</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.084</span> + <span style="color:blue">+0.025</span> + <span style="color:red">-0.137</span> + <span style="color:red">-0.361</span> + <span style="color:red">-0.076</span> + <span style="color:blue">+0.056</span> + <span style="color:red">-0.091</span> + <span style="color:blue">+0.042</span> + <span style="color:red">-0.393</span> + <span style="color:red">-0.186</span> + <span style="color:red">-0.712</span> = <span style="color:red; text-decoration:underline">-1.826</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.072</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.079</span> + <span style="color:red">-0.240</span> + <span style="color:red">-0.041</span> + <span style="color:blue">+0.023</span> + <span style="color:red">-0.047</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.235</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.519</span> = <span style="color:red; text-decoration:underline">-1.274</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.073</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.083</span> + <span style="color:blue">+0.024</span> + <span style="color:red">-0.131</span> + <span style="color:red">-0.344</span> + <span style="color:red">-0.073</span> + <span style="color:blue">+0.054</span> + <span style="color:red">-0.088</span> + <span style="color:blue">+0.040</span> + <span style="color:red">-0.379</span> + <span style="color:red">-0.180</span> + <span style="color:red">-0.687</span> = <span style="color:red; text-decoration:underline">-1.754</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.073</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.090</span> + <span style="color:red">-0.262</span> + <span style="color:red">-0.045</span> + <span style="color:blue">+0.029</span> + <span style="color:red">-0.054</span> + <span style="color:blue">+0.022</span> + <span style="color:red">-0.206</span> + <span style="color:red">-0.076</span> + <span style="color:red">-0.290</span> = <span style="color:red; text-decoration:underline">-1.025</span></div>


    w1=-0.462, w2=+0.135, w3=+1.000, w4=-0.477, w5=+0.079
    



<div style="font-weight:bold">Iteration 3</div>


    Output: [+0.535, +0.506, +0.561, +0.595, +0.628, +0.660, +0.707, +0.683, +0.712, +0.688, +0.674, +0.649, +0.634]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.139, +0.025, +0.245, +0.384, +0.522, +0.661, +0.881, +0.768, +0.906, +0.792, +0.728, +0.614, +0.550]
    Input : [-0.349, +0.286, -0.462, -0.349, -0.349, -0.349, -0.462, +0.286, -0.349, +0.286, +0.135, +0.286, +0.135]
    Gate  : [-0.398, -0.398, -0.477, -0.398, -0.398, -0.398, -0.477, -0.398, -0.398, -0.398, -0.477, -0.398, -0.477]



<div>Loss: <span style="font-weight:bold">0.57942</span>= <span style="font-family:monaco">0.002 + 0.000 + 0.062 + 0.041 + 0.024 + 0.012 + 0.086 + 0.107 + 0.082 + 0.103 + 0.008 + 0.015 + 0.037</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.014</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.088</span> + <span style="color:blue">+0.127</span> + <span style="color:blue">+0.139</span> + <span style="color:blue">+0.126</span> + <span style="color:blue">+0.388</span> + <span style="color:blue">+0.368</span> + <span style="color:blue">+0.383</span> + <span style="color:blue">+0.366</span> + <span style="color:blue">+0.108</span> + <span style="color:blue">+0.126</span> + <span style="color:red">-0.205</span> = <span style="color:blue; text-decoration:underline">+1.998</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.013</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.046</span> + <span style="color:blue">+0.075</span> + <span style="color:blue">+0.079</span> + <span style="color:blue">+0.192</span> + <span style="color:blue">+0.135</span> + <span style="color:blue">+0.180</span> + <span style="color:blue">+0.123</span> + <span style="color:blue">+0.063</span> + <span style="color:blue">+0.057</span> + <span style="color:red">-0.156</span> = <span style="color:blue; text-decoration:underline">+0.774</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.012</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.119</span> + <span style="color:blue">+0.126</span> + <span style="color:blue">+0.112</span> + <span style="color:blue">+0.353</span> + <span style="color:blue">+0.345</span> + <span style="color:blue">+0.353</span> + <span style="color:blue">+0.348</span> + <span style="color:blue">+0.095</span> + <span style="color:blue">+0.114</span> + <span style="color:red">-0.168</span> = <span style="color:blue; text-decoration:underline">+1.875</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.012</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.056</span> + <span style="color:blue">+0.079</span> + <span style="color:blue">+0.079</span> + <span style="color:blue">+0.193</span> + <span style="color:blue">+0.163</span> + <span style="color:blue">+0.197</span> + <span style="color:blue">+0.170</span> + <span style="color:blue">+0.050</span> + <span style="color:blue">+0.049</span> + <span style="color:red">-0.080</span> = <span style="color:blue; text-decoration:underline">+0.954</span></div>


    w1=-0.662, w2=+0.058, w3=+1.000, w4=-0.664, w5=-0.016
    



<div style="font-weight:bold">Iteration 4</div>


    Output: [+0.606, +0.513, +0.620, +0.715, +0.794, +0.855, +0.902, +0.863, +0.906, +0.869, +0.864, +0.814, +0.808]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.429, +0.051, +0.491, +0.920, +1.349, +1.778, +2.218, +1.839, +2.268, +1.890, +1.852, +1.474, +1.435]
    Input : [-0.631, +0.556, -0.662, -0.631, -0.631, -0.631, -0.662, +0.556, -0.631, +0.556, +0.058, +0.556, +0.058]
    Gate  : [-0.681, -0.681, -0.664, -0.681, -0.681, -0.681, -0.664, -0.681, -0.681, -0.681, -0.664, -0.681, -0.664]



<div>Loss: <span style="font-weight:bold">0.44256</span>= <span style="font-family:monaco">0.023 + 0.000 + 0.027 + 0.001 + 0.011 + 0.052 + 0.002 + 0.001 + 0.004 + 0.001 + 0.062 + 0.020 + 0.238</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.075</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.081</span> + <span style="color:blue">+0.023</span> + <span style="color:red">-0.135</span> + <span style="color:red">-0.355</span> + <span style="color:red">-0.074</span> + <span style="color:blue">+0.052</span> + <span style="color:red">-0.091</span> + <span style="color:blue">+0.035</span> + <span style="color:red">-0.393</span> + <span style="color:red">-0.191</span> + <span style="color:red">-0.713</span> = <span style="color:red; text-decoration:underline">-1.836</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.069</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.009</span> + <span style="color:red">-0.078</span> + <span style="color:red">-0.236</span> + <span style="color:red">-0.040</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.046</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.234</span> + <span style="color:red">-0.085</span> + <span style="color:red">-0.521</span> = <span style="color:red; text-decoration:underline">-1.274</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.067</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.082</span> + <span style="color:blue">+0.022</span> + <span style="color:red">-0.126</span> + <span style="color:red">-0.327</span> + <span style="color:red">-0.069</span> + <span style="color:blue">+0.049</span> + <span style="color:red">-0.086</span> + <span style="color:blue">+0.034</span> + <span style="color:red">-0.367</span> + <span style="color:red">-0.181</span> + <span style="color:red">-0.658</span> = <span style="color:red; text-decoration:underline">-1.694</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.067</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.084</span> + <span style="color:red">-0.245</span> + <span style="color:red">-0.041</span> + <span style="color:blue">+0.025</span> + <span style="color:red">-0.052</span> + <span style="color:blue">+0.018</span> + <span style="color:red">-0.198</span> + <span style="color:red">-0.077</span> + <span style="color:red">-0.286</span> = <span style="color:red; text-decoration:underline">-0.987</span></div>


    w1=-0.478, w2=+0.185, w3=+1.000, w4=-0.495, w5=+0.083
    



<div style="font-weight:bold">Iteration 5</div>


    Output: [+0.533, +0.507, +0.566, +0.598, +0.629, +0.659, +0.710, +0.688, +0.716, +0.694, +0.675, +0.652, +0.631]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.131, +0.029, +0.265, +0.396, +0.527, +0.658, +0.895, +0.792, +0.923, +0.821, +0.729, +0.627, +0.536]
    Input : [-0.317, +0.248, -0.478, -0.317, -0.317, -0.317, -0.478, +0.248, -0.317, +0.248, +0.185, +0.248, +0.185]
    Gate  : [-0.412, -0.412, -0.495, -0.412, -0.412, -0.412, -0.495, -0.412, -0.412, -0.412, -0.495, -0.412, -0.495]



<div>Loss: <span style="font-weight:bold">0.55541</span>= <span style="font-family:monaco">0.002 + 0.000 + 0.058 + 0.039 + 0.023 + 0.012 + 0.084 + 0.103 + 0.079 + 0.097 + 0.007 + 0.014 + 0.035</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.014</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">+0.142</span> + <span style="color:blue">+0.132</span> + <span style="color:blue">+0.396</span> + <span style="color:blue">+0.372</span> + <span style="color:blue">+0.389</span> + <span style="color:blue">+0.368</span> + <span style="color:blue">+0.111</span> + <span style="color:blue">+0.126</span> + <span style="color:red">-0.208</span> = <span style="color:blue; text-decoration:underline">+2.031</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.013</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.047</span> + <span style="color:blue">+0.077</span> + <span style="color:blue">+0.083</span> + <span style="color:blue">+0.196</span> + <span style="color:blue">+0.136</span> + <span style="color:blue">+0.182</span> + <span style="color:blue">+0.124</span> + <span style="color:blue">+0.065</span> + <span style="color:blue">+0.057</span> + <span style="color:red">-0.158</span> = <span style="color:blue; text-decoration:underline">+0.789</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.010</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.090</span> + <span style="color:blue">+0.115</span> + <span style="color:blue">+0.121</span> + <span style="color:blue">+0.108</span> + <span style="color:blue">+0.338</span> + <span style="color:blue">+0.333</span> + <span style="color:blue">+0.338</span> + <span style="color:blue">+0.335</span> + <span style="color:blue">+0.091</span> + <span style="color:blue">+0.108</span> + <span style="color:red">-0.154</span> = <span style="color:blue; text-decoration:underline">+1.813</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.010</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.052</span> + <span style="color:blue">+0.072</span> + <span style="color:blue">+0.074</span> + <span style="color:blue">+0.175</span> + <span style="color:blue">+0.149</span> + <span style="color:blue">+0.180</span> + <span style="color:blue">+0.157</span> + <span style="color:blue">+0.048</span> + <span style="color:blue">+0.047</span> + <span style="color:red">-0.078</span> = <span style="color:blue; text-decoration:underline">+0.875</span></div>


    w1=-0.681, w2=+0.106, w3=+1.000, w4=-0.676, w5=-0.005
    



<div style="font-weight:bold">Iteration 6</div>


    Output: [+0.601, +0.514, +0.626, +0.717, +0.792, +0.852, +0.901, +0.865, +0.906, +0.871, +0.863, +0.815, +0.804]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.412, +0.056, +0.517, +0.928, +1.340, +1.751, +2.212, +1.856, +2.268, +1.912, +1.840, +1.485, +1.413]
    Input : [-0.604, +0.522, -0.681, -0.604, -0.604, -0.604, -0.681, +0.522, -0.604, +0.522, +0.106, +0.522, +0.106]
    Gate  : [-0.681, -0.681, -0.676, -0.681, -0.681, -0.681, -0.676, -0.681, -0.681, -0.681, -0.676, -0.681, -0.676]



<div>Loss: <span style="font-weight:bold">0.42587</span>= <span style="font-family:monaco">0.021 + 0.000 + 0.025 + 0.001 + 0.011 + 0.049 + 0.002 + 0.001 + 0.004 + 0.000 + 0.060 + 0.021 + 0.231</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.072</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.078</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.133</span> + <span style="color:red">-0.347</span> + <span style="color:red">-0.073</span> + <span style="color:blue">+0.046</span> + <span style="color:red">-0.092</span> + <span style="color:blue">+0.028</span> + <span style="color:red">-0.393</span> + <span style="color:red">-0.197</span> + <span style="color:red">-0.712</span> = <span style="color:red; text-decoration:underline">-1.845</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.067</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.008</span> + <span style="color:red">-0.076</span> + <span style="color:red">-0.230</span> + <span style="color:red">-0.039</span> + <span style="color:blue">+0.019</span> + <span style="color:red">-0.046</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.234</span> + <span style="color:red">-0.088</span> + <span style="color:red">-0.523</span> = <span style="color:red; text-decoration:underline">-1.271</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.061</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.080</span> + <span style="color:blue">+0.020</span> + <span style="color:red">-0.121</span> + <span style="color:red">-0.312</span> + <span style="color:red">-0.067</span> + <span style="color:blue">+0.044</span> + <span style="color:red">-0.085</span> + <span style="color:blue">+0.027</span> + <span style="color:red">-0.358</span> + <span style="color:red">-0.184</span> + <span style="color:red">-0.633</span> = <span style="color:red; text-decoration:underline">-1.652</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.061</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.079</span> + <span style="color:red">-0.229</span> + <span style="color:red">-0.039</span> + <span style="color:blue">+0.022</span> + <span style="color:red">-0.050</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.192</span> + <span style="color:red">-0.078</span> + <span style="color:red">-0.283</span> = <span style="color:red; text-decoration:underline">-0.959</span></div>


    w1=-0.497, w2=+0.233, w3=+1.000, w4=-0.511, w5=+0.091
    



<div style="font-weight:bold">Iteration 7</div>


    Output: [+0.530, +0.508, +0.571, +0.601, +0.629, +0.657, +0.712, +0.693, +0.719, +0.700, +0.675, +0.655, +0.627]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.122, +0.032, +0.286, +0.408, +0.530, +0.652, +0.906, +0.816, +0.938, +0.848, +0.729, +0.639, +0.520]
    Input : [-0.290, +0.214, -0.497, -0.290, -0.290, -0.290, -0.497, +0.214, -0.290, +0.214, +0.233, +0.214, +0.233]
    Gate  : [-0.420, -0.420, -0.511, -0.420, -0.420, -0.420, -0.511, -0.420, -0.420, -0.420, -0.511, -0.420, -0.511]



<div>Loss: <span style="font-weight:bold">0.53373</span>= <span style="font-family:monaco">0.002 + 0.000 + 0.055 + 0.037 + 0.023 + 0.013 + 0.082 + 0.098 + 0.077 + 0.092 + 0.008 + 0.013 + 0.033</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.013</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">+0.145</span> + <span style="color:blue">+0.137</span> + <span style="color:blue">+0.400</span> + <span style="color:blue">+0.371</span> + <span style="color:blue">+0.392</span> + <span style="color:blue">+0.366</span> + <span style="color:blue">+0.114</span> + <span style="color:blue">+0.125</span> + <span style="color:red">-0.207</span> = <span style="color:blue; text-decoration:underline">+2.046</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.012</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.047</span> + <span style="color:blue">+0.078</span> + <span style="color:blue">+0.086</span> + <span style="color:blue">+0.197</span> + <span style="color:blue">+0.135</span> + <span style="color:blue">+0.183</span> + <span style="color:blue">+0.122</span> + <span style="color:blue">+0.067</span> + <span style="color:blue">+0.056</span> + <span style="color:red">-0.159</span> = <span style="color:blue; text-decoration:underline">+0.794</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.009</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.092</span> + <span style="color:blue">+0.113</span> + <span style="color:blue">+0.117</span> + <span style="color:blue">+0.106</span> + <span style="color:blue">+0.327</span> + <span style="color:blue">+0.324</span> + <span style="color:blue">+0.327</span> + <span style="color:blue">+0.326</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.104</span> + <span style="color:red">-0.143</span> = <span style="color:blue; text-decoration:underline">+1.772</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.009</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.012</span> + <span style="color:blue">+0.048</span> + <span style="color:blue">+0.067</span> + <span style="color:blue">+0.070</span> + <span style="color:blue">+0.160</span> + <span style="color:blue">+0.137</span> + <span style="color:blue">+0.166</span> + <span style="color:blue">+0.146</span> + <span style="color:blue">+0.046</span> + <span style="color:blue">+0.046</span> + <span style="color:red">-0.076</span> = <span style="color:blue; text-decoration:underline">+0.812</span></div>


    w1=-0.702, w2=+0.154, w3=+1.000, w4=-0.688, w5=+0.010
    



<div style="font-weight:bold">Iteration 8</div>


    Output: [+0.597, +0.515, +0.633, +0.718, +0.791, +0.848, +0.901, +0.867, +0.906, +0.874, +0.861, +0.817, +0.800]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.393, +0.060, +0.543, +0.936, +1.329, +1.722, +2.205, +1.872, +2.265, +1.933, +1.827, +1.495, +1.389]
    Input : [-0.579, +0.490, -0.702, -0.579, -0.579, -0.579, -0.702, +0.490, -0.579, +0.490, +0.154, +0.490, +0.154]
    Gate  : [-0.679, -0.679, -0.688, -0.679, -0.679, -0.679, -0.688, -0.679, -0.679, -0.679, -0.688, -0.679, -0.688]



<div>Loss: <span style="font-weight:bold">0.40831</span>= <span style="font-family:monaco">0.019 + 0.000 + 0.022 + 0.000 + 0.010 + 0.045 + 0.002 + 0.001 + 0.003 + 0.000 + 0.058 + 0.022 + 0.224</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.068</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.075</span> + <span style="color:blue">+0.019</span> + <span style="color:red">-0.129</span> + <span style="color:red">-0.337</span> + <span style="color:red">-0.071</span> + <span style="color:blue">+0.041</span> + <span style="color:red">-0.091</span> + <span style="color:blue">+0.022</span> + <span style="color:red">-0.390</span> + <span style="color:red">-0.202</span> + <span style="color:red">-0.709</span> = <span style="color:red; text-decoration:underline">-1.843</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.063</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.074</span> + <span style="color:red">-0.222</span> + <span style="color:red">-0.038</span> + <span style="color:blue">+0.016</span> + <span style="color:red">-0.046</span> + <span style="color:blue">+0.008</span> + <span style="color:red">-0.232</span> + <span style="color:red">-0.090</span> + <span style="color:red">-0.523</span> = <span style="color:red; text-decoration:underline">-1.262</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.056</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.078</span> + <span style="color:blue">+0.017</span> + <span style="color:red">-0.116</span> + <span style="color:red">-0.296</span> + <span style="color:red">-0.064</span> + <span style="color:blue">+0.039</span> + <span style="color:red">-0.083</span> + <span style="color:blue">+0.020</span> + <span style="color:red">-0.349</span> + <span style="color:red">-0.187</span> + <span style="color:red">-0.610</span> = <span style="color:red; text-decoration:underline">-1.609</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.056</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.009</span> + <span style="color:red">-0.074</span> + <span style="color:red">-0.214</span> + <span style="color:red">-0.036</span> + <span style="color:blue">+0.019</span> + <span style="color:red">-0.048</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.186</span> + <span style="color:red">-0.080</span> + <span style="color:red">-0.281</span> = <span style="color:red; text-decoration:underline">-0.931</span></div>


    w1=-0.517, w2=+0.280, w3=+1.000, w4=-0.527, w5=+0.103
    



<div style="font-weight:bold">Iteration 9</div>


    Output: [+0.528, +0.509, +0.576, +0.604, +0.630, +0.656, +0.715, +0.699, +0.722, +0.706, +0.675, +0.658, +0.624]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.113, +0.035, +0.308, +0.421, +0.534, +0.647, +0.920, +0.843, +0.956, +0.878, +0.730, +0.653, +0.505]
    Input : [-0.266, +0.183, -0.517, -0.266, -0.266, -0.266, -0.517, +0.183, -0.266, +0.183, +0.280, +0.183, +0.280]
    Gate  : [-0.425, -0.425, -0.527, -0.425, -0.425, -0.425, -0.527, -0.425, -0.425, -0.425, -0.527, -0.425, -0.527]



<div>Loss: <span style="font-weight:bold">0.50985</span>= <span style="font-family:monaco">0.002 + 0.000 + 0.052 + 0.036 + 0.023 + 0.013 + 0.080 + 0.093 + 0.074 + 0.087 + 0.007 + 0.012 + 0.032</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.012</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.088</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">+0.146</span> + <span style="color:blue">+0.141</span> + <span style="color:blue">+0.401</span> + <span style="color:blue">+0.368</span> + <span style="color:blue">+0.391</span> + <span style="color:blue">+0.360</span> + <span style="color:blue">+0.116</span> + <span style="color:blue">+0.123</span> + <span style="color:red">-0.207</span> = <span style="color:blue; text-decoration:underline">+2.043</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.012</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.046</span> + <span style="color:blue">+0.078</span> + <span style="color:blue">+0.088</span> + <span style="color:blue">+0.196</span> + <span style="color:blue">+0.132</span> + <span style="color:blue">+0.180</span> + <span style="color:blue">+0.119</span> + <span style="color:blue">+0.068</span> + <span style="color:blue">+0.055</span> + <span style="color:red">-0.159</span> = <span style="color:blue; text-decoration:underline">+0.787</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.008</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.093</span> + <span style="color:blue">+0.110</span> + <span style="color:blue">+0.114</span> + <span style="color:blue">+0.104</span> + <span style="color:blue">+0.318</span> + <span style="color:blue">+0.315</span> + <span style="color:blue">+0.317</span> + <span style="color:blue">+0.317</span> + <span style="color:blue">+0.086</span> + <span style="color:blue">+0.099</span> + <span style="color:red">-0.133</span> = <span style="color:blue; text-decoration:underline">+1.734</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.008</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.013</span> + <span style="color:blue">+0.045</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.066</span> + <span style="color:blue">+0.146</span> + <span style="color:blue">+0.127</span> + <span style="color:blue">+0.153</span> + <span style="color:blue">+0.137</span> + <span style="color:blue">+0.044</span> + <span style="color:blue">+0.044</span> + <span style="color:red">-0.074</span> = <span style="color:blue; text-decoration:underline">+0.754</span></div>


    w1=-0.722, w2=+0.201, w3=+1.000, w4=-0.701, w5=+0.027
    



<div style="font-weight:bold">Iteration 10</div>


    Output: [+0.592, +0.516, +0.639, +0.720, +0.789, +0.844, +0.900, +0.868, +0.905, +0.876, +0.859, +0.818, +0.796]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.373, +0.065, +0.570, +0.943, +1.316, +1.689, +2.195, +1.887, +2.260, +1.952, +1.811, +1.502, +1.361]
    Input : [-0.554, +0.458, -0.722, -0.554, -0.554, -0.554, -0.722, +0.458, -0.554, +0.458, +0.201, +0.458, +0.201]
    Gate  : [-0.673, -0.673, -0.701, -0.673, -0.673, -0.673, -0.701, -0.673, -0.673, -0.673, -0.701, -0.673, -0.701]



<div>Loss: <span style="font-weight:bold">0.38915</span>= <span style="font-family:monaco">0.017 + 0.001 + 0.019 + 0.000 + 0.009 + 0.042 + 0.002 + 0.001 + 0.003 + 0.000 + 0.056 + 0.023 + 0.216</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.064</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.071</span> + <span style="color:blue">+0.017</span> + <span style="color:red">-0.125</span> + <span style="color:red">-0.324</span> + <span style="color:red">-0.068</span> + <span style="color:blue">+0.036</span> + <span style="color:red">-0.090</span> + <span style="color:blue">+0.016</span> + <span style="color:red">-0.386</span> + <span style="color:red">-0.207</span> + <span style="color:red">-0.704</span> = <span style="color:red; text-decoration:underline">-1.829</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.060</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.071</span> + <span style="color:red">-0.212</span> + <span style="color:red">-0.036</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.045</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.229</span> + <span style="color:red">-0.092</span> + <span style="color:red">-0.522</span> = <span style="color:red; text-decoration:underline">-1.245</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.051</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.075</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.111</span> + <span style="color:red">-0.280</span> + <span style="color:red">-0.061</span> + <span style="color:blue">+0.034</span> + <span style="color:red">-0.081</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.339</span> + <span style="color:red">-0.190</span> + <span style="color:red">-0.586</span> = <span style="color:red; text-decoration:underline">-1.561</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.051</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.069</span> + <span style="color:red">-0.199</span> + <span style="color:red">-0.033</span> + <span style="color:blue">+0.016</span> + <span style="color:red">-0.046</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.179</span> + <span style="color:red">-0.082</span> + <span style="color:red">-0.278</span> = <span style="color:red; text-decoration:underline">-0.899</span></div>


    w1=-0.539, w2=+0.326, w3=+1.000, w4=-0.545, w5=+0.117
    



<div style="font-weight:bold">Iteration 11</div>


    Output: [+0.526, +0.510, +0.582, +0.607, +0.632, +0.656, +0.719, +0.705, +0.727, +0.713, +0.676, +0.661, +0.620]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.104, +0.039, +0.332, +0.437, +0.541, +0.645, +0.939, +0.873, +0.977, +0.912, +0.734, +0.669, +0.491]
    Input : [-0.244, +0.154, -0.539, -0.244, -0.244, -0.244, -0.539, +0.154, -0.244, +0.154, +0.326, +0.154, +0.326]
    Gate  : [-0.427, -0.427, -0.545, -0.427, -0.427, -0.427, -0.545, -0.427, -0.427, -0.427, -0.545, -0.427, -0.545]



<div>Loss: <span style="font-weight:bold">0.48277</span>= <span style="font-family:monaco">0.001 + 0.000 + 0.048 + 0.034 + 0.022 + 0.013 + 0.077 + 0.088 + 0.071 + 0.081 + 0.007 + 0.011 + 0.030</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.012</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.088</span> + <span style="color:blue">+0.128</span> + <span style="color:blue">+0.146</span> + <span style="color:blue">+0.144</span> + <span style="color:blue">+0.399</span> + <span style="color:blue">+0.362</span> + <span style="color:blue">+0.386</span> + <span style="color:blue">+0.353</span> + <span style="color:blue">+0.117</span> + <span style="color:blue">+0.119</span> + <span style="color:red">-0.206</span> = <span style="color:blue; text-decoration:underline">+2.023</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.011</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.045</span> + <span style="color:blue">+0.077</span> + <span style="color:blue">+0.089</span> + <span style="color:blue">+0.193</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">+0.177</span> + <span style="color:blue">+0.115</span> + <span style="color:blue">+0.068</span> + <span style="color:blue">+0.054</span> + <span style="color:red">-0.159</span> = <span style="color:blue; text-decoration:underline">+0.771</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.006</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.094</span> + <span style="color:blue">+0.108</span> + <span style="color:blue">+0.111</span> + <span style="color:blue">+0.102</span> + <span style="color:blue">+0.308</span> + <span style="color:blue">+0.307</span> + <span style="color:blue">+0.307</span> + <span style="color:blue">+0.308</span> + <span style="color:blue">+0.084</span> + <span style="color:blue">+0.095</span> + <span style="color:red">-0.124</span> = <span style="color:blue; text-decoration:underline">+1.691</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.006</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.013</span> + <span style="color:blue">+0.041</span> + <span style="color:blue">+0.057</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.133</span> + <span style="color:blue">+0.118</span> + <span style="color:blue">+0.141</span> + <span style="color:blue">+0.127</span> + <span style="color:blue">+0.042</span> + <span style="color:blue">+0.042</span> + <span style="color:red">-0.073</span> = <span style="color:blue; text-decoration:underline">+0.697</span></div>


    w1=-0.741, w2=+0.249, w3=+1.000, w4=-0.714, w5=+0.048
    



<div style="font-weight:bold">Iteration 12</div>


    Output: [+0.587, +0.517, +0.645, +0.721, +0.786, +0.839, +0.899, +0.870, +0.905, +0.877, +0.857, +0.819, +0.791]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.352, +0.069, +0.598, +0.950, +1.302, +1.654, +2.183, +1.900, +2.252, +1.968, +1.791, +1.508, +1.330]
    Input : [-0.528, +0.425, -0.741, -0.528, -0.528, -0.528, -0.741, +0.425, -0.528, +0.425, +0.249, +0.425, +0.249]
    Gate  : [-0.666, -0.666, -0.714, -0.666, -0.666, -0.666, -0.714, -0.666, -0.666, -0.666, -0.714, -0.666, -0.714]



<div>Loss: <span style="font-weight:bold">0.36830</span>= <span style="font-family:monaco">0.015 + 0.001 + 0.017 + 0.000 + 0.009 + 0.038 + 0.002 + 0.001 + 0.003 + 0.000 + 0.054 + 0.023 + 0.207</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.060</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.067</span> + <span style="color:blue">+0.015</span> + <span style="color:red">-0.119</span> + <span style="color:red">-0.309</span> + <span style="color:red">-0.064</span> + <span style="color:blue">+0.032</span> + <span style="color:red">-0.087</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.380</span> + <span style="color:red">-0.210</span> + <span style="color:red">-0.696</span> = <span style="color:red; text-decoration:underline">-1.803</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.056</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.067</span> + <span style="color:red">-0.201</span> + <span style="color:red">-0.033</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.043</span> + <span style="color:blue">+0.004</span> + <span style="color:red">-0.225</span> + <span style="color:red">-0.094</span> + <span style="color:red">-0.519</span> = <span style="color:red; text-decoration:underline">-1.221</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.046</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.073</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.105</span> + <span style="color:red">-0.263</span> + <span style="color:red">-0.057</span> + <span style="color:blue">+0.030</span> + <span style="color:red">-0.079</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.328</span> + <span style="color:red">-0.191</span> + <span style="color:red">-0.561</span> = <span style="color:red; text-decoration:underline">-1.504</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.046</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.006</span> + <span style="color:red">-0.064</span> + <span style="color:red">-0.183</span> + <span style="color:red">-0.030</span> + <span style="color:blue">+0.014</span> + <span style="color:red">-0.043</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.172</span> + <span style="color:red">-0.083</span> + <span style="color:red">-0.274</span> = <span style="color:red; text-decoration:underline">-0.863</span></div>


    w1=-0.561, w2=+0.371, w3=+1.000, w4=-0.563, w5=+0.134
    



<div style="font-weight:bold">Iteration 13</div>


    Output: [+0.524, +0.510, +0.589, +0.612, +0.634, +0.656, +0.724, +0.713, +0.732, +0.721, +0.677, +0.665, +0.617]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.096, +0.042, +0.358, +0.454, +0.550, +0.646, +0.962, +0.908, +1.004, +0.950, +0.741, +0.687, +0.478]
    Input : [-0.224, +0.126, -0.561, -0.224, -0.224, -0.224, -0.561, +0.126, -0.224, +0.126, +0.371, +0.126, +0.371]
    Gate  : [-0.429, -0.429, -0.563, -0.429, -0.429, -0.429, -0.563, -0.429, -0.429, -0.429, -0.563, -0.429, -0.563]



<div>Loss: <span style="font-weight:bold">0.45267</span>= <span style="font-family:monaco">0.001 + 0.000 + 0.044 + 0.032 + 0.021 + 0.013 + 0.073 + 0.082 + 0.067 + 0.075 + 0.007 + 0.010 + 0.028</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.011</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.087</span> + <span style="color:blue">+0.126</span> + <span style="color:blue">+0.145</span> + <span style="color:blue">+0.146</span> + <span style="color:blue">+0.394</span> + <span style="color:blue">+0.354</span> + <span style="color:blue">+0.380</span> + <span style="color:blue">+0.343</span> + <span style="color:blue">+0.116</span> + <span style="color:blue">+0.115</span> + <span style="color:red">-0.205</span> = <span style="color:blue; text-decoration:underline">+1.989</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.010</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.044</span> + <span style="color:blue">+0.076</span> + <span style="color:blue">+0.090</span> + <span style="color:blue">+0.188</span> + <span style="color:blue">+0.124</span> + <span style="color:blue">+0.171</span> + <span style="color:blue">+0.110</span> + <span style="color:blue">+0.067</span> + <span style="color:blue">+0.052</span> + <span style="color:red">-0.159</span> = <span style="color:blue; text-decoration:underline">+0.748</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.005</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.094</span> + <span style="color:blue">+0.105</span> + <span style="color:blue">+0.107</span> + <span style="color:blue">+0.100</span> + <span style="color:blue">+0.297</span> + <span style="color:blue">+0.297</span> + <span style="color:blue">+0.296</span> + <span style="color:blue">+0.297</span> + <span style="color:blue">+0.080</span> + <span style="color:blue">+0.090</span> + <span style="color:red">-0.117</span> = <span style="color:blue; text-decoration:underline">+1.641</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.005</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.038</span> + <span style="color:blue">+0.053</span> + <span style="color:blue">+0.058</span> + <span style="color:blue">+0.121</span> + <span style="color:blue">+0.108</span> + <span style="color:blue">+0.129</span> + <span style="color:blue">+0.118</span> + <span style="color:blue">+0.040</span> + <span style="color:blue">+0.040</span> + <span style="color:red">-0.072</span> = <span style="color:blue; text-decoration:underline">+0.641</span></div>


    w1=-0.760, w2=+0.296, w3=+1.000, w4=-0.727, w5=+0.070
    



<div style="font-weight:bold">Iteration 14</div>


    Output: [+0.582, +0.518, +0.651, +0.722, +0.783, +0.834, +0.897, +0.871, +0.904, +0.879, +0.854, +0.819, +0.785]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.330, +0.073, +0.625, +0.955, +1.286, +1.616, +2.168, +1.911, +2.241, +1.983, +1.768, +1.511, +1.295]
    Input : [-0.502, +0.392, -0.760, -0.502, -0.502, -0.502, -0.760, +0.392, -0.502, +0.392, +0.296, +0.392, +0.296]
    Gate  : [-0.658, -0.658, -0.727, -0.658, -0.658, -0.658, -0.727, -0.658, -0.658, -0.658, -0.727, -0.658, -0.727]



<div>Loss: <span style="font-weight:bold">0.34592</span>= <span style="font-family:monaco">0.014 + 0.001 + 0.015 + 0.000 + 0.008 + 0.034 + 0.001 + 0.000 + 0.003 + 0.000 + 0.051 + 0.024 + 0.196</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.056</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.063</span> + <span style="color:blue">+0.013</span> + <span style="color:red">-0.113</span> + <span style="color:red">-0.293</span> + <span style="color:red">-0.059</span> + <span style="color:blue">+0.029</span> + <span style="color:red">-0.084</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.372</span> + <span style="color:red">-0.212</span> + <span style="color:red">-0.687</span> = <span style="color:red; text-decoration:underline">-1.767</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.052</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.063</span> + <span style="color:red">-0.189</span> + <span style="color:red">-0.030</span> + <span style="color:blue">+0.011</span> + <span style="color:red">-0.041</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.220</span> + <span style="color:red">-0.095</span> + <span style="color:red">-0.515</span> = <span style="color:red; text-decoration:underline">-1.191</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.041</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.069</span> + <span style="color:blue">+0.012</span> + <span style="color:red">-0.098</span> + <span style="color:red">-0.245</span> + <span style="color:red">-0.052</span> + <span style="color:blue">+0.027</span> + <span style="color:red">-0.075</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.315</span> + <span style="color:red">-0.191</span> + <span style="color:red">-0.533</span> = <span style="color:red; text-decoration:underline">-1.440</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.041</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.058</span> + <span style="color:red">-0.167</span> + <span style="color:red">-0.027</span> + <span style="color:blue">+0.012</span> + <span style="color:red">-0.040</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.164</span> + <span style="color:red">-0.083</span> + <span style="color:red">-0.269</span> = <span style="color:red; text-decoration:underline">-0.823</span></div>


    w1=-0.583, w2=+0.415, w3=+1.000, w4=-0.583, w5=+0.152
    



<div style="font-weight:bold">Iteration 49</div>


    Output: [+0.494, +0.523, +0.710, +0.705, +0.699, +0.694, +0.835, +0.850, +0.847, +0.862, +0.714, +0.737, +0.529]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [-0.024, +0.090, +0.894, +0.869, +0.845, +0.820, +1.624, +1.738, +1.714, +1.829, +0.914, +1.029, +0.115]
    Input : [+0.052, -0.243, -0.847, +0.052, +0.052, +0.052, -0.847, -0.243, +0.052, -0.243, +0.965, -0.243, +0.965]
    Gate  : [-0.472, -0.472, -0.948, -0.472, -0.472, -0.472, -0.948, -0.472, -0.472, -0.472, -0.948, -0.472, -0.948]



<div>Loss: <span style="font-weight:bold">0.03034</span>= <span style="font-family:monaco">0.000 + 0.001 + 0.001 + 0.002 + 0.002 + 0.003 + 0.008 + 0.004 + 0.005 + 0.002 + 0.001 + 0.000 + 0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.003</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.039</span> + <span style="color:blue">+0.062</span> + <span style="color:blue">+0.091</span> + <span style="color:blue">+0.155</span> + <span style="color:blue">+0.090</span> + <span style="color:blue">+0.116</span> + <span style="color:blue">+0.058</span> + <span style="color:blue">+0.052</span> + <span style="color:red">-0.015</span> + <span style="color:red">-0.074</span> = <span style="color:blue; text-decoration:underline">+0.599</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.027</span> + <span style="color:blue">+0.048</span> + <span style="color:blue">+0.060</span> + <span style="color:blue">+0.025</span> + <span style="color:blue">+0.042</span> + <span style="color:blue">+0.015</span> + <span style="color:blue">+0.029</span> + <span style="color:red">-0.007</span> + <span style="color:red">-0.062</span> = <span style="color:blue; text-decoration:underline">+0.192</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.026</span> + <span style="color:blue">+0.030</span> + <span style="color:blue">+0.032</span> + <span style="color:blue">+0.079</span> + <span style="color:blue">+0.060</span> + <span style="color:blue">+0.064</span> + <span style="color:blue">+0.042</span> + <span style="color:blue">+0.021</span> + <span style="color:red">-0.008</span> + <span style="color:red">-0.014</span> = <span style="color:blue; text-decoration:underline">+0.349</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.008</span> + <span style="color:red">-0.004</span> + <span style="color:red">-0.020</span> = <span style="color:blue; text-decoration:underline">+0.018</span></div>


    w1=-0.907, w2=+0.945, w3=+1.000, w4=-0.983, w5=+0.474
    



<div style="font-weight:bold">Iteration 99</div>


    Output: [+0.495, +0.518, +0.734, +0.730, +0.726, +0.722, +0.869, +0.880, +0.878, +0.887, +0.730, +0.748, +0.505]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [-0.020, +0.074, +1.014, +0.995, +0.975, +0.955, +1.896, +1.989, +1.970, +2.063, +0.994, +1.088, +0.018]
    Input : [+0.056, -0.266, -0.932, +0.056, +0.056, +0.056, -0.932, -0.266, +0.056, -0.266, +1.060, -0.266, +1.060]
    Gate  : [-0.352, -0.352, -1.009, -0.352, -0.352, -0.352, -1.009, -0.352, -0.352, -0.352, -1.009, -0.352, -1.009]



<div>Loss: <span style="font-weight:bold">0.00266</span>= <span style="font-family:monaco">0.000 + 0.001 + 0.000 + 0.000 + 0.000 + 0.000 + 0.001 + 0.000 + 0.000 + 0.000 + 0.000 + 0.001 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.002</span> + <span style="color:red">-0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.019</span> + <span style="color:blue">+0.036</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.010</span> + <span style="color:red">-0.019</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.043</span> + <span style="color:red">-0.012</span> = <span style="color:blue; text-decoration:underline">+0.007</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.020</span> + <span style="color:red">-0.010</span> = <span style="color:red; text-decoration:underline">-0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.004</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.009</span> + <span style="color:blue">+0.022</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.007</span> + <span style="color:red">-0.015</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.027</span> + <span style="color:red">-0.002</span> = <span style="color:red; text-decoration:underline">-0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.004</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.013</span> + <span style="color:red">-0.004</span> = <span style="color:red; text-decoration:underline">-0.022</span></div>


    w1=-0.933, w2=+1.060, w3=+1.000, w4=-1.008, w5=+0.659
    



<div style="font-weight:bold">Iteration 149</div>


    Output: [+0.498, +0.514, +0.732, +0.730, +0.729, +0.727, +0.873, +0.880, +0.879, +0.886, +0.730, +0.742, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [-0.008, +0.054, +1.004, +0.996, +0.988, +0.980, +1.930, +1.992, +1.984, +2.047, +0.993, +1.055, +0.001]
    Input : [+0.032, -0.246, -0.961, +0.032, +0.032, +0.032, -0.961, -0.246, +0.032, -0.246, +1.066, -0.246, +1.066]
    Gate  : [-0.253, -0.253, -0.988, -0.253, -0.253, -0.253, -0.988, -0.253, -0.253, -0.253, -0.988, -0.253, -0.988]



<div>Loss: <span style="font-weight:bold">0.00112</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.021</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.005</span> + <span style="color:red">-0.012</span> + <span style="color:blue">+0.004</span> + <span style="color:red">-0.025</span> + <span style="color:red">-0.000</span> = <span style="color:blue; text-decoration:underline">+0.005</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.012</span> + <span style="color:red">-0.000</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.015</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.004</span> + <span style="color:red">-0.012</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.018</span> + <span style="color:red">-0.000</span> = <span style="color:red; text-decoration:underline">-0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.003</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.009</span> + <span style="color:red">-0.000</span> = <span style="color:red; text-decoration:underline">-0.011</span></div>


    w1=-0.962, w2=+1.066, w3=+1.000, w4=-0.988, w5=+0.736
    



<div style="font-weight:bold">Iteration 199</div>


    Output: [+0.499, +0.511, +0.731, +0.730, +0.730, +0.729, +0.875, +0.880, +0.880, +0.884, +0.730, +0.739, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [-0.003, +0.042, +0.999, +0.997, +0.994, +0.991, +1.948, +1.993, +1.991, +2.035, +0.994, +1.038, -0.004]
    Input : [+0.013, -0.229, -0.981, +0.013, +0.013, +0.013, -0.981, -0.229, +0.013, -0.229, +1.068, -0.229, +1.068]
    Gate  : [-0.196, -0.196, -0.975, -0.196, -0.196, -0.196, -0.975, -0.196, -0.196, -0.196, -0.975, -0.196, -0.975]



<div>Loss: <span style="font-weight:bold">0.00060</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.014</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.003</span> + <span style="color:red">-0.017</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.012</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.013</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.007</span></div>


    w1=-0.981, w2=+1.068, w3=+1.000, w4=-0.975, w5=+0.781
    



<div style="font-weight:bold">Iteration 249</div>


    Output: [+0.500, +0.509, +0.730, +0.730, +0.730, +0.731, +0.876, +0.880, +0.880, +0.884, +0.730, +0.737, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.000, +0.034, +0.997, +0.997, +0.997, +0.997, +1.960, +1.994, +1.994, +2.028, +0.995, +1.029, -0.005]
    Input : [-0.001, -0.216, -0.995, -0.001, -0.001, -0.001, -0.995, -0.216, -0.001, -0.216, +1.069, -0.216, +1.069]
    Gate  : [-0.157, -0.157, -0.967, -0.157, -0.157, -0.157, -0.967, -0.157, -0.157, -0.157, -0.967, -0.157, -0.967]



<div>Loss: <span style="font-weight:bold">0.00037</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.011</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.012</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.003</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.010</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.010</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.005</span></div>


    w1=-0.996, w2=+1.069, w3=+1.000, w4=-0.967, w5=+0.810
    



<div style="font-weight:bold">Iteration 299</div>


    Output: [+0.500, +0.507, +0.730, +0.730, +0.731, +0.731, +0.877, +0.880, +0.880, +0.883, +0.730, +0.735, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.028, +0.995, +0.997, +0.999, +1.000, +1.967, +1.994, +1.996, +2.023, +0.996, +1.022, -0.005]
    Input : [-0.012, -0.206, -1.007, -0.012, -0.012, -0.012, -1.007, -0.206, -0.012, -0.206, +1.069, -0.206, +1.069]
    Gate  : [-0.130, -0.130, -0.961, -0.130, -0.130, -0.130, -0.961, -0.130, -0.130, -0.130, -0.961, -0.130, -0.961]



<div>Loss: <span style="font-weight:bold">0.00024</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.009</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.008</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.008</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.004</span></div>


    w1=-1.007, w2=+1.069, w3=+1.000, w4=-0.961, w5=+0.831
    



<div style="font-weight:bold">Iteration 349</div>


    Output: [+0.501, +0.506, +0.730, +0.731, +0.731, +0.731, +0.878, +0.880, +0.880, +0.883, +0.730, +0.735, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.024, +0.995, +0.997, +1.000, +1.002, +1.973, +1.995, +1.997, +2.019, +0.996, +1.018, -0.004]
    Input : [-0.022, -0.198, -1.016, -0.022, -0.022, -0.022, -1.016, -0.198, -0.022, -0.198, +1.069, -0.198, +1.069]
    Gate  : [-0.109, -0.109, -0.956, -0.109, -0.109, -0.109, -0.956, -0.109, -0.109, -0.109, -0.956, -0.109, -0.956]



<div>Loss: <span style="font-weight:bold">0.00017</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.002</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.002</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.007</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.007</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.003</span></div>


    w1=-1.016, w2=+1.069, w3=+1.000, w4=-0.956, w5=+0.847
    



<div style="font-weight:bold">Iteration 399</div>


    Output: [+0.501, +0.505, +0.730, +0.731, +0.731, +0.732, +0.878, +0.880, +0.881, +0.882, +0.730, +0.734, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.003, +0.020, +0.995, +0.998, +1.000, +1.003, +1.977, +1.995, +1.998, +2.015, +0.997, +1.015, -0.004]
    Input : [-0.029, -0.191, -1.023, -0.029, -0.029, -0.029, -1.023, -0.191, -0.029, -0.191, +1.069, -0.191, +1.069]
    Gate  : [-0.092, -0.092, -0.952, -0.092, -0.092, -0.092, -0.952, -0.092, -0.092, -0.092, -0.952, -0.092, -0.952]



<div>Loss: <span style="font-weight:bold">0.00012</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.006</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.006</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div>


    w1=-1.023, w2=+1.069, w3=+1.000, w4=-0.952, w5=+0.860
    



<div style="font-weight:bold">Iteration 449</div>


    Output: [+0.501, +0.504, +0.730, +0.731, +0.731, +0.732, +0.879, +0.880, +0.881, +0.882, +0.731, +0.733, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.003, +0.017, +0.995, +0.998, +1.001, +1.003, +1.981, +1.996, +1.998, +2.013, +0.998, +1.012, -0.003]
    Input : [-0.036, -0.185, -1.030, -0.036, -0.036, -0.036, -1.030, -0.185, -0.036, -0.185, +1.069, -0.185, +1.069]
    Gate  : [-0.079, -0.079, -0.950, -0.079, -0.079, -0.079, -0.950, -0.079, -0.079, -0.079, -0.950, -0.079, -0.950]



<div>Loss: <span style="font-weight:bold">0.00009</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.002</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.002</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.005</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.005</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div>


    w1=-1.030, w2=+1.069, w3=+1.000, w4=-0.949, w5=+0.871
    



<div style="font-weight:bold">Iteration 499</div>


    Output: [+0.501, +0.504, +0.730, +0.731, +0.731, +0.732, +0.879, +0.880, +0.881, +0.882, +0.731, +0.733, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.003, +0.015, +0.995, +0.998, +1.001, +1.004, +1.984, +1.996, +1.999, +2.011, +0.998, +1.010, -0.003]
    Input : [-0.041, -0.180, -1.035, -0.041, -0.041, -0.041, -1.035, -0.180, -0.041, -0.180, +1.069, -0.180, +1.069]
    Gate  : [-0.067, -0.067, -0.947, -0.067, -0.067, -0.067, -0.947, -0.067, -0.067, -0.067, -0.947, -0.067, -0.947]



<div>Loss: <span style="font-weight:bold">0.00007</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.004</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.002</span></div>


    w1=-1.035, w2=+1.069, w3=+1.000, w4=-0.947, w5=+0.880
    



<div style="font-weight:bold">Iteration 549</div>


    Output: [+0.501, +0.503, +0.730, +0.731, +0.731, +0.732, +0.879, +0.880, +0.881, +0.882, +0.731, +0.733, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.003, +0.013, +0.996, +0.998, +1.001, +1.004, +1.986, +1.996, +1.999, +2.009, +0.998, +1.008, -0.002]
    Input : [-0.046, -0.176, -1.040, -0.046, -0.046, -0.046, -1.040, -0.176, -0.046, -0.176, +1.069, -0.176, +1.069]
    Gate  : [-0.058, -0.058, -0.945, -0.058, -0.058, -0.058, -0.945, -0.058, -0.058, -0.058, -0.945, -0.058, -0.945]



<div>Loss: <span style="font-weight:bold">0.00005</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.004</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div>


    w1=-1.040, w2=+1.069, w3=+1.000, w4=-0.945, w5=+0.888
    



<div style="font-weight:bold">Iteration 599</div>


    Output: [+0.501, +0.503, +0.730, +0.731, +0.731, +0.732, +0.880, +0.880, +0.881, +0.882, +0.731, +0.732, +0.499]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.011, +0.996, +0.998, +1.001, +1.003, +1.988, +1.997, +1.999, +2.008, +0.999, +1.007, -0.002]
    Input : [-0.050, -0.172, -1.044, -0.050, -0.050, -0.050, -1.044, -0.172, -0.050, -0.172, +1.070, -0.172, +1.070]
    Gate  : [-0.050, -0.050, -0.944, -0.050, -0.050, -0.050, -0.944, -0.050, -0.050, -0.050, -0.944, -0.050, -0.944]



<div>Loss: <span style="font-weight:bold">0.00004</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:red">-0.003</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.001</span></div>


    w1=-1.044, w2=+1.070, w3=+1.000, w4=-0.944, w5=+0.894
    



<div style="font-weight:bold">Iteration 649</div>


    Output: [+0.501, +0.502, +0.730, +0.731, +0.731, +0.732, +0.880, +0.880, +0.881, +0.881, +0.731, +0.732, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.009, +0.996, +0.999, +1.001, +1.003, +1.990, +1.997, +1.999, +2.007, +0.999, +1.006, -0.002]
    Input : [-0.054, -0.169, -1.047, -0.054, -0.054, -0.054, -1.047, -0.169, -0.054, -0.169, +1.070, -0.169, +1.070]
    Gate  : [-0.043, -0.043, -0.942, -0.043, -0.043, -0.043, -0.942, -0.043, -0.043, -0.043, -0.942, -0.043, -0.942]



<div>Loss: <span style="font-weight:bold">0.00003</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.003</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.001</span></div>


    w1=-1.047, w2=+1.070, w3=+1.000, w4=-0.942, w5=+0.900
    



<div style="font-weight:bold">Iteration 699</div>


    Output: [+0.501, +0.502, +0.730, +0.731, +0.731, +0.732, +0.880, +0.881, +0.881, +0.881, +0.731, +0.732, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.008, +0.997, +0.999, +1.001, +1.003, +1.991, +1.997, +2.000, +2.006, +0.999, +1.005, -0.002]
    Input : [-0.057, -0.166, -1.050, -0.057, -0.057, -0.057, -1.050, -0.166, -0.057, -0.166, +1.070, -0.166, +1.070]
    Gate  : [-0.037, -0.037, -0.941, -0.037, -0.037, -0.037, -0.941, -0.037, -0.037, -0.037, -0.941, -0.037, -0.941]



<div>Loss: <span style="font-weight:bold">0.00002</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.001</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.001</span></div>


    w1=-1.050, w2=+1.070, w3=+1.000, w4=-0.941, w5=+0.905
    



<div style="font-weight:bold">Iteration 749</div>


    Output: [+0.500, +0.502, +0.730, +0.731, +0.731, +0.732, +0.880, +0.881, +0.881, +0.881, +0.731, +0.732, +0.500]
    Target: [+0.500, +0.500, +0.731, +0.731, +0.731, +0.731, +0.881, +0.881, +0.881, +0.881, +0.731, +0.731, +0.500]
    Memory: [+0.002, +0.007, +0.997, +0.999, +1.001, +1.003, +1.993, +1.998, +2.000, +2.005, +0.999, +1.004, -0.001]
    Input : [-0.060, -0.164, -1.053, -0.060, -0.060, -0.060, -1.053, -0.164, -0.060, -0.164, +1.070, -0.164, +1.070]
    Gate  : [-0.031, -0.031, -0.940, -0.031, -0.031, -0.031, -0.940, -0.031, -0.031, -0.031, -0.940, -0.031, -0.940]



<div>Loss: <span style="font-weight:bold">0.00001</span>= <span style="font-family:monaco">0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000 + 0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw1 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.001</span> = <span style="color:blue; text-decoration:underline">+0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw2 = <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.001</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw4 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.002</span> + <span style="color:blue">+0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.002</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.000</span></div><div style="font-family:monaco; font-size:12px">dL/dw5 = <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.000</span> + <span style="color:blue">+0.000</span> + <span style="color:red">-0.001</span> + <span style="color:blue">+0.000</span> = <span style="color:red; text-decoration:underline">-0.001</span></div>


    w1=-1.053, w2=+1.070, w3=+1.000, w4=-0.940, w5=+0.909
    





    (-1.0529977543939428,
     1.0696492814766454,
     1.0,
     -0.9401480352201477,
     0.9087447843399465)



5f940e35312bd052ace5b3a2315c8e5b42202196
ipynb
Jupyter Notebook
neural_network/CEC-test.ipynb
justhalf/jupyter_notebooks
247504257d206fd0dc27116689d2c29ee8bd0209
MIT
neural_network/CEC-test.ipynb
justhalf/jupyter_notebooks
247504257d206fd0dc27116689d2c29ee8bd0209
MIT
neural_network/CEC-test.ipynb
justhalf/jupyter_notebooks
247504257d206fd0dc27116689d2c29ee8bd0209
MIT
Qwen/Qwen-72B
1. YES
2. YES