###### Formulation:
* Stages: $i = 1,2,3,\cdots$.
* State Space: $\mathcal{X} = \{a,b,c\}$.
* Control Space: $\mathcal{U} = \{-1,1\}$
* Feasible control mapping $\mathcal{X}\rightarrow\mathcal{U}: \mathcal{U}(x)=\{-1,1\}$
* Transition mapping, given $x_i, u_i$, $\mathcal{X}\rightarrow [0,1]$, for $i\in \{1,2,\cdots\}$. 
$P(j\mid i, u=-1)$

| i\\j | a   | b   | c   
| ---  | --- | --- | --- 
| a    | 1.0 | 0.0 | 0.0 
| b    | 0.8| 0.0 | 0.2
| c    | 0.0 | 1.0 | 0.0 

$P(j\mid i, u=1)$

| i\\j | a   | b   | c   
| ---  | --- | --- | --- 
| a    | 0.0 | 1.0 | 0.0 
| b    | 0.6| 0.0 | 0.4
| c    | 0.0 | 0.0 | 1.0 


* Reward function $\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R}$: 
$$r(a, -1)=-1; \quad r(a, 1)=1;\quad r(c, -1)=10;\quad r(c,1)=-10 $$
For other state and action combinations $(x, u)\in (\mathcal{S}, \mathcal{A})$, $r(x, u)=0$.
* Let $v^*(x)$ be the optimal value function for state $x$, so it satisfies the following equation:

\begin{align}
    v^*(x_i) = &\max_{u\in \mathcal{U}(x)}\{r(x_i, u)+\alpha\mathbb{E}[v^*(x_{i+1})]\}\\
\end{align}

And it can be solved by value iteration:

\begin{align}
    v^{k+1}(x_i) = \mathcal{D}(v)(x_i) = &\max_{u\in \mathcal{U}(x)}\{r(x_i, u)+\alpha\mathbb{E}[v^k(x_{i+1})]\}\\
\end{align}
with arbitrary $v^0$, since $\mathcal{D}$ is a contraction mapping.


```python
from itertools import product
import numpy as np
import pandas as pd
from copy import deepcopy
import random
import matplotlib.pyplot as plt
import seaborn as sns
```


```python
pd.DataFrame({0: [1., 0.8, 0.], 1:[0.,0.,1.], 2:[0.,0.2, 0.]})
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.8</td>
      <td>0.0</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
state_space = [0,1,2]
control_space = [-1,1]
alpha = 0.95
tun1 = pd.DataFrame({0: [1., 1., 0.], 1:[0.,0.,1.], 2:[0.,0., 0.]})
tu1 = pd.DataFrame({0: [0.,0., 0.], 1:[1.,0.,0.], 2:[0.,1., 1.]})
def trans_prob(next_x, x, u):
    if u == 1:
        t = tu1
    elif u == -1:
        t = tun1
    else:
        raise ValueError
    return t.loc[x, next_x]

def reward(x, u):
    if x == 0 and u == 1:
        return 1.
    elif x == 2 and u == -1:
        return 10.
    elif x == 2 and u == 1:
        return -10.
    elif x == 0 and u == -1:
        return -1
    else:
        return 0.
```


```python
tu1
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
tun1
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
v = pd.Series([0.0 for _ in state_space], index = state_space)
pi = pd.Series([0 for _ in state_space], index = state_space)
```


```python
def value_iteration(state_space, control_space, trans_prob, reward, alpha, v, pi, MAX_ITER = 1000000, threshold = 1e-6):
    for _ in range(MAX_ITER):
        next_v = deepcopy(v)
        for x in state_space:
            tmp = [reward(x, u)+alpha*
                   np.sum([v[next_x]*trans_prob(next_x, x, u) for next_x in state_space]) 
                   for u in control_space]
            next_v[x]= np.max(tmp)
            pi[x]= control_space[np.argmax(tmp)]
        if abs(next_v-v).values.max()<1e-6:
            break
        else:
            v = next_v
            
    return v, pi
```


```python
v, pi = value_iteration(state_space, control_space, trans_prob, reward, 0.95, v, pi)
```


```python
v
```




    0     93.564093
    1     97.435888
    2    102.564093
    dtype: float64




```python
pi
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
q_true = pd.DataFrame({x:[0.0]*len(control_space) for x in state_space}, index = control_space) 
for x in state_space:
    for u in control_space:
        q_true.loc[u, x] = reward(x, u)+alpha*np.sum([v[next_x]*trans_prob(next_x, x, u) for next_x in state_space])
print(q_true)

```

                0          1           2
    -1  87.885889  88.885889  102.564093
     1  93.564093  97.435889   87.435889



```python
list(product(state_space, control_space))
```




    [(0, -1), (0, 1), (1, -1), (1, 1), (2, -1), (2, 1)]




```python
# Augmented MDP
state_space_d = list(product(state_space, control_space))
control_space_d = [-1,1]
alpha = 0.95
# tun1 = pd.DataFrame({0: [1., 0.8, 0.], 1:[0.,0.,1.], 2:[0.,0.2, 0.]})
# tu1 = pd.DataFrame({0: [0., 0.6, 0.], 1:[1.,0.,0.], 2:[0.,0.4, 1.]})
def trans_prob_d(next_i, i, u):
    if next_i[1]==u:
        return trans_prob(next_i[0], i[0], i[1])
    else:
        return 0.0

def reward_d(x, u):
    return reward(x[0], x[1])
```


```python
v_d = pd.Series([0.0 for _ in state_space_d], index = state_space_d)
pi_d = pd.Series([0 for _ in state_space_d], index = state_space_d)
v_d
```




    (0, -1)    0.0
    (0, 1)     0.0
    (1, -1)    0.0
    (1, 1)     0.0
    (2, -1)    0.0
    (2, 1)     0.0
    dtype: float64




```python
v_AMDP, pi_AMDP = value_iteration(state_space_d, control_space_d, trans_prob_d, reward_d, 0.95, v_d, pi_d)
```


```python
v_AMDP
```




    (0, -1)     87.885888
    (0, 1)      93.564093
    (1, -1)     88.885888
    (1, 1)      97.435888
    (2, -1)    102.564093
    (2, 1)      87.435888
    dtype: float64




```python
q_true.loc[-1, 0][-1]
```




    0.0




```python
next_x = x
v_AMDP[next_x]*trans_prob_d(next_x, x, u)
```




    87.8858875746682




```python
q_true = pd.DataFrame({x:[0.0]*len(control_space_d) for x in state_space_d}, index = control_space_d) 
for x in state_space_d:
    for u in control_space_d:
        q_true.loc[u, x[0]][x[1]] = reward_d(x, u)+alpha*np.sum([v_AMDP[next_x]*trans_prob_d(next_x, x, u) for next_x in state_space_d])
print(q_true)
```

                0                     1                      2           
               -1          1         -1          1          -1          1
    -1  82.491593  85.441593  83.491593  97.435889   94.441593  87.435889
     1  87.885889  93.564093  88.885889  83.064093  102.564093  73.064093



```python
q_true.loc[-1, 1][1]
```




    97.43588853613805




```python
reward
```


```python
pi_AMDP
```




    (0, -1)    1
    (0, 1)     1
    (1, -1)    1
    (1, 1)    -1
    (2, -1)    1
    (2, 1)    -1
    dtype: int64




```python
# Q_learning env
class Env:
    def __init__(self):
#         self.gamma = gamma
        self.obs = 0
        self.control_space = (-1, 1)
        self.state_space = tuple([0,1,2])
        self.state_dim = 1
        self.action_dim = 1
        
    def step(self, action, current_obs = None):
        
        obs = self.obs if current_obs is None else current_obs
        self.obs = self._next_state(obs, action)
        r = reward(obs, action)
#         r = reward((q,p), action)
        return self.obs, r, False, {}
    
    def reset(self, random=False):
        self.obs = 0 if not random else np.random.choice([0,1,2])
        return self.obs
    
    @staticmethod
    def _next_state(obs, action):
        p = tun1.loc[obs, :] if action == -1 else tu1.loc[obs, :]
        return np.random.choice([0,1,2], p = p)
      
```


```python
class Q_table:
    def __init__(self, q_data=None, random = False):
        self.control_space = (-1,1)
        self.state_space = tuple([0,1,2])
        if q_data is None:
            if not random:
                self.data = pd.DataFrame({x:[0.0]*len(self.control_space) for x in self.state_space},
                                         index = self.control_space) 
            else:
                self.data = pd.DataFrame({x:np.random.randn(len(self.control_space)) for x in self.state_space}, 
                                         index = self.control_space) 
        else:
#             assert isinstance(q, Q_table)
            self.data = q_data
        
    def __getitem__(self, key):
        x, u = key
        return self.data.loc[u,x]
    
    def __setitem__(self, key, item):
        x, u = key
        self.data.loc[u,x] = item
    
    def max_value_act(self, x):
        tmp = [self.data.loc[k,x] for k in self.control_space]
        return np.max(tmp), self.control_space[np.argmax(tmp)]
    
    def difference(self, q):
        assert isinstance(q, Q_table)
        return Q_table(self.data-q.data)
        
    def L_infty(self):
        return np.max(np.abs(self.data).values)
    
    def L_1(self):
        return np.mean(np.abs(self.data).values)
    
    def yield_v_pi(self):
        v = pd.Series([0.0 for _ in self.state_space], index = self.state_space)
        pi = pd.Series([0 for _ in self.state_space], index = self.state_space)
        for i in self.state_space:
            v[i], pi[i] = self.max_value_act(i)     
        return v, pi
        
    def __repr__(self):

        return self.data.__repr__()
    
class Pool:
    def __init__(self, length):
        self.ls = [None]*length
        self.len = length
        self.idx = 0
        self.isFull = False
        
    def append(self, item):
        self.ls[self.idx] = item
        self.idx = (self.idx + 1) % self.len
        if not self.isFull:
            if self.idx + 1 == self.len:
                self.isFull = True
                
    def shuffle(self):
        cp = deepcopy(self.ls)
        random.shuffle(cp)
        return cp
        
    def __getitem__(self, key):
        idx = (self.idx+key) % self.len
        return self.ls[idx]
    
    def __iter__(self):
        return iter(self.ls[self.idx:]+self.ls[:self.idx])
    
    def __repr__(self):
        return str(self.ls[self.idx:]+self.ls[:self.idx])
```


```python
# Q-learning with pool


class Q_learning:
    def __init__(self, q, env, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(40)
        self.qs = []
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random=True)
            r_ls = []
            for _ in range(10):
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (q.max_value_act(obs)[1], False)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q
```


```python
# Q-learning with epsilon-greedy policy,  learning rate = 1/t
q = Q_table(random = False)
env = Env()
q_one_over_t = Q_learning(q, env, MAX_ITER = 200)
lr_fn = lambda i, s, a: 1/(i+1)
lr_fn = lambda i, s, a: 0.1
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
learned_q = q_one_over_t.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    -0.7
    EP 3 learning finished: 10.515714285714285, 2.08502268009768
    4
    EP 4 simulation finished, start learning
    1.1
    EP 4 learning finished: 0.44802973947149716, 0.1945739008212918
    5
    EP 5 simulation finished, start learning
    1.4
    EP 5 learning finished: 10.245666781810186, 2.231536866915941
    6
    EP 6 simulation finished, start learning
    3.1
    EP 6 learning finished: 5.52838391027313, 2.878741990053412
    7
    EP 7 simulation finished, start learning
    4.1
    EP 7 learning finished: 3.0910624744318085, 1.7677704266421828
    8
    EP 8 simulation finished, start learning
    4.1
    EP 8 learning finished: 2.444823659022436, 1.225055624075892
    9
    EP 9 simulation finished, start learning
    4.1
    EP 9 learning finished: 1.7259065044907338, 0.824736058471223
    10
    EP 10 simulation finished, start learning
    5.0
    EP 10 learning finished: 1.294735790160761, 0.5105794548199727
    11
    EP 11 simulation finished, start learning
    4.1
    EP 11 learning finished: 0.9078351197851617, 0.41168433884935296
    12
    EP 12 simulation finished, start learning
    3.1
    EP 12 learning finished: 1.2432670831874209, 0.553581929782876
    13
    EP 13 simulation finished, start learning
    5.0
    EP 13 learning finished: 1.0039551820643648, 0.431109448184845
    14
    EP 14 simulation finished, start learning
    4.1
    EP 14 learning finished: 0.7811723531224288, 0.3854407933893705
    15
    EP 15 simulation finished, start learning
    5.0
    EP 15 learning finished: 0.6410853383183213, 0.3138910825079601
    16
    EP 16 simulation finished, start learning
    1.1
    EP 16 learning finished: 0.9767260039290644, 0.3473712400962124
    17
    EP 17 simulation finished, start learning
    5.0
    EP 17 learning finished: 0.6918028381530235, 0.28534496737046045
    18
    EP 18 simulation finished, start learning
    5.0
    EP 18 learning finished: 0.5418814319430503, 0.22580477569106586
    19
    EP 19 simulation finished, start learning
    4.1
    EP 19 learning finished: 0.4611824717480886, 0.251970058158467
    20
    EP 20 simulation finished, start learning
    3.0
    EP 20 learning finished: 0.28600332985865506, 0.1796811054942551
    21
    EP 21 simulation finished, start learning
    4.1
    EP 21 learning finished: 0.2906166814264388, 0.19198668648636877
    22
    EP 22 simulation finished, start learning
    4.1
    EP 22 learning finished: 0.41497044755948664, 0.2045014809919047
    23
    EP 23 simulation finished, start learning
    5.0
    EP 23 learning finished: 0.26301595807521316, 0.14929460971214206
    24
    EP 24 simulation finished, start learning
    4.1
    EP 24 learning finished: 0.3802501025647178, 0.13367010000398682
    25
    EP 25 simulation finished, start learning
    3.0
    EP 25 learning finished: 0.24337400516045093, 0.1367361208815394
    26
    EP 26 simulation finished, start learning
    5.0
    EP 26 learning finished: 0.19735935497014623, 0.11255024450422013
    27
    EP 27 simulation finished, start learning
    4.1
    EP 27 learning finished: 0.23504815791912748, 0.126064418004218
    28
    EP 28 simulation finished, start learning
    4.1
    EP 28 learning finished: 0.22819659996751085, 0.15159295709313683
    29
    EP 29 simulation finished, start learning
    3.1
    EP 29 learning finished: 0.3286420314741427, 0.16284488960237598
    30
    EP 30 simulation finished, start learning
    5.0
    EP 30 learning finished: 0.3108365768114254, 0.15624932964029123
    31
    EP 31 simulation finished, start learning
    4.1
    EP 31 learning finished: 0.29689780142582833, 0.15117693242857402
    32
    EP 32 simulation finished, start learning
    5.0
    EP 32 learning finished: 0.19142198128403365, 0.10127728505589732
    33
    EP 33 simulation finished, start learning
    5.0
    EP 33 learning finished: 0.14770693336267726, 0.06338352390622681
    34
    EP 34 simulation finished, start learning
    5.0
    EP 34 learning finished: 0.14080047687544095, 0.06137987908957262
    35
    EP 35 simulation finished, start learning
    5.0
    EP 35 learning finished: 0.141658085781998, 0.0451125663338973
    36
    EP 36 simulation finished, start learning
    5.0
    EP 36 learning finished: 0.13511220415372094, 0.043332810326903136
    37
    EP 37 simulation finished, start learning
    2.0
    EP 37 learning finished: 0.28586860213992793, 0.08728342165324143
    38
    EP 38 simulation finished, start learning
    5.0
    EP 38 learning finished: 0.2568413122462445, 0.08106131303540387
    39
    EP 39 simulation finished, start learning
    5.0
    EP 39 learning finished: 0.23289136777770736, 0.07577739377296074
    40
    EP 40 simulation finished, start learning
    3.2
    EP 40 learning finished: 0.21565074408016116, 0.1333581532505089
    41
    EP 41 simulation finished, start learning
    3.0
    EP 41 learning finished: 0.1890534274711868, 0.11218609099134502
    42
    EP 42 simulation finished, start learning
    5.0
    EP 42 learning finished: 0.18614546643157404, 0.10991050730876821
    43
    EP 43 simulation finished, start learning
    5.0
    EP 43 learning finished: 0.18297589093204447, 0.10720776772663543
    44
    EP 44 simulation finished, start learning
    3.2
    EP 44 learning finished: 0.3530724135780252, 0.13246808115433106
    45
    EP 45 simulation finished, start learning
    5.0
    EP 45 learning finished: 0.33350458677471195, 0.11327357881079969
    46
    EP 46 simulation finished, start learning
    5.0
    EP 46 learning finished: 0.32018405229881264, 0.1095106540785129
    47
    EP 47 simulation finished, start learning
    5.0
    EP 47 learning finished: 0.30988145845076254, 0.10621610461752251
    48
    EP 48 simulation finished, start learning
    3.0
    EP 48 learning finished: 0.09531312677665937, 0.045283234428702336
    49
    EP 49 simulation finished, start learning
    3.1
    EP 49 learning finished: 0.183442422341205, 0.09519314877546374
    50
    EP 50 simulation finished, start learning
    3.1
    EP 50 learning finished: 0.4320005026300979, 0.1961940345752269
    51
    EP 51 simulation finished, start learning
    4.0
    EP 51 learning finished: 0.5306900221168744, 0.25197208979065105
    52
    EP 52 simulation finished, start learning
    5.0
    EP 52 learning finished: 0.4981165257540656, 0.22874577759747253
    53
    EP 53 simulation finished, start learning
    4.1
    EP 53 learning finished: 0.3836478501679743, 0.18726983772216943
    54
    EP 54 simulation finished, start learning
    3.2
    EP 54 learning finished: 0.33461583071784773, 0.15039203242784302
    55
    EP 55 simulation finished, start learning
    3.0
    EP 55 learning finished: 0.21543659330231613, 0.10081250241228161
    56
    EP 56 simulation finished, start learning
    5.0
    EP 56 learning finished: 0.2116416802860499, 0.09826678153292316
    57
    EP 57 simulation finished, start learning
    5.0
    EP 57 learning finished: 0.20451466398462514, 0.0868290554792767
    58
    EP 58 simulation finished, start learning
    5.0
    EP 58 learning finished: 0.07251858479493833, 0.03576742786309772
    59
    EP 59 simulation finished, start learning
    4.1
    EP 59 learning finished: 0.07099613990277831, 0.03274590151377588
    60
    EP 60 simulation finished, start learning
    5.0
    EP 60 learning finished: 0.06952483398652376, 0.03219129933052519
    61
    EP 61 simulation finished, start learning
    4.1
    EP 61 learning finished: 0.11014530697017832, 0.040179372237550005
    62
    EP 62 simulation finished, start learning
    4.1
    EP 62 learning finished: 0.16120421194350243, 0.04768260972669758
    63
    EP 63 simulation finished, start learning
    4.0
    EP 63 learning finished: 0.195921529916113, 0.07856559623083648
    64
    EP 64 simulation finished, start learning
    4.0
    EP 64 learning finished: 0.19258696312092338, 0.08877468264266786
    65
    EP 65 simulation finished, start learning
    3.2
    EP 65 learning finished: 0.19465938856585296, 0.1118767013690934
    66
    EP 66 simulation finished, start learning
    5.0
    EP 66 learning finished: 0.18870681428950675, 0.10272801737540564
    67
    EP 67 simulation finished, start learning
    3.1
    EP 67 learning finished: 0.13790220123891217, 0.08112467263528966
    68
    EP 68 simulation finished, start learning
    5.0
    EP 68 learning finished: 0.13304932115046952, 0.0685098939243245
    69
    EP 69 simulation finished, start learning
    5.0
    EP 69 learning finished: 0.06482695359907353, 0.037046548114950895
    70
    EP 70 simulation finished, start learning
    4.1
    EP 70 learning finished: 0.10473684181154752, 0.06003505280013736
    71
    EP 71 simulation finished, start learning
    4.1
    EP 71 learning finished: 0.20533502836844342, 0.06570454408104857
    72
    EP 72 simulation finished, start learning
    4.1
    EP 72 learning finished: 0.1990189377526228, 0.07049742374670369
    73
    EP 73 simulation finished, start learning
    4.1
    EP 73 learning finished: 0.38111306460800165, 0.10525462151059213
    74
    EP 74 simulation finished, start learning
    5.0
    EP 74 learning finished: 0.27066592881783436, 0.08023911644752424
    75
    EP 75 simulation finished, start learning
    5.0
    EP 75 learning finished: 0.17435722728369463, 0.0580861823607289
    76
    EP 76 simulation finished, start learning
    4.1
    EP 76 learning finished: 0.25266143629554705, 0.07027191196947946
    77
    EP 77 simulation finished, start learning
    5.0
    EP 77 learning finished: 0.08196589600981596, 0.036583646120901626
    78
    EP 78 simulation finished, start learning
    3.2
    EP 78 learning finished: 0.23863731837782787, 0.0728577466316566
    79
    EP 79 simulation finished, start learning
    4.0
    EP 79 learning finished: 0.22959837753197476, 0.08166073892870784
    80
    EP 80 simulation finished, start learning
    5.0
    EP 80 learning finished: 0.14912391542728187, 0.062433777704173465
    81
    EP 81 simulation finished, start learning
    4.1
    EP 81 learning finished: 0.1454876265010352, 0.06652641710670171
    82
    EP 82 simulation finished, start learning
    4.0
    EP 82 learning finished: 0.12568339612851886, 0.04179942508897957
    83
    EP 83 simulation finished, start learning
    2.1
    EP 83 learning finished: 0.11940460773514161, 0.05728390138814573
    84
    EP 84 simulation finished, start learning
    4.1
    EP 84 learning finished: 0.11372074268744115, 0.06092299192235758
    85
    EP 85 simulation finished, start learning
    3.0
    EP 85 learning finished: 0.1600192989820748, 0.06265356764818453
    86
    EP 86 simulation finished, start learning
    5.0
    EP 86 learning finished: 0.10050932090193854, 0.05277064987545099
    87
    EP 87 simulation finished, start learning
    3.1
    EP 87 learning finished: 0.09610188086986327, 0.0404024348420838
    88
    EP 88 simulation finished, start learning
    5.0
    EP 88 learning finished: 0.09247680621384902, 0.04594813826797761
    89
    EP 89 simulation finished, start learning
    3.1
    EP 89 learning finished: 0.13785887924382045, 0.060825011159401456
    90
    EP 90 simulation finished, start learning
    5.0
    EP 90 learning finished: 0.1344736223955696, 0.059440395056982766
    91
    EP 91 simulation finished, start learning
    5.0
    EP 91 learning finished: 0.13081306901966983, 0.04702664205144558
    92
    EP 92 simulation finished, start learning
    3.0
    EP 92 learning finished: 0.08141551549260484, 0.042354504752609934
    93
    EP 93 simulation finished, start learning
    4.1
    EP 93 learning finished: 0.04250122928699085, 0.025218248709457807
    94
    EP 94 simulation finished, start learning
    5.0
    EP 94 learning finished: 0.04185581036587038, 0.024928798995109247
    95
    EP 95 simulation finished, start learning
    3.1
    EP 95 learning finished: 0.07627584651693553, 0.0455156766314051
    96
    EP 96 simulation finished, start learning
    4.1
    EP 96 learning finished: 0.08821325538889369, 0.04360657236406418
    97
    EP 97 simulation finished, start learning
    5.0
    EP 97 learning finished: 0.06329891570175583, 0.03858154594969113
    98
    EP 98 simulation finished, start learning
    3.2
    EP 98 learning finished: 0.12473088953002431, 0.05682327387772427
    99
    EP 99 simulation finished, start learning
    3.1
    EP 99 learning finished: 0.11007855722125726, 0.04594036126721276
    100
    EP 100 simulation finished, start learning
    5.0
    EP 100 learning finished: 0.08071450363343402, 0.04110050631134706
    101
    EP 101 simulation finished, start learning
    5.0
    EP 101 learning finished: 0.07908122790358618, 0.0406345140085295
    102
    EP 102 simulation finished, start learning
    2.0
    EP 102 learning finished: 0.10245658421184345, 0.033006635412998385
    103
    EP 103 simulation finished, start learning
    4.0
    EP 103 learning finished: 0.09695172765214721, 0.02793071764779557
    104
    EP 104 simulation finished, start learning
    5.0
    EP 104 learning finished: 0.09232155656691887, 0.027026664798311728
    105
    EP 105 simulation finished, start learning
    5.0
    EP 105 learning finished: 0.0883526919141886, 0.026235336749226807
    106
    EP 106 simulation finished, start learning
    3.0
    EP 106 learning finished: 0.05656335191575579, 0.02111448473699736
    107
    EP 107 simulation finished, start learning
    3.1
    EP 107 learning finished: 0.055111852431629416, 0.02486293631929352
    108
    EP 108 simulation finished, start learning
    3.2
    EP 108 learning finished: 0.12243728826982192, 0.05235626595294285
    109
    EP 109 simulation finished, start learning
    4.1
    EP 109 learning finished: 0.17932812529977937, 0.06490860728353454
    110
    EP 110 simulation finished, start learning
    4.1
    EP 110 learning finished: 0.17364950524316924, 0.0630614634996256
    111
    EP 111 simulation finished, start learning
    4.1
    EP 111 learning finished: 0.225083246450696, 0.06656042576566268
    112
    EP 112 simulation finished, start learning
    2.2
    EP 112 learning finished: 0.16415016757540712, 0.059951589914480984
    113
    EP 113 simulation finished, start learning
    4.1
    EP 113 learning finished: 0.1121545507219821, 0.049994238511210064
    114
    EP 114 simulation finished, start learning
    5.0
    EP 114 learning finished: 0.10551463430937957, 0.04575225378195569
    115
    EP 115 simulation finished, start learning
    4.1
    EP 115 learning finished: 0.08510449319472357, 0.03670681433346464
    116
    EP 116 simulation finished, start learning
    3.1
    EP 116 learning finished: 0.06255310599480168, 0.024635253802012258
    117
    EP 117 simulation finished, start learning
    4.1
    EP 117 learning finished: 0.06164492318907122, 0.02437138000339978
    118
    EP 118 simulation finished, start learning
    4.0
    EP 118 learning finished: 0.060580357728166945, 0.027804050596050917
    119
    EP 119 simulation finished, start learning
    4.1
    EP 119 learning finished: 0.05963007706774803, 0.03599105072278164
    120
    EP 120 simulation finished, start learning
    4.1
    EP 120 learning finished: 0.05879198803771857, 0.031953570376459815
    121
    EP 121 simulation finished, start learning
    4.0
    EP 121 learning finished: 0.05237763955768138, 0.0322141151995643
    122
    EP 122 simulation finished, start learning
    4.1
    EP 122 learning finished: 0.05723848662064768, 0.03131521136504389
    123
    EP 123 simulation finished, start learning
    3.1
    EP 123 learning finished: 0.056475091204728045, 0.026097524356354523
    124
    EP 124 simulation finished, start learning
    4.0
    EP 124 learning finished: 0.0653308706522786, 0.02619818531371661
    125
    EP 125 simulation finished, start learning
    5.0
    EP 125 learning finished: 0.0425174680770688, 0.02252688404443859
    126
    EP 126 simulation finished, start learning
    4.1
    EP 126 learning finished: 0.04162401218852452, 0.022244580055094332
    127
    EP 127 simulation finished, start learning
    2.0
    EP 127 learning finished: 0.22872429800067984, 0.05983853537116968
    128
    EP 128 simulation finished, start learning
    3.1
    EP 128 learning finished: 0.2244947226437306, 0.06158264998274173
    129
    EP 129 simulation finished, start learning
    3.0
    EP 129 learning finished: 0.21984830231380492, 0.06344783754610155
    130
    EP 130 simulation finished, start learning
    4.1
    EP 130 learning finished: 0.21552762512592682, 0.0794479862762573
    131
    EP 131 simulation finished, start learning
    5.0
    EP 131 learning finished: 0.10432978952790961, 0.03760230713939746
    132
    EP 132 simulation finished, start learning
    4.0
    EP 132 learning finished: 0.10253608769670919, 0.0344789575328317
    133
    EP 133 simulation finished, start learning
    4.1
    EP 133 learning finished: 0.1499128650805197, 0.04186367445846469
    134
    EP 134 simulation finished, start learning
    5.0
    EP 134 learning finished: 0.04893194267061318, 0.0227977235944478
    135
    EP 135 simulation finished, start learning
    4.1
    EP 135 learning finished: 0.048660471696369, 0.02523987363609888
    136
    EP 136 simulation finished, start learning
    4.1
    EP 136 learning finished: 0.05017294196511557, 0.024856701046195678
    137
    EP 137 simulation finished, start learning
    3.0
    EP 137 learning finished: 0.03305281523779158, 0.017095508256379805
    138
    EP 138 simulation finished, start learning
    2.1
    EP 138 learning finished: 0.04905653325915793, 0.029941867862185962
    139
    EP 139 simulation finished, start learning
    4.1
    EP 139 learning finished: 0.09503246676639776, 0.03724264035025904
    140
    EP 140 simulation finished, start learning
    2.1
    EP 140 learning finished: 0.1400029268659324, 0.04679786200772066
    141
    EP 141 simulation finished, start learning
    3.1
    EP 141 learning finished: 0.13642515643643094, 0.04815937448698845
    142
    EP 142 simulation finished, start learning
    4.1
    EP 142 learning finished: 0.13329933048930087, 0.044773747268579335
    143
    EP 143 simulation finished, start learning
    3.2
    EP 143 learning finished: 0.13014677212529335, 0.04616151786200599
    144
    EP 144 simulation finished, start learning
    5.0
    EP 144 learning finished: 0.08502933706556703, 0.0339094925768831
    145
    EP 145 simulation finished, start learning
    3.1
    EP 145 learning finished: 0.0843073446276108, 0.0335415991060426
    146
    EP 146 simulation finished, start learning
    5.0
    EP 146 learning finished: 0.042745631156215325, 0.024276605212044444
    147
    EP 147 simulation finished, start learning
    4.1
    EP 147 learning finished: 0.028209311738759624, 0.01531707181208534
    148
    EP 148 simulation finished, start learning
    4.1
    EP 148 learning finished: 0.041955112798063965, 0.01730949102361971
    149
    EP 149 simulation finished, start learning
    4.1
    EP 149 learning finished: 0.041421007319542014, 0.01480970095432473
    150
    EP 150 simulation finished, start learning
    3.0
    EP 150 learning finished: 0.04104192754181568, 0.01703146327339411
    151
    EP 151 simulation finished, start learning
    4.1
    EP 151 learning finished: 0.04203394493040946, 0.023684836429263
    152
    EP 152 simulation finished, start learning
    3.0
    EP 152 learning finished: 0.041812847371097916, 0.023823304890556212
    153
    EP 153 simulation finished, start learning
    4.0
    EP 153 learning finished: 0.04463904738585711, 0.02386622431825754
    154
    EP 154 simulation finished, start learning
    3.1
    EP 154 learning finished: 0.043629818968447864, 0.025605423757064944
    155
    EP 155 simulation finished, start learning
    4.1
    EP 155 learning finished: 0.042531491836008684, 0.01866302693993956
    156
    EP 156 simulation finished, start learning
    4.1
    EP 156 learning finished: 0.03927717591949431, 0.018307669563297917
    157
    EP 157 simulation finished, start learning
    5.0
    EP 157 learning finished: 0.03883853230626144, 0.016036637473921072
    158
    EP 158 simulation finished, start learning
    4.0
    EP 158 learning finished: 0.02563550187461061, 0.013993839528026205
    159
    EP 159 simulation finished, start learning
    4.1
    EP 159 learning finished: 0.025479298634248693, 0.01389639449626697
    160
    EP 160 simulation finished, start learning
    4.0
    EP 160 learning finished: 0.22315841587906426, 0.05081577893808831
    161
    EP 161 simulation finished, start learning
    4.1
    EP 161 learning finished: 0.21875877552995515, 0.05190562071043958
    162
    EP 162 simulation finished, start learning
    3.2
    EP 162 learning finished: 0.2147361958986842, 0.05955562653026606
    163
    EP 163 simulation finished, start learning
    4.0
    EP 163 learning finished: 0.2107109200942192, 0.058881671443493655
    164
    EP 164 simulation finished, start learning
    2.1
    EP 164 learning finished: 0.08294806553206158, 0.032489096862802214
    165
    EP 165 simulation finished, start learning
    4.1
    EP 165 learning finished: 0.12226506615310129, 0.038629036887810386
    166
    EP 166 simulation finished, start learning
    5.0
    EP 166 learning finished: 0.07997434380821389, 0.028100286191899453
    167
    EP 167 simulation finished, start learning
    5.0
    EP 167 learning finished: 0.07861491208684157, 0.02577882131258254
    168
    EP 168 simulation finished, start learning
    3.1
    EP 168 learning finished: 0.20827306279501023, 0.05977914710999036
    169
    EP 169 simulation finished, start learning
    5.0
    EP 169 learning finished: 0.20443419153194942, 0.050988212961141276
    170
    EP 170 simulation finished, start learning
    4.0
    EP 170 learning finished: 0.20049318154963913, 0.052264011537688525
    171
    EP 171 simulation finished, start learning
    5.0
    EP 171 learning finished: 0.1968782047317701, 0.05155144159786499
    172
    EP 172 simulation finished, start learning
    5.0
    EP 172 learning finished: 0.022538703165764673, 0.009346293528860627
    173
    EP 173 simulation finished, start learning
    5.0
    EP 173 learning finished: 0.022366374776439102, 0.009302913524167048
    174
    EP 174 simulation finished, start learning
    4.1
    EP 174 learning finished: 0.021142955407434272, 0.008932251604577482
    175
    EP 175 simulation finished, start learning
    4.1
    EP 175 learning finished: 0.022593251489890775, 0.010574325893768943
    176
    EP 176 simulation finished, start learning
    3.0
    EP 176 learning finished: 0.022507660374948557, 0.012561451787813501
    177
    EP 177 simulation finished, start learning
    4.1
    EP 177 learning finished: 0.033480529028416584, 0.014145707155562329
    178
    EP 178 simulation finished, start learning
    3.2
    EP 178 learning finished: 0.044088712554668064, 0.02186156553200706
    179
    EP 179 simulation finished, start learning
    5.0
    EP 179 learning finished: 0.03790864764140167, 0.020079167137919757
    180
    EP 180 simulation finished, start learning
    5.0
    EP 180 learning finished: 0.07509023925360125, 0.023991337140683317
    181
    EP 181 simulation finished, start learning
    3.2
    EP 181 learning finished: 0.11059413606378143, 0.03124445539205034
    182
    EP 182 simulation finished, start learning
    4.1
    EP 182 learning finished: 0.0725783018815811, 0.02337484222873994
    183
    EP 183 simulation finished, start learning
    4.1
    EP 183 learning finished: 0.07151174841621, 0.02466204932655695
    184
    EP 184 simulation finished, start learning
    5.0
    EP 184 learning finished: 0.04110585577420878, 0.018680782132167433
    185
    EP 185 simulation finished, start learning
    4.1
    EP 185 learning finished: 0.03051023085746607, 0.011319249833672274
    186
    EP 186 simulation finished, start learning
    5.0
    EP 186 learning finished: 0.020178823058380146, 0.009731594927099962
    187
    EP 187 simulation finished, start learning
    4.0
    EP 187 learning finished: 0.019652225839507764, 0.010309880518025949
    188
    EP 188 simulation finished, start learning
    5.0
    EP 188 learning finished: 0.019506557116883982, 0.010250857538491568
    189
    EP 189 simulation finished, start learning
    1.1
    EP 189 learning finished: 0.04047817947766674, 0.014354278255979716
    190
    EP 190 simulation finished, start learning
    5.0
    EP 190 learning finished: 0.03968968402524453, 0.014186797255071918
    191
    EP 191 simulation finished, start learning
    4.1
    EP 191 learning finished: 0.02593864187619843, 0.01351839705078047
    192
    EP 192 simulation finished, start learning
    5.0
    EP 192 learning finished: 0.025603400748110516, 0.01340831406113511
    193
    EP 193 simulation finished, start learning
    4.1
    EP 193 learning finished: 0.03581578358371473, 0.015229365941256864
    194
    EP 194 simulation finished, start learning
    4.1
    EP 194 learning finished: 0.03559380719028482, 0.016613830955982678
    195
    EP 195 simulation finished, start learning
    5.0
    EP 195 learning finished: 0.03541318301829577, 0.015045693568280191
    196
    EP 196 simulation finished, start learning
    4.1
    EP 196 learning finished: 0.0702136030719025, 0.022099063591560626
    197
    EP 197 simulation finished, start learning
    5.0
    EP 197 learning finished: 0.03476642825214782, 0.014828756190554534
    198
    EP 198 simulation finished, start learning
    5.0
    EP 198 learning finished: 0.034517935595385296, 0.0133017895031499
    199
    EP 199 simulation finished, start learning
    5.0
    EP 199 learning finished: 0.03423500169706273, 0.013220119011133255



```python
q_one_over_t.q
```




                0          1          2
    -1   5.711105  18.870838  39.215383
     1  28.630627  34.099738  24.283151




```python
v_l, pi_l= q_one_over_t.q.yield_v_pi()
```


```python
v_l
```




    0    28.630627
    1    34.099738
    2    39.215383
    dtype: float64




```python
pi_l
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
# 1 action delay, 0 obs. delay

class delayed_wrapper:
    def __init__(self, env, delay_steps):
        self.env = env
        self.num = delay_steps
        self.state_dim = env.state_dim
        self.action_dim = env.action_dim
        self.action_buffer = [-1]*delay_steps
        
    def step(self, action):
        app_action = self.action_buffer.pop(0)
        self.action_buffer.append(action)
        return  self.env.step(app_action)
    
    def reset(self, random = False):
        self.action_buffer = [0]*self.num
        return self.env.reset(random)
    

```


```python
# Q-learning using model, async reward collection


class Q_learning:
    def __init__(self, q, env, delay_step, model, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(40)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        model = self.model
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            for _ in range(10):
                
                est_obs = model(obs, self.action_memory)
                last_action = self.action_memory.pop(0)
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (q.max_value_act(est_obs)[1], False)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q
```


```python
mu1 = np.argmax(tu1.values,axis=1)
mun1 = np.argmax(tun1.values,axis=1)
```


```python

def model_model(x, u):
    assert type(u)==list
    u = u[0]
    m = mu1 
    if u == 1:
        m = mu1
    elif u == -1:
        m = mun1
    else:
        raise ValueError
    return m[x]
```


```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
# lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_model = Q_learning(q, env, 1, model_model, MAX_ITER = 1000)
learned_q = q_model.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    0.3
    EP 3 learning finished: 10.491779817708332, 3.353074275173611
    4
    EP 4 simulation finished, start learning
    0.4
    EP 4 learning finished: 1.0948317696902343, 0.6247967056367255
    5
    EP 5 simulation finished, start learning
    1.4
    EP 5 learning finished: 6.0034744249761065, 1.3239661593389365
    6
    EP 6 simulation finished, start learning
    0.3
    EP 6 learning finished: 0.7402245690214073, 0.34318380620871
    7
    EP 7 simulation finished, start learning
    1.4
    EP 7 learning finished: 1.2018972749264982, 0.309976692894988
    8
    EP 8 simulation finished, start learning
    3.2
    EP 8 learning finished: 12.884762076328155, 2.66800005049082
    9
    EP 9 simulation finished, start learning
    0.0
    EP 9 learning finished: 1.8775275037056272, 0.7612175600775687
    10
    EP 10 simulation finished, start learning
    3.0
    EP 10 learning finished: 1.719995331593779, 0.7265500585911252
    11
    EP 11 simulation finished, start learning
    -1.0
    EP 11 learning finished: 1.4409922557199444, 0.49218909567511293
    12
    EP 12 simulation finished, start learning
    3.1
    EP 12 learning finished: 1.0476704586663779, 0.4691813667508324
    13
    EP 13 simulation finished, start learning
    4.0
    EP 13 learning finished: 0.932967874794052, 0.41906290711662963
    14
    EP 14 simulation finished, start learning
    3.1
    EP 14 learning finished: 0.7092027770574845, 0.39176651845523125
    15
    EP 15 simulation finished, start learning
    3.0
    EP 15 learning finished: 0.6820661870663187, 0.36653416998833527
    16
    EP 16 simulation finished, start learning
    4.0
    EP 16 learning finished: 0.6193980911068673, 0.3440470170693339
    17
    EP 17 simulation finished, start learning
    5.0
    EP 17 learning finished: 0.5519826493708884, 0.28450003934403406
    18
    EP 18 simulation finished, start learning
    4.1
    EP 18 learning finished: 0.5112051404086451, 0.2820644569243906
    19
    EP 19 simulation finished, start learning
    4.0
    EP 19 learning finished: 0.44297350024257653, 0.2512110596317845
    20
    EP 20 simulation finished, start learning
    3.0
    EP 20 learning finished: 0.3910402911335211, 0.2389035401748599
    21
    EP 21 simulation finished, start learning
    3.0
    EP 21 learning finished: 0.41723108403879117, 0.23359810708344733
    22
    EP 22 simulation finished, start learning
    3.0
    EP 22 learning finished: 0.3052684992999879, 0.1961430973997594
    23
    EP 23 simulation finished, start learning
    4.0
    EP 23 learning finished: 0.2822782184744561, 0.2017751837491543
    24
    EP 24 simulation finished, start learning
    3.0
    EP 24 learning finished: 0.2604603360590936, 0.1931275270667463
    25
    EP 25 simulation finished, start learning
    3.0
    EP 25 learning finished: 0.39157424193681223, 0.21926027385431812
    26
    EP 26 simulation finished, start learning
    4.1
    EP 26 learning finished: 0.37738097229726986, 0.23283635067190214
    27
    EP 27 simulation finished, start learning
    5.0
    EP 27 learning finished: 0.5040818180857727, 0.2250677865185482
    28
    EP 28 simulation finished, start learning
    5.0
    EP 28 learning finished: 0.5894897646168271, 0.21624339849474605
    29
    EP 29 simulation finished, start learning
    5.0
    EP 29 learning finished: 0.6645817612210188, 0.19671084450864149
    30
    EP 30 simulation finished, start learning
    5.0
    EP 30 learning finished: 0.4657214646463288, 0.14366054030060896
    31
    EP 31 simulation finished, start learning
    3.1
    EP 31 learning finished: 0.2975795525170266, 0.1488650627441146
    32
    EP 32 simulation finished, start learning
    3.1
    EP 32 learning finished: 0.33741000715084724, 0.16904898557256423
    33
    EP 33 simulation finished, start learning
    3.0
    EP 33 learning finished: 0.323960393924164, 0.16455033951512035
    34
    EP 34 simulation finished, start learning
    2.1
    EP 34 learning finished: 0.4117206821454449, 0.17790047322741698
    35
    EP 35 simulation finished, start learning
    4.0
    EP 35 learning finished: 0.35112147633043556, 0.17014768454055496
    36
    EP 36 simulation finished, start learning
    3.0
    EP 36 learning finished: 0.17153144907647544, 0.12459936656718955
    37
    EP 37 simulation finished, start learning
    2.1
    EP 37 learning finished: 0.1952022421440418, 0.13782108815591312
    38
    EP 38 simulation finished, start learning
    0.1
    EP 38 learning finished: 0.3285228220838796, 0.16541241695916073
    39
    EP 39 simulation finished, start learning
    2.1
    EP 39 learning finished: 0.2751693780244686, 0.15218589028357066
    40
    EP 40 simulation finished, start learning
    4.0
    EP 40 learning finished: 0.3116533731886584, 0.17017992564286377
    41
    EP 41 simulation finished, start learning
    3.0
    EP 41 learning finished: 0.3021076714099209, 0.1506017905594431
    42
    EP 42 simulation finished, start learning
    3.0
    EP 42 learning finished: 0.2165117218031849, 0.12209932055411461
    43
    EP 43 simulation finished, start learning
    3.0
    EP 43 learning finished: 0.19802110651160376, 0.10560208712646091
    44
    EP 44 simulation finished, start learning
    4.0
    EP 44 learning finished: 0.19438746747654534, 0.10324337826866614
    45
    EP 45 simulation finished, start learning
    3.0
    EP 45 learning finished: 0.1418099448821497, 0.08489748036692173
    46
    EP 46 simulation finished, start learning
    3.0
    EP 46 learning finished: 0.14006668230276098, 0.08337574007581654
    47
    EP 47 simulation finished, start learning
    4.0
    EP 47 learning finished: 0.27270187456298345, 0.10151144247865733
    48
    EP 48 simulation finished, start learning
    3.0
    EP 48 learning finished: 0.13410154408543917, 0.08001761359660027
    49
    EP 49 simulation finished, start learning
    3.0
    EP 49 learning finished: 0.18666501371228073, 0.09444166379602208
    50
    EP 50 simulation finished, start learning
    4.0
    EP 50 learning finished: 0.25783826243077357, 0.09578620892100265
    51
    EP 51 simulation finished, start learning
    3.0
    EP 51 learning finished: 0.12649627025123245, 0.07556228672255709
    52
    EP 52 simulation finished, start learning
    3.0
    EP 52 learning finished: 0.12484860481776927, 0.07415067906268795
    53
    EP 53 simulation finished, start learning
    3.0
    EP 53 learning finished: 0.12342457979982058, 0.07304651340053396
    54
    EP 54 simulation finished, start learning
    4.0
    EP 54 learning finished: 0.12156480717375118, 0.07181914512661525
    55
    EP 55 simulation finished, start learning
    3.0
    EP 55 learning finished: 0.17829370974929404, 0.08596208226917934
    56
    EP 56 simulation finished, start learning
    2.0
    EP 56 learning finished: 0.26166632674183, 0.10185824766382723
    57
    EP 57 simulation finished, start learning
    5.0
    EP 57 learning finished: 0.17069407637960943, 0.08363236817546878
    58
    EP 58 simulation finished, start learning
    4.0
    EP 58 learning finished: 0.16780007970854705, 0.07713936011625044
    59
    EP 59 simulation finished, start learning
    1.1
    EP 59 learning finished: 0.17505299850500933, 0.09240668104185328
    60
    EP 60 simulation finished, start learning
    4.1
    EP 60 learning finished: 0.25804882486105996, 0.09831778156901194
    61
    EP 61 simulation finished, start learning
    4.0
    EP 61 learning finished: 0.3226572831908321, 0.10770235492317178
    62
    EP 62 simulation finished, start learning
    3.0
    EP 62 learning finished: 0.23462205498692512, 0.1073500444619994
    63
    EP 63 simulation finished, start learning
    3.0
    EP 63 learning finished: 0.15845000609319548, 0.08975076407849336
    64
    EP 64 simulation finished, start learning
    3.0
    EP 64 learning finished: 0.15143957360082183, 0.06593425798353447
    65
    EP 65 simulation finished, start learning
    3.0
    EP 65 learning finished: 0.14802085576093837, 0.05501751494381768
    66
    EP 66 simulation finished, start learning
    4.0
    EP 66 learning finished: 0.11969626708192749, 0.060312745208371155
    67
    EP 67 simulation finished, start learning
    3.0
    EP 67 learning finished: 0.11800294515366083, 0.04727033214641067
    68
    EP 68 simulation finished, start learning
    3.0
    EP 68 learning finished: 0.11692873275006122, 0.04654573817926414
    69
    EP 69 simulation finished, start learning
    3.0
    EP 69 learning finished: 0.2284821007437472, 0.06494632492567949
    70
    EP 70 simulation finished, start learning
    5.0
    EP 70 learning finished: 0.11179739898381591, 0.04568875769653117
    71
    EP 71 simulation finished, start learning
    5.0
    EP 71 learning finished: 0.11078412234488866, 0.044009067136531854
    72
    EP 72 simulation finished, start learning
    4.0
    EP 72 learning finished: 0.10908635543624001, 0.05357850162949814
    73
    EP 73 simulation finished, start learning
    3.0
    EP 73 learning finished: 0.07290253581921213, 0.04709204267438979
    74
    EP 74 simulation finished, start learning
    3.0
    EP 74 learning finished: 0.14441002485536636, 0.06004267945711028
    75
    EP 75 simulation finished, start learning
    3.0
    EP 75 learning finished: 0.14204921432053474, 0.06045360911251644
    76
    EP 76 simulation finished, start learning
    3.0
    EP 76 learning finished: 0.2092979411022, 0.06153332195463612
    77
    EP 77 simulation finished, start learning
    4.0
    EP 77 learning finished: 0.1367374528170835, 0.06532565087458526
    78
    EP 78 simulation finished, start learning
    4.0
    EP 78 learning finished: 0.21495492276591754, 0.0691261650912504
    79
    EP 79 simulation finished, start learning
    3.0
    EP 79 learning finished: 0.20882897996504646, 0.06773461550409572
    80
    EP 80 simulation finished, start learning
    3.0
    EP 80 learning finished: 0.20342571304545132, 0.0550796922853003
    81
    EP 81 simulation finished, start learning
    3.0
    EP 81 learning finished: 0.19781276839098894, 0.05579716267249483
    82
    EP 82 simulation finished, start learning
    3.1
    EP 82 learning finished: 0.14876044768932672, 0.06255060422881738
    83
    EP 83 simulation finished, start learning
    4.0
    EP 83 learning finished: 0.19056825941208366, 0.07577247518935426
    84
    EP 84 simulation finished, start learning
    4.1
    EP 84 learning finished: 0.21510541787799298, 0.09338832619075532
    85
    EP 85 simulation finished, start learning
    5.0
    EP 85 learning finished: 0.2081003802818273, 0.07496896766430024
    86
    EP 86 simulation finished, start learning
    5.0
    EP 86 learning finished: 0.11057103756082398, 0.061453513381321066
    87
    EP 87 simulation finished, start learning
    5.0
    EP 87 learning finished: 0.10900605477896619, 0.0463434522010262
    88
    EP 88 simulation finished, start learning
    5.0
    EP 88 learning finished: 0.10774660140492642, 0.034743849297554995
    89
    EP 89 simulation finished, start learning
    4.1
    EP 89 learning finished: 0.10638143476272433, 0.04543064825731472
    90
    EP 90 simulation finished, start learning
    5.0
    EP 90 learning finished: 0.06845618057702652, 0.03658058689290004
    91
    EP 91 simulation finished, start learning
    4.0
    EP 91 learning finished: 0.12834787139644988, 0.04600039451953197
    92
    EP 92 simulation finished, start learning
    3.0
    EP 92 learning finished: 0.1257938942658221, 0.04928975184217865
    93
    EP 93 simulation finished, start learning
    3.0
    EP 93 learning finished: 0.13039623440507775, 0.05003973678685222
    94
    EP 94 simulation finished, start learning
    2.0
    EP 94 learning finished: 0.19232871270193108, 0.06295709468144899
    95
    EP 95 simulation finished, start learning
    5.0
    EP 95 learning finished: 0.18830853593521368, 0.052624035544778756
    96
    EP 96 simulation finished, start learning
    4.0
    EP 96 learning finished: 0.12330176351006727, 0.04960152919314057
    97
    EP 97 simulation finished, start learning
    4.0
    EP 97 learning finished: 0.09207869880597208, 0.052779055652935725
    98
    EP 98 simulation finished, start learning
    3.1
    EP 98 learning finished: 0.18094226149964499, 0.06399220618891348
    99
    EP 99 simulation finished, start learning
    4.0
    EP 99 learning finished: 0.2626386912779406, 0.0767309206407664
    100
    EP 100 simulation finished, start learning
    2.0
    EP 100 learning finished: 0.25296400946708886, 0.07891531808449355
    101
    EP 101 simulation finished, start learning
    4.1
    EP 101 learning finished: 0.16346845280965816, 0.07380474749487487
    102
    EP 102 simulation finished, start learning
    4.1
    EP 102 learning finished: 0.11767884974362985, 0.06735679115496485
    103
    EP 103 simulation finished, start learning
    3.0
    EP 103 learning finished: 0.1665832520070225, 0.06322415787004647
    104
    EP 104 simulation finished, start learning
    5.0
    EP 104 learning finished: 0.16155697024698412, 0.057306348516901195
    105
    EP 105 simulation finished, start learning
    5.0
    EP 105 learning finished: 0.14926416874192938, 0.054622459373199796
    106
    EP 106 simulation finished, start learning
    5.0
    EP 106 learning finished: 0.1455281563737465, 0.04476028884521691
    107
    EP 107 simulation finished, start learning
    5.0
    EP 107 learning finished: 0.17658191268793644, 0.04209134938444947
    108
    EP 108 simulation finished, start learning
    3.0
    EP 108 learning finished: 0.1384948898187801, 0.04485230193282774
    109
    EP 109 simulation finished, start learning
    3.0
    EP 109 learning finished: 0.11423885102452047, 0.0476628641920532
    110
    EP 110 simulation finished, start learning
    3.0
    EP 110 learning finished: 0.16867040756078922, 0.05037126784853463
    111
    EP 111 simulation finished, start learning
    4.0
    EP 111 learning finished: 0.16538586954512624, 0.05624153435168585
    112
    EP 112 simulation finished, start learning
    3.0
    EP 112 learning finished: 0.16230362935093368, 0.055464033343805084
    113
    EP 113 simulation finished, start learning
    5.0
    EP 113 learning finished: 0.10652434042696157, 0.04517645556055161
    114
    EP 114 simulation finished, start learning
    3.0
    EP 114 learning finished: 0.0785375064295124, 0.04417360033123844
    115
    EP 115 simulation finished, start learning
    4.0
    EP 115 learning finished: 0.09708068316834328, 0.03891339147847125
    116
    EP 116 simulation finished, start learning
    2.1
    EP 116 learning finished: 0.14735304871528498, 0.04678403074758878
    117
    EP 117 simulation finished, start learning
    3.0
    EP 117 learning finished: 0.1434077112143548, 0.05525827630193151
    118
    EP 118 simulation finished, start learning
    3.0
    EP 118 learning finished: 0.1537795320176336, 0.055180290442736535
    119
    EP 119 simulation finished, start learning
    4.0
    EP 119 learning finished: 0.15155907838896354, 0.05987926517668137
    120
    EP 120 simulation finished, start learning
    3.0
    EP 120 learning finished: 0.0981836880018605, 0.049444241467655324
    121
    EP 121 simulation finished, start learning
    2.1
    EP 121 learning finished: 0.09743776146031635, 0.056502804056636435
    122
    EP 122 simulation finished, start learning
    4.0
    EP 122 learning finished: 0.09234404607920155, 0.05428591802399474
    123
    EP 123 simulation finished, start learning
    3.0
    EP 123 learning finished: 0.09078830039387142, 0.041886696465346894
    124
    EP 124 simulation finished, start learning
    3.0
    EP 124 learning finished: 0.13067188332948731, 0.04225723892136498
    125
    EP 125 simulation finished, start learning
    3.1
    EP 125 learning finished: 0.16754288038699627, 0.03986877598246134
    126
    EP 126 simulation finished, start learning
    2.0
    EP 126 learning finished: 0.12300954595755087, 0.03411286849984959
    127
    EP 127 simulation finished, start learning
    5.0
    EP 127 learning finished: 0.12001173238400753, 0.03300185405046877
    128
    EP 128 simulation finished, start learning
    3.0
    EP 128 learning finished: 0.07985138869760533, 0.026570821318306532
    129
    EP 129 simulation finished, start learning
    2.0
    EP 129 learning finished: 0.04182516098799738, 0.02285570043155764
    130
    EP 130 simulation finished, start learning
    5.0
    EP 130 learning finished: 0.04147365689766147, 0.021490991815075706
    131
    EP 131 simulation finished, start learning
    4.0
    EP 131 learning finished: 0.07767506454689865, 0.027152883683466317
    132
    EP 132 simulation finished, start learning
    2.0
    EP 132 learning finished: 0.07644742737135246, 0.03507237901179483
    133
    EP 133 simulation finished, start learning
    3.1
    EP 133 learning finished: 0.11031257361818092, 0.037719038101903024
    134
    EP 134 simulation finished, start learning
    2.0
    EP 134 learning finished: 0.10781962333365769, 0.04589394857227269
    135
    EP 135 simulation finished, start learning
    2.1
    EP 135 learning finished: 0.10950389039131636, 0.05222035283622972
    136
    EP 136 simulation finished, start learning
    3.0
    EP 136 learning finished: 0.10709010127799345, 0.050434319170954346
    137
    EP 137 simulation finished, start learning
    2.1
    EP 137 learning finished: 0.13176818298056503, 0.051848795988683825
    138
    EP 138 simulation finished, start learning
    3.0
    EP 138 learning finished: 0.08795629237918945, 0.04320307609291696
    139
    EP 139 simulation finished, start learning
    4.0
    EP 139 learning finished: 0.08379869086180136, 0.03607497021998244
    140
    EP 140 simulation finished, start learning
    4.0
    EP 140 learning finished: 0.07568176712062424, 0.0409491333346755
    141
    EP 141 simulation finished, start learning
    3.0
    EP 141 learning finished: 0.07493127852695025, 0.028097197876062314
    142
    EP 142 simulation finished, start learning
    3.0
    EP 142 learning finished: 0.14731275444925274, 0.04017780453148904
    143
    EP 143 simulation finished, start learning
    5.0
    EP 143 learning finished: 0.144431983318686, 0.03482773130206004
    144
    EP 144 simulation finished, start learning
    5.0
    EP 144 learning finished: 0.07105044309013131, 0.027587726915958122
    145
    EP 145 simulation finished, start learning
    4.1
    EP 145 learning finished: 0.07038253835887609, 0.0375530088350251
    146
    EP 146 simulation finished, start learning
    5.0
    EP 146 learning finished: 0.08933098123127792, 0.02988225003083163
    147
    EP 147 simulation finished, start learning
    4.0
    EP 147 learning finished: 0.08827630174411283, 0.03432874269347567
    148
    EP 148 simulation finished, start learning
    3.0
    EP 148 learning finished: 0.06336719094946375, 0.02963245954506455
    149
    EP 149 simulation finished, start learning
    3.0
    EP 149 learning finished: 0.03902656569522023, 0.019409837678226793
    150
    EP 150 simulation finished, start learning
    2.1
    EP 150 learning finished: 0.062039346276062446, 0.027368719478794528
    151
    EP 151 simulation finished, start learning
    3.0
    EP 151 learning finished: 0.08254667569341478, 0.029528962047820666
    152
    EP 152 simulation finished, start learning
    2.1
    EP 152 learning finished: 0.08463820038478076, 0.03463692378742363
    153
    EP 153 simulation finished, start learning
    4.0
    EP 153 learning finished: 0.0833812664083311, 0.04550317474638573
    154
    EP 154 simulation finished, start learning
    3.0
    EP 154 learning finished: 0.06986128957785454, 0.03328212309163211
    155
    EP 155 simulation finished, start learning
    3.0
    EP 155 learning finished: 0.0690896665227676, 0.026630850783065558
    156
    EP 156 simulation finished, start learning
    4.0
    EP 156 learning finished: 0.13626534044468386, 0.031964906666323976
    157
    EP 157 simulation finished, start learning
    3.0
    EP 157 learning finished: 0.06712314860440216, 0.021172901156845114
    158
    EP 158 simulation finished, start learning
    4.0
    EP 158 learning finished: 0.06649730682281785, 0.02516866385622703
    159
    EP 159 simulation finished, start learning
    4.0
    EP 159 learning finished: 0.13101029738568215, 0.03536984263936205
    160
    EP 160 simulation finished, start learning
    3.0
    EP 160 learning finished: 0.06449980659958854, 0.03105811612368588
    161
    EP 161 simulation finished, start learning
    3.1
    EP 161 learning finished: 0.12715002435672673, 0.04580513861449376
    162
    EP 162 simulation finished, start learning
    3.0
    EP 162 learning finished: 0.12475987179639247, 0.0474884526794502
    163
    EP 163 simulation finished, start learning
    3.0
    EP 163 learning finished: 0.07611180777971782, 0.037171347917773545
    164
    EP 164 simulation finished, start learning
    3.1
    EP 164 learning finished: 0.12117584806412651, 0.04540824493835949
    165
    EP 165 simulation finished, start learning
    4.0
    EP 165 learning finished: 0.11886499129702699, 0.03975202902875511
    166
    EP 166 simulation finished, start learning
    3.1
    EP 166 learning finished: 0.17416763761440635, 0.047693590400818074
    167
    EP 167 simulation finished, start learning
    3.0
    EP 167 learning finished: 0.16909937538624575, 0.05110619796647183
    168
    EP 168 simulation finished, start learning
    4.0
    EP 168 learning finished: 0.11001890745952991, 0.0402818013202797
    169
    EP 169 simulation finished, start learning
    2.0
    EP 169 learning finished: 0.0776865559135409, 0.03782736488805168
    170
    EP 170 simulation finished, start learning
    5.0
    EP 170 learning finished: 0.04863598244240208, 0.023491680262731407
    171
    EP 171 simulation finished, start learning
    4.0
    EP 171 learning finished: 0.0481637234680079, 0.02272581612028605
    172
    EP 172 simulation finished, start learning
    3.0
    EP 172 learning finished: 0.03444926885530819, 0.019123918309020976
    173
    EP 173 simulation finished, start learning
    3.0
    EP 173 learning finished: 0.02375513755991676, 0.01258286150024368
    174
    EP 174 simulation finished, start learning
    3.0
    EP 174 learning finished: 0.023661959209260175, 0.012812641690568801
    175
    EP 175 simulation finished, start learning
    3.1
    EP 175 learning finished: 0.05432133309659193, 0.022568136545259538
    176
    EP 176 simulation finished, start learning
    2.0
    EP 176 learning finished: 0.05391295676552588, 0.028855328330593604
    177
    EP 177 simulation finished, start learning
    5.0
    EP 177 learning finished: 0.05349320369238697, 0.02836821082988313
    178
    EP 178 simulation finished, start learning
    5.0
    EP 178 learning finished: 0.053014419070461116, 0.03185581211638677
    179
    EP 179 simulation finished, start learning
    3.0
    EP 179 learning finished: 0.06330843454380286, 0.02293091622914112
    180
    EP 180 simulation finished, start learning
    2.0
    EP 180 learning finished: 0.062309490995403394, 0.022701575754282406
    181
    EP 181 simulation finished, start learning
    4.0
    EP 181 learning finished: 0.06125697659181739, 0.026146504090280704
    182
    EP 182 simulation finished, start learning
    4.0
    EP 182 learning finished: 0.05289653770556413, 0.03113393349722493
    183
    EP 183 simulation finished, start learning
    2.1
    EP 183 learning finished: 0.05238546004657252, 0.03140532870899667
    184
    EP 184 simulation finished, start learning
    3.0
    EP 184 learning finished: 0.052015670239804024, 0.025060967827887026
    185
    EP 185 simulation finished, start learning
    3.0
    EP 185 learning finished: 0.051609346508396214, 0.02711907596326668
    186
    EP 186 simulation finished, start learning
    4.0
    EP 186 learning finished: 0.05116814061027597, 0.026574542463039325
    187
    EP 187 simulation finished, start learning
    3.0
    EP 187 learning finished: 0.05075791635225002, 0.021983730396366735
    188
    EP 188 simulation finished, start learning
    3.0
    EP 188 learning finished: 0.0503480981726554, 0.021843316614914936
    189
    EP 189 simulation finished, start learning
    4.0
    EP 189 learning finished: 0.09945449592602884, 0.024106102423347764
    190
    EP 190 simulation finished, start learning
    2.1
    EP 190 learning finished: 0.049117362391168484, 0.020537207641135424
    191
    EP 191 simulation finished, start learning
    2.0
    EP 191 learning finished: 0.048719538239478055, 0.020926630861642554
    192
    EP 192 simulation finished, start learning
    5.0
    EP 192 learning finished: 0.0483644670532577, 0.02414693154633459
    193
    EP 193 simulation finished, start learning
    2.0
    EP 193 learning finished: 0.04229759275449396, 0.017118637225112938
    194
    EP 194 simulation finished, start learning
    5.0
    EP 194 learning finished: 0.03766933108656545, 0.012362988531474883
    195
    EP 195 simulation finished, start learning
    2.0
    EP 195 learning finished: 0.05881173354776692, 0.015944327675889863
    196
    EP 196 simulation finished, start learning
    3.0
    EP 196 learning finished: 0.04013758320012428, 0.016314161232302265
    197
    EP 197 simulation finished, start learning
    3.0
    EP 197 learning finished: 0.04408687505235065, 0.01921691095352429
    198
    EP 198 simulation finished, start learning
    4.0
    EP 198 learning finished: 0.0651829130462005, 0.022436582580745323
    199
    EP 199 simulation finished, start learning
    3.0
    EP 199 learning finished: 0.06410682381839194, 0.018195838292809203
    200
    EP 200 simulation finished, start learning
    3.0
    EP 200 learning finished: 0.0421347377441208, 0.014401158616232479
    201
    EP 201 simulation finished, start learning
    3.1
    EP 201 learning finished: 0.04862151760975664, 0.022789664369824436
    202
    EP 202 simulation finished, start learning
    4.0
    EP 202 learning finished: 0.09600903168404074, 0.027160432359709812
    203
    EP 203 simulation finished, start learning
    3.0
    EP 203 learning finished: 0.094391625561002, 0.032273381217873855
    204
    EP 204 simulation finished, start learning
    4.0
    EP 204 learning finished: 0.138714668936462, 0.039247811200135395
    205
    EP 205 simulation finished, start learning
    3.0
    EP 205 learning finished: 0.09056070622934698, 0.02726865868890253
    206
    EP 206 simulation finished, start learning
    4.0
    EP 206 learning finished: 0.08915577563450228, 0.026990334974287933
    207
    EP 207 simulation finished, start learning
    4.0
    EP 207 learning finished: 0.1310369500599009, 0.028237800195867042
    208
    EP 208 simulation finished, start learning
    2.0
    EP 208 learning finished: 0.08565167622401759, 0.026760423113974124
    209
    EP 209 simulation finished, start learning
    4.1
    EP 209 learning finished: 0.08427834884142982, 0.033713974764734665
    210
    EP 210 simulation finished, start learning
    4.0
    EP 210 learning finished: 0.044096053803688307, 0.029914787822156914
    211
    EP 211 simulation finished, start learning
    4.0
    EP 211 learning finished: 0.043615452573337876, 0.029707726348256358
    212
    EP 212 simulation finished, start learning
    2.0
    EP 212 learning finished: 0.04320698054921834, 0.0242626111802912
    213
    EP 213 simulation finished, start learning
    3.0
    EP 213 learning finished: 0.0408103485180078, 0.020415767356800174
    214
    EP 214 simulation finished, start learning
    3.0
    EP 214 learning finished: 0.040561561816829794, 0.022944192003522563
    215
    EP 215 simulation finished, start learning
    4.0
    EP 215 learning finished: 0.03411181030216426, 0.019371738157754947
    216
    EP 216 simulation finished, start learning
    3.0
    EP 216 learning finished: 0.03393944436912477, 0.01884711769324099
    217
    EP 217 simulation finished, start learning
    3.0
    EP 217 learning finished: 0.033681325636123916, 0.015476181499934683
    218
    EP 218 simulation finished, start learning
    4.0
    EP 218 learning finished: 0.040480241522118376, 0.016345648736587986
    219
    EP 219 simulation finished, start learning
    4.0
    EP 219 learning finished: 0.0800622064811769, 0.019630073463130653
    220
    EP 220 simulation finished, start learning
    4.0
    EP 220 learning finished: 0.11788331380363815, 0.02602205467575421
    221
    EP 221 simulation finished, start learning
    3.0
    EP 221 learning finished: 0.11517571995664255, 0.025775757051993864
    222
    EP 222 simulation finished, start learning
    3.0
    EP 222 learning finished: 0.07540087593159583, 0.019424836978632953
    223
    EP 223 simulation finished, start learning
    3.1
    EP 223 learning finished: 0.07436877211391035, 0.02326890209843076
    224
    EP 224 simulation finished, start learning
    2.0
    EP 224 learning finished: 0.036768685239032095, 0.01725613930558782
    225
    EP 225 simulation finished, start learning
    5.0
    EP 225 learning finished: 0.036547870433171425, 0.01680116779726089
    226
    EP 226 simulation finished, start learning
    5.0
    EP 226 learning finished: 0.03631283688716991, 0.016571748473041954
    227
    EP 227 simulation finished, start learning
    3.0
    EP 227 learning finished: 0.029876954179588466, 0.011847819771371467
    228
    EP 228 simulation finished, start learning
    3.0
    EP 228 learning finished: 0.029779734726666618, 0.011408370306549879
    229
    EP 229 simulation finished, start learning
    5.0
    EP 229 learning finished: 0.02971207574433521, 0.014293099000877163
    230
    EP 230 simulation finished, start learning
    5.0
    EP 230 learning finished: 0.02959693207099079, 0.014216756962868024
    231
    EP 231 simulation finished, start learning
    5.0
    EP 231 learning finished: 0.03956218834125025, 0.011893761808393558
    232
    EP 232 simulation finished, start learning
    3.0
    EP 232 learning finished: 0.039269629300321185, 0.017956792774153147
    233
    EP 233 simulation finished, start learning
    5.0
    EP 233 learning finished: 0.03612711396241153, 0.015000751662975395
    234
    EP 234 simulation finished, start learning
    5.0
    EP 234 learning finished: 0.03886849227778555, 0.017758189612184932
    235
    EP 235 simulation finished, start learning
    3.0
    EP 235 learning finished: 0.035692936203886916, 0.015174862555217791
    236
    EP 236 simulation finished, start learning
    3.0
    EP 236 learning finished: 0.02366936276297693, 0.009103551887806205
    237
    EP 237 simulation finished, start learning
    4.0
    EP 237 learning finished: 0.035537480731424154, 0.014821369507198886
    238
    EP 238 simulation finished, start learning
    3.1
    EP 238 learning finished: 0.07043667246138341, 0.02172255467167261
    239
    EP 239 simulation finished, start learning
    3.0
    EP 239 learning finished: 0.06939838517757835, 0.02631940650772731
    240
    EP 240 simulation finished, start learning
    3.0
    EP 240 learning finished: 0.0684778634847838, 0.02609302117895851
    241
    EP 241 simulation finished, start learning
    3.0
    EP 241 learning finished: 0.06755966154627302, 0.026218716607159465
    242
    EP 242 simulation finished, start learning
    3.0
    EP 242 learning finished: 0.03347953266925785, 0.01697035888502067
    243
    EP 243 simulation finished, start learning
    3.0
    EP 243 learning finished: 0.033253418114775, 0.01689288305177013
    244
    EP 244 simulation finished, start learning
    4.0
    EP 244 learning finished: 0.065942807598109, 0.022028635605991553
    245
    EP 245 simulation finished, start learning
    3.0
    EP 245 learning finished: 0.05772054367020374, 0.021237188121329353
    246
    EP 246 simulation finished, start learning
    3.0
    EP 246 learning finished: 0.06469436340301371, 0.026346513052997622
    247
    EP 247 simulation finished, start learning
    5.0
    EP 247 learning finished: 0.06383257977981671, 0.02386935167146061
    248
    EP 248 simulation finished, start learning
    4.0
    EP 248 learning finished: 0.03160156273326109, 0.021746942649677692
    249
    EP 249 simulation finished, start learning
    2.0
    EP 249 learning finished: 0.03143879121559934, 0.02197908828103638
    250
    EP 250 simulation finished, start learning
    5.0
    EP 250 learning finished: 0.03885655772015184, 0.019108653563418915
    251
    EP 251 simulation finished, start learning
    4.1
    EP 251 learning finished: 0.04189613747148968, 0.02023861973233802
    252
    EP 252 simulation finished, start learning
    4.0
    EP 252 learning finished: 0.04149758775022505, 0.020397890056716033
    253
    EP 253 simulation finished, start learning
    3.0
    EP 253 learning finished: 0.0410969682747897, 0.015477448291202123
    254
    EP 254 simulation finished, start learning
    2.0
    EP 254 learning finished: 0.04067344539944884, 0.013204795226775682
    255
    EP 255 simulation finished, start learning
    5.0
    EP 255 learning finished: 0.018805349090424528, 0.009375453733503273
    256
    EP 256 simulation finished, start learning
    4.0
    EP 256 learning finished: 0.018736740286328768, 0.00903448687241415
    257
    EP 257 simulation finished, start learning
    2.1
    EP 257 learning finished: 0.04001308983823293, 0.012547296379269213
    258
    EP 258 simulation finished, start learning
    4.0
    EP 258 learning finished: 0.039638499391404736, 0.0171794486205048
    259
    EP 259 simulation finished, start learning
    3.0
    EP 259 learning finished: 0.039269796084216324, 0.01719837599604901
    260
    EP 260 simulation finished, start learning
    4.0
    EP 260 learning finished: 0.06226776577663529, 0.019266939059177684
    261
    EP 261 simulation finished, start learning
    2.0
    EP 261 learning finished: 0.06149793594969921, 0.015733495422866877
    262
    EP 262 simulation finished, start learning
    3.0
    EP 262 learning finished: 0.030472283587634763, 0.015431497245557443
    263
    EP 263 simulation finished, start learning
    2.1
    EP 263 learning finished: 0.057372731651021525, 0.023653713913006413
    264
    EP 264 simulation finished, start learning
    3.0
    EP 264 learning finished: 0.08365329034113245, 0.023256429642929177
    265
    EP 265 simulation finished, start learning
    3.0
    EP 265 learning finished: 0.1089237083433936, 0.027188984894369977
    266
    EP 266 simulation finished, start learning
    4.0
    EP 266 learning finished: 0.08093411961388597, 0.027212049308442825
    267
    EP 267 simulation finished, start learning
    3.0
    EP 267 learning finished: 0.07768805738393425, 0.023443061342247162
    268
    EP 268 simulation finished, start learning
    2.1
    EP 268 learning finished: 0.053389582653931456, 0.022451835938279412
    269
    EP 269 simulation finished, start learning
    2.0
    EP 269 learning finished: 0.052845986189062444, 0.022608256455242188
    270
    EP 270 simulation finished, start learning
    4.0
    EP 270 learning finished: 0.05428057169309497, 0.020663921093560706
    271
    EP 271 simulation finished, start learning
    4.0
    EP 271 learning finished: 0.05493700231065546, 0.01892993430092697
    272
    EP 272 simulation finished, start learning
    3.0
    EP 272 learning finished: 0.05336850707212193, 0.019782240496204768
    273
    EP 273 simulation finished, start learning
    5.0
    EP 273 learning finished: 0.03412554001396373, 0.01746691052176826
    274
    EP 274 simulation finished, start learning
    3.0
    EP 274 learning finished: 0.026252677709258876, 0.014776728214556556
    275
    EP 275 simulation finished, start learning
    2.1
    EP 275 learning finished: 0.02969792905455293, 0.02046276901326453
    276
    EP 276 simulation finished, start learning
    4.0
    EP 276 learning finished: 0.03618996964457111, 0.01865020668048878
    277
    EP 277 simulation finished, start learning
    3.0
    EP 277 learning finished: 0.03585808980996319, 0.016265854252413448
    278
    EP 278 simulation finished, start learning
    2.1
    EP 278 learning finished: 0.05446984072178296, 0.019255282517362343
    279
    EP 279 simulation finished, start learning
    3.0
    EP 279 learning finished: 0.03505904290746287, 0.015036910433931183
    280
    EP 280 simulation finished, start learning
    5.0
    EP 280 learning finished: 0.02587098593339121, 0.012209492315579476
    281
    EP 281 simulation finished, start learning
    4.1
    EP 281 learning finished: 0.03717193400276386, 0.017569558727177064
    282
    EP 282 simulation finished, start learning
    3.1
    EP 282 learning finished: 0.052574788868522404, 0.01987224716840667
    283
    EP 283 simulation finished, start learning
    3.0
    EP 283 learning finished: 0.05185213669671995, 0.015750474241473356
    284
    EP 284 simulation finished, start learning
    3.0
    EP 284 learning finished: 0.05114387635930484, 0.01591222664596496
    285
    EP 285 simulation finished, start learning
    4.0
    EP 285 learning finished: 0.03290443377635199, 0.015186986767114755
    286
    EP 286 simulation finished, start learning
    4.0
    EP 286 learning finished: 0.029121138255650436, 0.012052630757040106
    287
    EP 287 simulation finished, start learning
    3.0
    EP 287 learning finished: 0.028955311166882325, 0.011998364965405628
    288
    EP 288 simulation finished, start learning
    5.0
    EP 288 learning finished: 0.028801837575937128, 0.011853352105041756
    289
    EP 289 simulation finished, start learning
    5.0
    EP 289 learning finished: 0.014988957516276713, 0.007152009278747447
    290
    EP 290 simulation finished, start learning
    5.0
    EP 290 learning finished: 0.016793070063080506, 0.007043802011541089
    291
    EP 291 simulation finished, start learning
    2.1
    EP 291 learning finished: 0.028713741448026298, 0.014928139301746413
    292
    EP 292 simulation finished, start learning
    3.0
    EP 292 learning finished: 0.02858128643482516, 0.018926389089324758
    293
    EP 293 simulation finished, start learning
    5.0
    EP 293 learning finished: 0.048703374411996236, 0.02257839528318743
    294
    EP 294 simulation finished, start learning
    5.0
    EP 294 learning finished: 0.04828431242451714, 0.022448025124532478
    295
    EP 295 simulation finished, start learning
    2.0
    EP 295 learning finished: 0.04571510663962641, 0.014843171923324405
    296
    EP 296 simulation finished, start learning
    2.9
    EP 296 learning finished: 0.024822403732560616, 0.013948276348996203
    297
    EP 297 simulation finished, start learning
    4.0
    EP 297 learning finished: 0.03175492659780943, 0.01283618497275718
    298
    EP 298 simulation finished, start learning
    5.0
    EP 298 learning finished: 0.03150201779594042, 0.012792064315654272
    299
    EP 299 simulation finished, start learning
    4.0
    EP 299 learning finished: 0.04557204318614083, 0.014692680435066071
    300
    EP 300 simulation finished, start learning
    3.0
    EP 300 learning finished: 0.028501349080588767, 0.011558187781478916
    301
    EP 301 simulation finished, start learning
    2.0
    EP 301 learning finished: 0.02242483922339744, 0.01314380558440457
    302
    EP 302 simulation finished, start learning
    3.0
    EP 302 learning finished: 0.02236550156595385, 0.01116191284391673
    303
    EP 303 simulation finished, start learning
    3.0
    EP 303 learning finished: 0.04669022900231923, 0.01299429349637767
    304
    EP 304 simulation finished, start learning
    4.0
    EP 304 learning finished: 0.04625900976943953, 0.015058870217645884
    305
    EP 305 simulation finished, start learning
    4.0
    EP 305 learning finished: 0.02766138005947738, 0.015428354887619141
    306
    EP 306 simulation finished, start learning
    5.0
    EP 306 learning finished: 0.04562784786828189, 0.01862750496706944
    307
    EP 307 simulation finished, start learning
    3.0
    EP 307 learning finished: 0.04333098428020321, 0.018308913221983875
    308
    EP 308 simulation finished, start learning
    3.1
    EP 308 learning finished: 0.054336322060887454, 0.023224463690141306
    309
    EP 309 simulation finished, start learning
    3.0
    EP 309 learning finished: 0.0445549874116935, 0.018771808850440053
    310
    EP 310 simulation finished, start learning
    5.0
    EP 310 learning finished: 0.026794499167397845, 0.015081159660657187
    311
    EP 311 simulation finished, start learning
    4.0
    EP 311 learning finished: 0.02984697700013328, 0.013532952617116104
    312
    EP 312 simulation finished, start learning
    4.0
    EP 312 learning finished: 0.026534836260641015, 0.010913511946417506
    313
    EP 313 simulation finished, start learning
    5.0
    EP 313 learning finished: 0.02640197938821842, 0.010781235071611661
    314
    EP 314 simulation finished, start learning
    4.0
    EP 314 learning finished: 0.027164429240833954, 0.013051050655469965
    315
    EP 315 simulation finished, start learning
    3.0
    EP 315 learning finished: 0.026154202794803183, 0.010965591759360885
    316
    EP 316 simulation finished, start learning
    4.0
    EP 316 learning finished: 0.026020883276796525, 0.010705469060952785
    317
    EP 317 simulation finished, start learning
    4.0
    EP 317 learning finished: 0.05164994179250826, 0.014852530624995216
    318
    EP 318 simulation finished, start learning
    3.0
    EP 318 learning finished: 0.07637085958331014, 0.01699268976443058
    319
    EP 319 simulation finished, start learning
    5.0
    EP 319 learning finished: 0.07510011764731672, 0.016660139489407772
    320
    EP 320 simulation finished, start learning
    5.0
    EP 320 learning finished: 0.049370687434826976, 0.014602970988017105
    321
    EP 321 simulation finished, start learning
    5.0
    EP 321 learning finished: 0.028998570081189, 0.012669317549366474
    322
    EP 322 simulation finished, start learning
    4.0
    EP 322 learning finished: 0.026626975814565412, 0.010522894946617475
    323
    EP 323 simulation finished, start learning
    5.0
    EP 323 learning finished: 0.02655457423100316, 0.010332964440294154
    324
    EP 324 simulation finished, start learning
    5.0
    EP 324 learning finished: 0.026445601179482026, 0.010294625493619
    325
    EP 325 simulation finished, start learning
    5.0
    EP 325 learning finished: 0.01337734701821347, 0.008166153530295853
    326
    EP 326 simulation finished, start learning
    3.0
    EP 326 learning finished: 0.015330067706308625, 0.008403357232458811
    327
    EP 327 simulation finished, start learning
    5.0
    EP 327 learning finished: 0.015279053933195286, 0.008531563710620915
    328
    EP 328 simulation finished, start learning
    3.0
    EP 328 learning finished: 0.02647528145118372, 0.008776274501001899
    329
    EP 329 simulation finished, start learning
    5.0
    EP 329 learning finished: 0.026281671202248447, 0.008731921996283512
    330
    EP 330 simulation finished, start learning
    5.0
    EP 330 learning finished: 0.013066815263101716, 0.006361291255415106
    331
    EP 331 simulation finished, start learning
    4.1
    EP 331 learning finished: 0.028239207030342328, 0.01111525891550554
    332
    EP 332 simulation finished, start learning
    3.0
    EP 332 learning finished: 0.015160467855906745, 0.009020730867894974
    333
    EP 333 simulation finished, start learning
    3.0
    EP 333 learning finished: 0.02347572778068141, 0.012691534749847003
    334
    EP 334 simulation finished, start learning
    4.0
    EP 334 learning finished: 0.027851211668657783, 0.014672835465042647
    335
    EP 335 simulation finished, start learning
    3.0
    EP 335 learning finished: 0.02478322202478722, 0.0141861790433128
    336
    EP 336 simulation finished, start learning
    3.0
    EP 336 learning finished: 0.04020347475498198, 0.018585325186065187
    337
    EP 337 simulation finished, start learning
    5.0
    EP 337 learning finished: 0.03974190391151211, 0.014813941919701534
    338
    EP 338 simulation finished, start learning
    3.1
    EP 338 learning finished: 0.05367510754232896, 0.017308278200220073
    339
    EP 339 simulation finished, start learning
    2.0
    EP 339 learning finished: 0.05285202105531894, 0.016891876694206925
    340
    EP 340 simulation finished, start learning
    5.0
    EP 340 learning finished: 0.0261171636956945, 0.014254005104433531
    341
    EP 341 simulation finished, start learning
    5.0
    EP 341 learning finished: 0.02594023988550731, 0.014187477947117463
    342
    EP 342 simulation finished, start learning
    4.1
    EP 342 learning finished: 0.02947046718503543, 0.014254900374128946
    343
    EP 343 simulation finished, start learning
    5.0
    EP 343 learning finished: 0.025225049111114117, 0.01039173803993402
    344
    EP 344 simulation finished, start learning
    5.0
    EP 344 learning finished: 0.0138788202148028, 0.008363825800275984
    345
    EP 345 simulation finished, start learning
    5.0
    EP 345 learning finished: 0.013838709363625412, 0.008347506875629426
    346
    EP 346 simulation finished, start learning
    4.0
    EP 346 learning finished: 0.011713352831804968, 0.005679279865604532
    347
    EP 347 simulation finished, start learning
    4.0
    EP 347 learning finished: 0.024207923470275006, 0.009600413891676615
    348
    EP 348 simulation finished, start learning
    3.0
    EP 348 learning finished: 0.024103617076054462, 0.01312774497639424
    349
    EP 349 simulation finished, start learning
    5.0
    EP 349 learning finished: 0.02400004260458033, 0.015086535187932645
    350
    EP 350 simulation finished, start learning
    3.0
    EP 350 learning finished: 0.0238956918373745, 0.013441189377643306
    351
    EP 351 simulation finished, start learning
    3.0
    EP 351 learning finished: 0.020313259170276154, 0.00961678341581577
    352
    EP 352 simulation finished, start learning
    4.0
    EP 352 learning finished: 0.02379809017629242, 0.009927177280610286
    353
    EP 353 simulation finished, start learning
    4.0
    EP 353 learning finished: 0.04724689127345982, 0.011750002869340173
    354
    EP 354 simulation finished, start learning
    2.1
    EP 354 learning finished: 0.046780564337950636, 0.01743939431105801
    355
    EP 355 simulation finished, start learning
    4.0
    EP 355 learning finished: 0.06926054571189866, 0.020958921635070737
    356
    EP 356 simulation finished, start learning
    3.1
    EP 356 learning finished: 0.04558757725533269, 0.021368421866208465
    357
    EP 357 simulation finished, start learning
    3.0
    EP 357 learning finished: 0.03838263485139137, 0.01792418259825605
    358
    EP 358 simulation finished, start learning
    2.1
    EP 358 learning finished: 0.037976822354909956, 0.018071674295990547
    359
    EP 359 simulation finished, start learning
    4.0
    EP 359 learning finished: 0.0486185744241574, 0.016079676415661364
    360
    EP 360 simulation finished, start learning
    4.0
    EP 360 learning finished: 0.026226617460519464, 0.015484474909703986
    361
    EP 361 simulation finished, start learning
    1.0
    EP 361 learning finished: 0.04454037410798506, 0.01911086369386557
    362
    EP 362 simulation finished, start learning
    2.1
    EP 362 learning finished: 0.04409517383712114, 0.015438640219902785
    363
    EP 363 simulation finished, start learning
    4.0
    EP 363 learning finished: 0.06532323274456076, 0.01715041031635896
    364
    EP 364 simulation finished, start learning
    3.0
    EP 364 learning finished: 0.04300626366109839, 0.013602369012513549
    365
    EP 365 simulation finished, start learning
    2.0
    EP 365 learning finished: 0.02135535407499134, 0.009933028867152416
    366
    EP 366 simulation finished, start learning
    5.0
    EP 366 learning finished: 0.021254267517655023, 0.009396474922511425
    367
    EP 367 simulation finished, start learning
    4.1
    EP 367 learning finished: 0.029687741279357027, 0.01013855121148488
    368
    EP 368 simulation finished, start learning
    3.0
    EP 368 learning finished: 0.0294253792246586, 0.011911005214419864
    369
    EP 369 simulation finished, start learning
    5.0
    EP 369 learning finished: 0.03722275778523709, 0.013415193485814806
    370
    EP 370 simulation finished, start learning
    2.0
    EP 370 learning finished: 0.0289305480229487, 0.01500156382338543
    371
    EP 371 simulation finished, start learning
    3.0
    EP 371 learning finished: 0.021346966162330006, 0.011301491891809215
    372
    EP 372 simulation finished, start learning
    5.0
    EP 372 learning finished: 0.02125225705757927, 0.009386260753394401
    373
    EP 373 simulation finished, start learning
    3.1
    EP 373 learning finished: 0.023439900811261793, 0.011436298848231866
    374
    EP 374 simulation finished, start learning
    3.0
    EP 374 learning finished: 0.023284492836207704, 0.011169395412874058
    375
    EP 375 simulation finished, start learning
    2.1
    EP 375 learning finished: 0.03475935324499346, 0.012932636306093906
    376
    EP 376 simulation finished, start learning
    4.0
    EP 376 learning finished: 0.03439584883857805, 0.016308350519775416
    377
    EP 377 simulation finished, start learning
    4.0
    EP 377 learning finished: 0.04238918452473328, 0.016149516266994013
    378
    EP 378 simulation finished, start learning
    4.0
    EP 378 learning finished: 0.04197791253587013, 0.014452648196887333
    379
    EP 379 simulation finished, start learning
    4.0
    EP 379 learning finished: 0.06222634601382637, 0.01545559171092492
    380
    EP 380 simulation finished, start learning
    4.0
    EP 380 learning finished: 0.06130557913983736, 0.015583846419716588
    381
    EP 381 simulation finished, start learning
    4.0
    EP 381 learning finished: 0.040386676969635715, 0.013451783522998065
    382
    EP 382 simulation finished, start learning
    3.0
    EP 382 learning finished: 0.04001977913871002, 0.014968687117556575
    383
    EP 383 simulation finished, start learning
    3.0
    EP 383 learning finished: 0.02042386931755047, 0.011756059163872715
    384
    EP 384 simulation finished, start learning
    3.1
    EP 384 learning finished: 0.02160519636545999, 0.013585690319546965
    385
    EP 385 simulation finished, start learning
    3.1
    EP 385 learning finished: 0.03934564849664923, 0.01730540272812379
    386
    EP 386 simulation finished, start learning
    2.2
    EP 386 learning finished: 0.0582929682989608, 0.021461080228731316
    387
    EP 387 simulation finished, start learning
    2.1
    EP 387 learning finished: 0.05748533543658496, 0.02635867736157671
    388
    EP 388 simulation finished, start learning
    4.0
    EP 388 learning finished: 0.05408826033994529, 0.022382088455544274
    389
    EP 389 simulation finished, start learning
    3.0
    EP 389 learning finished: 0.04226540069438656, 0.01717942998169993
    390
    EP 390 simulation finished, start learning
    3.0
    EP 390 learning finished: 0.020041830528214177, 0.010051310535221939
    391
    EP 391 simulation finished, start learning
    3.0
    EP 391 learning finished: 0.01733166701292177, 0.00799030180743543
    392
    EP 392 simulation finished, start learning
    3.0
    EP 392 learning finished: 0.03454276590715821, 0.009557028660402386
    393
    EP 393 simulation finished, start learning
    3.0
    EP 393 learning finished: 0.05148632216208071, 0.01241685622689559
    394
    EP 394 simulation finished, start learning
    2.1
    EP 394 learning finished: 0.07009127378010405, 0.017202357995564316
    395
    EP 395 simulation finished, start learning
    2.1
    EP 395 learning finished: 0.05450086831750767, 0.016176334281460925
    396
    EP 396 simulation finished, start learning
    3.0
    EP 396 learning finished: 0.053867941142133446, 0.01603788661628632
    397
    EP 397 simulation finished, start learning
    2.2
    EP 397 learning finished: 0.0426509418226253, 0.01995203644083136
    398
    EP 398 simulation finished, start learning
    4.0
    EP 398 learning finished: 0.03748417532672477, 0.01828622392469666
    399
    EP 399 simulation finished, start learning
    2.0
    EP 399 learning finished: 0.03712388525821808, 0.016607146434990245
    400
    EP 400 simulation finished, start learning
    4.0
    EP 400 learning finished: 0.0368113906758154, 0.01505248254840256
    401
    EP 401 simulation finished, start learning
    3.0
    EP 401 learning finished: 0.018285153840416513, 0.008348470684891751
    402
    EP 402 simulation finished, start learning
    4.0
    EP 402 learning finished: 0.018212563426235562, 0.008083118567447775
    403
    EP 403 simulation finished, start learning
    2.2
    EP 403 learning finished: 0.03621587373773494, 0.014302295156764918
    404
    EP 404 simulation finished, start learning
    3.1
    EP 404 learning finished: 0.05371002243790457, 0.017635030530267787
    405
    EP 405 simulation finished, start learning
    3.0
    EP 405 learning finished: 0.052988526923854806, 0.0201619156614606
    406
    EP 406 simulation finished, start learning
    2.0
    EP 406 learning finished: 0.03492792156705349, 0.01747079198096439
    407
    EP 407 simulation finished, start learning
    5.0
    EP 407 learning finished: 0.017744289939784608, 0.01093280409652131
    408
    EP 408 simulation finished, start learning
    5.0
    EP 408 learning finished: 0.016133721799757694, 0.00800965708066966
    409
    EP 409 simulation finished, start learning
    3.1
    EP 409 learning finished: 0.01810304770096849, 0.008339601700348235
    410
    EP 410 simulation finished, start learning
    4.0
    EP 410 learning finished: 0.018004328464822805, 0.010823628162394646
    411
    EP 411 simulation finished, start learning
    3.0
    EP 411 learning finished: 0.017914704114222246, 0.013522394181993747
    412
    EP 412 simulation finished, start learning
    4.0
    EP 412 learning finished: 0.03432457366556463, 0.014535951841555317
    413
    EP 413 simulation finished, start learning
    5.0
    EP 413 learning finished: 0.03404220402633129, 0.011417289214406878
    414
    EP 414 simulation finished, start learning
    5.0
    EP 414 learning finished: 0.01691154436150555, 0.008631554639051112
    415
    EP 415 simulation finished, start learning
    4.0
    EP 415 learning finished: 0.033651238437215625, 0.008612059103992209
    416
    EP 416 simulation finished, start learning
    3.1
    EP 416 learning finished: 0.017817226684812226, 0.008915302518240983
    417
    EP 417 simulation finished, start learning
    1.0
    EP 417 learning finished: 0.033256779733282116, 0.011999752199321515
    418
    EP 418 simulation finished, start learning
    4.0
    EP 418 learning finished: 0.04936876518614142, 0.014584445225242604
    419
    EP 419 simulation finished, start learning
    2.1
    EP 419 learning finished: 0.03256901465794115, 0.01657037113559312
    420
    EP 420 simulation finished, start learning
    5.0
    EP 420 learning finished: 0.03228960253971991, 0.015181540352951828
    421
    EP 421 simulation finished, start learning
    3.0
    EP 421 learning finished: 0.024684955830920785, 0.013990005802082687
    422
    EP 422 simulation finished, start learning
    3.0
    EP 422 learning finished: 0.034197548174258685, 0.01399197130497202
    423
    EP 423 simulation finished, start learning
    4.0
    EP 423 learning finished: 0.024353619256803682, 0.011887521211033794
    424
    EP 424 simulation finished, start learning
    3.0
    EP 424 learning finished: 0.03199189813610559, 0.01304516454465047
    425
    EP 425 simulation finished, start learning
    4.0
    EP 425 learning finished: 0.031766123429616044, 0.012354579823549372
    426
    EP 426 simulation finished, start learning
    4.0
    EP 426 learning finished: 0.03161564912495507, 0.0122509573084848
    427
    EP 427 simulation finished, start learning
    3.0
    EP 427 learning finished: 0.03146295471219318, 0.012255105259497986
    428
    EP 428 simulation finished, start learning
    3.0
    EP 428 learning finished: 0.02954882273595061, 0.011994853815053238
    429
    EP 429 simulation finished, start learning
    3.0
    EP 429 learning finished: 0.04408568313957417, 0.01330229696199936
    430
    EP 430 simulation finished, start learning
    3.0
    EP 430 learning finished: 0.058337691619964716, 0.013272228353721141
    431
    EP 431 simulation finished, start learning
    3.0
    EP 431 learning finished: 0.05782141365740756, 0.013176681615388356
    432
    EP 432 simulation finished, start learning
    2.1
    EP 432 learning finished: 0.04446477868611609, 0.012321969012776416
    433
    EP 433 simulation finished, start learning
    4.0
    EP 433 learning finished: 0.029906843284543072, 0.012282227158173953
    434
    EP 434 simulation finished, start learning
    3.0
    EP 434 learning finished: 0.029723405055719354, 0.012231698716703931
    435
    EP 435 simulation finished, start learning
    3.0
    EP 435 learning finished: 0.03092980371734555, 0.012414095293694416
    436
    EP 436 simulation finished, start learning
    5.0
    EP 436 learning finished: 0.030677935736704853, 0.010633884651052602
    437
    EP 437 simulation finished, start learning
    4.0
    EP 437 learning finished: 0.015246679046772016, 0.009347144307100189
    438
    EP 438 simulation finished, start learning
    2.2
    EP 438 learning finished: 0.03033419453268138, 0.012583518013578043
    439
    EP 439 simulation finished, start learning
    3.0
    EP 439 learning finished: 0.026211549433355685, 0.012331312714629542
    440
    EP 440 simulation finished, start learning
    4.0
    EP 440 learning finished: 0.030009268987598148, 0.01468845184298928
    441
    EP 441 simulation finished, start learning
    2.0
    EP 441 learning finished: 0.030440448595200564, 0.016057573515148842
    442
    EP 442 simulation finished, start learning
    5.0
    EP 442 learning finished: 0.027526445564703295, 0.012005233094643112
    443
    EP 443 simulation finished, start learning
    5.0
    EP 443 learning finished: 0.023333280563893055, 0.011155037808146417
    444
    EP 444 simulation finished, start learning
    3.1
    EP 444 learning finished: 0.023196285741953204, 0.011506196202563027
    445
    EP 445 simulation finished, start learning
    3.0
    EP 445 learning finished: 0.02149473014864256, 0.011265966152412451
    446
    EP 446 simulation finished, start learning
    3.0
    EP 446 learning finished: 0.016459752831060825, 0.00971539124244103
    447
    EP 447 simulation finished, start learning
    3.1
    EP 447 learning finished: 0.025316870311176132, 0.01218919953987374
    448
    EP 448 simulation finished, start learning
    5.0
    EP 448 learning finished: 0.01608484122316156, 0.011008872273338355
    449
    EP 449 simulation finished, start learning
    2.1
    EP 449 learning finished: 0.022783747842666457, 0.011928018551195407
    450
    EP 450 simulation finished, start learning
    4.0
    EP 450 learning finished: 0.029130869348403365, 0.01416052166554671
    451
    EP 451 simulation finished, start learning
    3.0
    EP 451 learning finished: 0.02892134964615778, 0.012588277548635082
    452
    EP 452 simulation finished, start learning
    4.0
    EP 452 learning finished: 0.028700058246840854, 0.012107904133245029
    453
    EP 453 simulation finished, start learning
    3.0
    EP 453 learning finished: 0.02847480882933695, 0.011126094010968165
    454
    EP 454 simulation finished, start learning
    3.0
    EP 454 learning finished: 0.014170807002507502, 0.008868284921978406
    455
    EP 455 simulation finished, start learning
    1.0
    EP 455 learning finished: 0.014122304833751542, 0.009021328483311683
    456
    EP 456 simulation finished, start learning
    2.1
    EP 456 learning finished: 0.014571074875796342, 0.009461489874748708
    457
    EP 457 simulation finished, start learning
    4.0
    EP 457 learning finished: 0.027998077777947827, 0.009379082880371925
    458
    EP 458 simulation finished, start learning
    2.0
    EP 458 learning finished: 0.027787428281886406, 0.009511425328527187
    459
    EP 459 simulation finished, start learning
    5.0
    EP 459 learning finished: 0.0138250592542839, 0.006898683611568757
    460
    EP 460 simulation finished, start learning
    4.1
    EP 460 learning finished: 0.013772902549835209, 0.006823260204945687
    461
    EP 461 simulation finished, start learning
    4.0
    EP 461 learning finished: 0.01569192192515345, 0.005699214406516735
    462
    EP 462 simulation finished, start learning
    3.0
    EP 462 learning finished: 0.015614446607088439, 0.005507878136384647
    463
    EP 463 simulation finished, start learning
    4.0
    EP 463 learning finished: 0.015538226571820246, 0.007711408716805461
    464
    EP 464 simulation finished, start learning
    4.0
    EP 464 learning finished: 0.027386688110798474, 0.008392764596945193
    465
    EP 465 simulation finished, start learning
    3.0
    EP 465 learning finished: 0.02719787355794523, 0.009565598405706885
    466
    EP 466 simulation finished, start learning
    4.0
    EP 466 learning finished: 0.04043470737049404, 0.01162546223295339
    467
    EP 467 simulation finished, start learning
    3.0
    EP 467 learning finished: 0.026712947684263355, 0.011695951590513213
    468
    EP 468 simulation finished, start learning
    3.0
    EP 468 learning finished: 0.026152101098595892, 0.00957311521880655
    469
    EP 469 simulation finished, start learning
    3.1
    EP 469 learning finished: 0.02643914152401905, 0.010887607472431782
    470
    EP 470 simulation finished, start learning
    3.0
    EP 470 learning finished: 0.014252880691678627, 0.008796318589072408
    471
    EP 471 simulation finished, start learning
    3.0
    EP 471 learning finished: 0.013110780285000345, 0.006573939858887101
    472
    EP 472 simulation finished, start learning
    3.0
    EP 472 learning finished: 0.014202241447620167, 0.00875798665008117
    473
    EP 473 simulation finished, start learning
    3.0
    EP 473 learning finished: 0.01291798071014938, 0.005220631989722928
    474
    EP 474 simulation finished, start learning
    3.1
    EP 474 learning finished: 0.014122502957324201, 0.008725366198181797
    475
    EP 475 simulation finished, start learning
    3.0
    EP 475 learning finished: 0.02689453426365418, 0.010878397750172297
    476
    EP 476 simulation finished, start learning
    3.0
    EP 476 learning finished: 0.013986476973574469, 0.008671911800577211
    477
    EP 477 simulation finished, start learning
    3.0
    EP 477 learning finished: 0.02664478272476245, 0.010807612362087937
    478
    EP 478 simulation finished, start learning
    5.0
    EP 478 learning finished: 0.02535798071083306, 0.007236416818391452
    479
    EP 479 simulation finished, start learning
    5.0
    EP 479 learning finished: 0.012645617888964011, 0.005003942818494996
    480
    EP 480 simulation finished, start learning
    5.0
    EP 480 learning finished: 0.012622841323562284, 0.006393884463559966
    481
    EP 481 simulation finished, start learning
    5.0
    EP 481 learning finished: 0.01834313756956618, 0.005635810819529278
    482
    EP 482 simulation finished, start learning
    3.0
    EP 482 learning finished: 0.028858738537163475, 0.007246836270478359
    483
    EP 483 simulation finished, start learning
    3.0
    EP 483 learning finished: 0.030096044552518464, 0.009416743529698715
    484
    EP 484 simulation finished, start learning
    2.0
    EP 484 learning finished: 0.02363953416644904, 0.008157540718678172
    485
    EP 485 simulation finished, start learning
    3.0
    EP 485 learning finished: 0.015869147328658073, 0.006849988796141441
    486
    EP 486 simulation finished, start learning
    3.0
    EP 486 learning finished: 0.012565164523621775, 0.005296044410624627
    487
    EP 487 simulation finished, start learning
    5.0
    EP 487 learning finished: 0.006329935763673689, 0.0029943524477271666
    488
    EP 488 simulation finished, start learning
    3.1
    EP 488 learning finished: 0.01550008102887901, 0.005329124111449701
    489
    EP 489 simulation finished, start learning
    4.0
    EP 489 learning finished: 0.015418965467095092, 0.0072580834551997
    490
    EP 490 simulation finished, start learning
    5.0
    EP 490 learning finished: 0.015351531341050162, 0.00859975029264485
    491
    EP 491 simulation finished, start learning
    5.0
    EP 491 learning finished: 0.017738497053120028, 0.01009598930921681
    492
    EP 492 simulation finished, start learning
    4.0
    EP 492 learning finished: 0.017696103826615683, 0.00862291058567403
    493
    EP 493 simulation finished, start learning
    3.0
    EP 493 learning finished: 0.018971847200436542, 0.00870134755696069
    494
    EP 494 simulation finished, start learning
    2.1
    EP 494 learning finished: 0.02607436293647325, 0.010883185257631212
    495
    EP 495 simulation finished, start learning
    3.0
    EP 495 learning finished: 0.038324208123363235, 0.01159231935376089
    496
    EP 496 simulation finished, start learning
    4.0
    EP 496 learning finished: 0.0380167087630241, 0.012547480350460086
    497
    EP 497 simulation finished, start learning
    2.0
    EP 497 learning finished: 0.037733959655387395, 0.012642248672960932
    498
    EP 498 simulation finished, start learning
    5.0
    EP 498 learning finished: 0.024352260487251698, 0.009069764387987789
    499
    EP 499 simulation finished, start learning
    4.0
    EP 499 learning finished: 0.012778443683679086, 0.007994566336340844
    500
    EP 500 simulation finished, start learning
    3.0
    EP 500 learning finished: 0.012128974858654828, 0.005972417576074918
    501
    EP 501 simulation finished, start learning
    2.0
    EP 501 learning finished: 0.012110292098714837, 0.00596333079351415
    502
    EP 502 simulation finished, start learning
    4.0
    EP 502 learning finished: 0.013459675575973051, 0.007006059286066346
    503
    EP 503 simulation finished, start learning
    4.0
    EP 503 learning finished: 0.012782235138441678, 0.008001330162781409
    504
    EP 504 simulation finished, start learning
    3.0
    EP 504 learning finished: 0.012745689073874189, 0.007985348954708963
    505
    EP 505 simulation finished, start learning
    2.1
    EP 505 learning finished: 0.014791256736064184, 0.0072497024995611765
    506
    EP 506 simulation finished, start learning
    4.0
    EP 506 learning finished: 0.025288496236509417, 0.008226410674235277
    507
    EP 507 simulation finished, start learning
    3.0
    EP 507 learning finished: 0.012589823185908955, 0.006220498086070947
    508
    EP 508 simulation finished, start learning
    4.0
    EP 508 learning finished: 0.0250454433095868, 0.00817276818887199
    509
    EP 509 simulation finished, start learning
    2.1
    EP 509 learning finished: 0.024880038636780455, 0.01007633972077985
    510
    EP 510 simulation finished, start learning
    3.0
    EP 510 learning finished: 0.024710461031933306, 0.010190528031445822
    511
    EP 511 simulation finished, start learning
    3.0
    EP 511 learning finished: 0.024542779062741715, 0.011604079691932037
    512
    EP 512 simulation finished, start learning
    2.1
    EP 512 learning finished: 0.015978647048935812, 0.011001646166433806
    513
    EP 513 simulation finished, start learning
    3.0
    EP 513 learning finished: 0.013558404258830592, 0.00759300123084472
    514
    EP 514 simulation finished, start learning
    3.0
    EP 514 learning finished: 0.013480556603187921, 0.007518160258896955
    515
    EP 515 simulation finished, start learning
    4.0
    EP 515 learning finished: 0.013660230017517705, 0.00806864440334761
    516
    EP 516 simulation finished, start learning
    5.0
    EP 516 learning finished: 0.012687487280615528, 0.00798282151684108
    517
    EP 517 simulation finished, start learning
    5.0
    EP 517 learning finished: 0.017929933722779623, 0.009243148073178867
    518
    EP 518 simulation finished, start learning
    5.0
    EP 518 learning finished: 0.02493772965551244, 0.00850813551139081
    519
    EP 519 simulation finished, start learning
    5.0
    EP 519 learning finished: 0.03310002169323312, 0.007771680403916885
    520
    EP 520 simulation finished, start learning
    5.0
    EP 520 learning finished: 0.02471707966357073, 0.006429508381862921
    521
    EP 521 simulation finished, start learning
    4.0
    EP 521 learning finished: 0.016432189036581946, 0.006130933600654605
    522
    EP 522 simulation finished, start learning
    1.2
    EP 522 learning finished: 0.02236079683627068, 0.007538505916048284
    523
    EP 523 simulation finished, start learning
    2.0
    EP 523 learning finished: 0.022198996043783836, 0.008365903450981685
    524
    EP 524 simulation finished, start learning
    4.1
    EP 524 learning finished: 0.02980218781164723, 0.009653838322273955
    525
    EP 525 simulation finished, start learning
    4.0
    EP 525 learning finished: 0.029506452558418772, 0.009596303625714361
    526
    EP 526 simulation finished, start learning
    4.0
    EP 526 learning finished: 0.013951919348823338, 0.008840183832500514
    527
    EP 527 simulation finished, start learning
    5.0
    EP 527 learning finished: 0.013888043089636426, 0.006588522754851667
    528
    EP 528 simulation finished, start learning
    5.0
    EP 528 learning finished: 0.012013345515658358, 0.005269237207428408
    529
    EP 529 simulation finished, start learning
    3.0
    EP 529 learning finished: 0.011984315297414838, 0.005714982439404419
    530
    EP 530 simulation finished, start learning
    3.0
    EP 530 learning finished: 0.011528337930990062, 0.00576738027278445
    531
    EP 531 simulation finished, start learning
    4.0
    EP 531 learning finished: 0.011960391025350958, 0.007781684049095479
    532
    EP 532 simulation finished, start learning
    4.0
    EP 532 learning finished: 0.023795344844614874, 0.009679661365033473
    533
    EP 533 simulation finished, start learning
    4.0
    EP 533 learning finished: 0.03540876059106779, 0.010125335960938267
    534
    EP 534 simulation finished, start learning
    3.0
    EP 534 learning finished: 0.035043520438385656, 0.008202639728572455
    535
    EP 535 simulation finished, start learning
    2.0
    EP 535 learning finished: 0.023175348822292108, 0.00783352677410143
    536
    EP 536 simulation finished, start learning
    3.0
    EP 536 learning finished: 0.012647989863761921, 0.007060643063674377
    537
    EP 537 simulation finished, start learning
    5.0
    EP 537 learning finished: 0.02054882832025129, 0.0064561637192633485
    538
    EP 538 simulation finished, start learning
    3.0
    EP 538 learning finished: 0.02832420753505005, 0.007689141481041162
    539
    EP 539 simulation finished, start learning
    3.0
    EP 539 learning finished: 0.019116645378190356, 0.006170349559220985
    540
    EP 540 simulation finished, start learning
    4.0
    EP 540 learning finished: 0.017916943662999785, 0.006903536162727282
    541
    EP 541 simulation finished, start learning
    4.0
    EP 541 learning finished: 0.022955108670355884, 0.007484207336375448
    542
    EP 542 simulation finished, start learning
    3.0
    EP 542 learning finished: 0.02281371160949419, 0.006113223726091614
    543
    EP 543 simulation finished, start learning
    4.0
    EP 543 learning finished: 0.03394518201840313, 0.007876958273686322
    544
    EP 544 simulation finished, start learning
    3.0
    EP 544 learning finished: 0.022440594376739398, 0.007992085228784992
    545
    EP 545 simulation finished, start learning
    5.0
    EP 545 learning finished: 0.011454545496821567, 0.006167726059855383
    546
    EP 546 simulation finished, start learning
    4.0
    EP 546 learning finished: 0.011437143873756384, 0.007115201762517269
    547
    EP 547 simulation finished, start learning
    3.0
    EP 547 learning finished: 0.012301836513699982, 0.005430362660457415
    548
    EP 548 simulation finished, start learning
    5.0
    EP 548 learning finished: 0.00634116037509358, 0.003457805570243148
    549
    EP 549 simulation finished, start learning
    5.0
    EP 549 learning finished: 0.006326026296427756, 0.0034528712930192995
    550
    EP 550 simulation finished, start learning
    5.0
    EP 550 learning finished: 0.008879658150881653, 0.003704512513369451
    551
    EP 551 simulation finished, start learning
    5.0
    EP 551 learning finished: 0.0077688905798964925, 0.003574537890467392
    552
    EP 552 simulation finished, start learning
    5.0
    EP 552 learning finished: 0.007765849503979183, 0.003568860679481034
    553
    EP 553 simulation finished, start learning
    5.0
    EP 553 learning finished: 0.007759148004605976, 0.0035626111553508366
    554
    EP 554 simulation finished, start learning
    5.0
    EP 554 learning finished: 0.007753133518431099, 0.003556499826213274
    555
    EP 555 simulation finished, start learning
    3.0
    EP 555 learning finished: 0.00883093101490573, 0.0036714366989573457
    556
    EP 556 simulation finished, start learning
    3.0
    EP 556 learning finished: 0.009902151172024531, 0.003694729607210808
    557
    EP 557 simulation finished, start learning
    2.1
    EP 557 learning finished: 0.010950438923025274, 0.005027249788600585
    558
    EP 558 simulation finished, start learning
    2.1
    EP 558 learning finished: 0.015292881246331547, 0.007062118637319277
    559
    EP 559 simulation finished, start learning
    3.1
    EP 559 learning finished: 0.022801074236841856, 0.01011704399820701
    560
    EP 560 simulation finished, start learning
    5.0
    EP 560 learning finished: 0.022631466077580598, 0.011265929713018133
    561
    EP 561 simulation finished, start learning
    3.0
    EP 561 learning finished: 0.021131038845972228, 0.010995089439387465
    562
    EP 562 simulation finished, start learning
    5.0
    EP 562 learning finished: 0.013560947400282686, 0.007752511235964358
    563
    EP 563 simulation finished, start learning
    5.0
    EP 563 learning finished: 0.008666549314568783, 0.004571975808353557
    564
    EP 564 simulation finished, start learning
    3.0
    EP 564 learning finished: 0.006329506451436373, 0.003469801731933231
    565
    EP 565 simulation finished, start learning
    2.1
    EP 565 learning finished: 0.012082938236947172, 0.005588306682211552
    566
    EP 566 simulation finished, start learning
    3.0
    EP 566 learning finished: 0.02316120501529184, 0.007491398079802518
    567
    EP 567 simulation finished, start learning
    3.0
    EP 567 learning finished: 0.034092394938557646, 0.009369410725404611
    568
    EP 568 simulation finished, start learning
    2.0
    EP 568 learning finished: 0.033859437541511994, 0.010643267009256737
    569
    EP 569 simulation finished, start learning
    4.0
    EP 569 learning finished: 0.02189557403757547, 0.00840551799799035
    570
    EP 570 simulation finished, start learning
    3.0
    EP 570 learning finished: 0.021821932318626835, 0.00837896077179027
    571
    EP 571 simulation finished, start learning
    2.1
    EP 571 learning finished: 0.022546591044037, 0.009590628800528597
    572
    EP 572 simulation finished, start learning
    2.0
    EP 572 learning finished: 0.033229254108640305, 0.010114752435256355
    573
    EP 573 simulation finished, start learning
    4.0
    EP 573 learning finished: 0.03299741479235507, 0.010939045072534626
    574
    EP 574 simulation finished, start learning
    4.0
    EP 574 learning finished: 0.022302164904875355, 0.010854253680024684
    575
    EP 575 simulation finished, start learning
    3.0
    EP 575 learning finished: 0.022153794904252067, 0.009591094384471551
    576
    EP 576 simulation finished, start learning
    2.1
    EP 576 learning finished: 0.022029077337119674, 0.0088535447502102
    577
    EP 577 simulation finished, start learning
    5.0
    EP 577 learning finished: 0.021862242247490116, 0.00876411295085037
    578
    EP 578 simulation finished, start learning
    4.0
    EP 578 learning finished: 0.021760664916996575, 0.007898820268839183
    579
    EP 579 simulation finished, start learning
    4.0
    EP 579 learning finished: 0.013078258271249865, 0.007829167538431511
    580
    EP 580 simulation finished, start learning
    3.0
    EP 580 learning finished: 0.020947748551254364, 0.008393134085332932
    581
    EP 581 simulation finished, start learning
    4.0
    EP 581 learning finished: 0.021736255943949345, 0.008439723587031859
    582
    EP 582 simulation finished, start learning
    3.0
    EP 582 learning finished: 0.021608200143059975, 0.00928887066453754
    583
    EP 583 simulation finished, start learning
    0.0
    EP 583 learning finished: 0.031148105761523937, 0.00954090830835952
    584
    EP 584 simulation finished, start learning
    2.1
    EP 584 learning finished: 0.0315705002102149, 0.010683451795713358
    585
    EP 585 simulation finished, start learning
    2.0
    EP 585 learning finished: 0.04158742280440819, 0.010791538607807164
    586
    EP 586 simulation finished, start learning
    3.0
    EP 586 learning finished: 0.031083112267666024, 0.00908308631699839
    587
    EP 587 simulation finished, start learning
    3.0
    EP 587 learning finished: 0.02078579615434606, 0.0070959178304252175
    588
    EP 588 simulation finished, start learning
    4.0
    EP 588 learning finished: 0.010719660367200845, 0.00590523287028392
    589
    EP 589 simulation finished, start learning
    3.0
    EP 589 learning finished: 0.02134878235137805, 0.005874636368328486
    590
    EP 590 simulation finished, start learning
    5.0
    EP 590 learning finished: 0.02121691102632539, 0.006878550947871413
    591
    EP 591 simulation finished, start learning
    5.0
    EP 591 learning finished: 0.021090952947570685, 0.007963361220407824
    592
    EP 592 simulation finished, start learning
    4.1
    EP 592 learning finished: 0.015239942077094781, 0.007407921417527774
    593
    EP 593 simulation finished, start learning
    5.0
    EP 593 learning finished: 0.02131600974180259, 0.006722124170024675
    594
    EP 594 simulation finished, start learning
    4.1
    EP 594 learning finished: 0.014210849291330874, 0.006741906292012582
    595
    EP 595 simulation finished, start learning
    5.0
    EP 595 learning finished: 0.014150595147075506, 0.00672682466190011
    596
    EP 596 simulation finished, start learning
    5.0
    EP 596 learning finished: 0.02116662484543852, 0.0066789123632648995
    597
    EP 597 simulation finished, start learning
    5.0
    EP 597 learning finished: 0.014074759795025926, 0.005543366080595173
    598
    EP 598 simulation finished, start learning
    5.0
    EP 598 learning finished: 0.014051017808430544, 0.004379802271101241
    599
    EP 599 simulation finished, start learning
    3.0
    EP 599 learning finished: 0.00794439274832115, 0.003354935038943315
    600
    EP 600 simulation finished, start learning
    4.1
    EP 600 learning finished: 0.007931946170046444, 0.004506796776687712
    601
    EP 601 simulation finished, start learning
    2.1
    EP 601 learning finished: 0.01405107019012064, 0.005670789262503935
    602
    EP 602 simulation finished, start learning
    3.1
    EP 602 learning finished: 0.02095279466090716, 0.008513213610183238
    603
    EP 603 simulation finished, start learning
    3.0
    EP 603 learning finished: 0.020817319152282465, 0.00839397167452501
    604
    EP 604 simulation finished, start learning
    4.0
    EP 604 learning finished: 0.020953288052652397, 0.007814650149975103
    605
    EP 605 simulation finished, start learning
    4.0
    EP 605 learning finished: 0.031194300039004474, 0.008306249157874626
    606
    EP 606 simulation finished, start learning
    3.0
    EP 606 learning finished: 0.020629173146456736, 0.005471514143394529
    607
    EP 607 simulation finished, start learning
    3.0
    EP 607 learning finished: 0.020518914994770654, 0.0071470159667429796
    608
    EP 608 simulation finished, start learning
    5.0
    EP 608 learning finished: 0.01992902979280231, 0.007086900013371533
    609
    EP 609 simulation finished, start learning
    4.0
    EP 609 learning finished: 0.01986672273468315, 0.006309520587792032
    610
    EP 610 simulation finished, start learning
    4.0
    EP 610 learning finished: 0.019802620040678676, 0.007924822983267058
    611
    EP 611 simulation finished, start learning
    4.0
    EP 611 learning finished: 0.02031101487831677, 0.007847611245669128
    612
    EP 612 simulation finished, start learning
    3.0
    EP 612 learning finished: 0.020196527290320176, 0.00624594641375668
    613
    EP 613 simulation finished, start learning
    4.0
    EP 613 learning finished: 0.03006621842419932, 0.006953317427643195
    614
    EP 614 simulation finished, start learning
    3.0
    EP 614 learning finished: 0.019904474994351773, 0.007007185054693703
    615
    EP 615 simulation finished, start learning
    4.0
    EP 615 learning finished: 0.01979380740069203, 0.0069835493928677534
    616
    EP 616 simulation finished, start learning
    2.1
    EP 616 learning finished: 0.02019145790560728, 0.009755331345195017
    617
    EP 617 simulation finished, start learning
    3.0
    EP 617 learning finished: 0.020101854853638912, 0.008162853641920265
    618
    EP 618 simulation finished, start learning
    2.1
    EP 618 learning finished: 0.013684255928041011, 0.007602253223240041
    619
    EP 619 simulation finished, start learning
    3.0
    EP 619 learning finished: 0.020446838808144463, 0.007678788453991577
    620
    EP 620 simulation finished, start learning
    4.0
    EP 620 learning finished: 0.01017455472125306, 0.006449900825458599
    621
    EP 621 simulation finished, start learning
    3.0
    EP 621 learning finished: 0.019794547609009783, 0.008075053788555664
    622
    EP 622 simulation finished, start learning
    2.0
    EP 622 learning finished: 0.01924347020730366, 0.007053490620622327
    623
    EP 623 simulation finished, start learning
    2.2
    EP 623 learning finished: 0.019038956781813, 0.008481492459062304
    624
    EP 624 simulation finished, start learning
    4.0
    EP 624 learning finished: 0.024369213693308467, 0.007747667681342089
    625
    EP 625 simulation finished, start learning
    3.0
    EP 625 learning finished: 0.024148581800435664, 0.007706846186059953
    626
    EP 626 simulation finished, start learning
    3.0
    EP 626 learning finished: 0.02392943347904719, 0.009167590948074178
    627
    EP 627 simulation finished, start learning
    3.0
    EP 627 learning finished: 0.018849843166186986, 0.006141141859224319
    628
    EP 628 simulation finished, start learning
    3.0
    EP 628 learning finished: 0.02816530058991873, 0.006904462628574137
    629
    EP 629 simulation finished, start learning
    3.0
    EP 629 learning finished: 0.028031156269392454, 0.006877605095112926
    630
    EP 630 simulation finished, start learning
    3.0
    EP 630 learning finished: 0.027901325900700158, 0.006852647510306416
    631
    EP 631 simulation finished, start learning
    3.0
    EP 631 learning finished: 0.027771556327770952, 0.006827423545052132
    632
    EP 632 simulation finished, start learning
    3.1
    EP 632 learning finished: 0.018802789304988465, 0.007839543459475943
    633
    EP 633 simulation finished, start learning
    3.0
    EP 633 learning finished: 0.009689665219720922, 0.006256633338467192
    634
    EP 634 simulation finished, start learning
    4.0
    EP 634 learning finished: 0.019305913416900466, 0.006219448036777209
    635
    EP 635 simulation finished, start learning
    4.0
    EP 635 learning finished: 0.028742024801466926, 0.00771685545020695
    636
    EP 636 simulation finished, start learning
    3.0
    EP 636 learning finished: 0.028491599414824265, 0.006726324186857961
    637
    EP 637 simulation finished, start learning
    5.0
    EP 637 learning finished: 0.028249480690710982, 0.007723218674948147
    638
    EP 638 simulation finished, start learning
    5.0
    EP 638 learning finished: 0.018701728248299077, 0.007237709994848037
    639
    EP 639 simulation finished, start learning
    3.0
    EP 639 learning finished: 0.02169338604689841, 0.006868135053816005
    640
    EP 640 simulation finished, start learning
    3.0
    EP 640 learning finished: 0.021583835829687814, 0.006825390121480386
    641
    EP 641 simulation finished, start learning
    3.0
    EP 641 learning finished: 0.01832907346058832, 0.007329100988949122
    642
    EP 642 simulation finished, start learning
    3.0
    EP 642 learning finished: 0.018278269114215817, 0.006277307761917446
    643
    EP 643 simulation finished, start learning
    3.0
    EP 643 learning finished: 0.01822446014901047, 0.005141301612728337
    644
    EP 644 simulation finished, start learning
    4.0
    EP 644 learning finished: 0.009318158957331235, 0.005073394088880671
    645
    EP 645 simulation finished, start learning
    5.0
    EP 645 learning finished: 0.00929968830527983, 0.004972037833134972
    646
    EP 646 simulation finished, start learning
    3.0
    EP 646 learning finished: 0.009276594886483025, 0.005041379058064048
    647
    EP 647 simulation finished, start learning
    3.0
    EP 647 learning finished: 0.009256573539047963, 0.005033392953243639
    648
    EP 648 simulation finished, start learning
    4.0
    EP 648 learning finished: 0.009235778989690857, 0.005025250278279714
    649
    EP 649 simulation finished, start learning
    4.0
    EP 649 learning finished: 0.018392252221353544, 0.00503781466440086
    650
    EP 650 simulation finished, start learning
    2.0
    EP 650 learning finished: 0.01829179689395488, 0.006574680005557572
    651
    EP 651 simulation finished, start learning
    3.0
    EP 651 learning finished: 0.018195167437092152, 0.007661267714428395
    652
    EP 652 simulation finished, start learning
    4.0
    EP 652 learning finished: 0.018100226567366207, 0.007636622215895154
    653
    EP 653 simulation finished, start learning
    3.0
    EP 653 learning finished: 0.01800075891057773, 0.0077052738574045305
    654
    EP 654 simulation finished, start learning
    5.0
    EP 654 learning finished: 0.01790547843531698, 0.006040232666520613
    655
    EP 655 simulation finished, start learning
    5.0
    EP 655 learning finished: 0.017811448102609262, 0.005927353149342451
    656
    EP 656 simulation finished, start learning
    4.0
    EP 656 learning finished: 0.008872203416551372, 0.005339066054207938
    657
    EP 657 simulation finished, start learning
    3.0
    EP 657 learning finished: 0.0072347934467345, 0.0038179440011649746
    658
    EP 658 simulation finished, start learning
    5.0
    EP 658 learning finished: 0.007226357675303774, 0.0038122231916985307
    659
    EP 659 simulation finished, start learning
    2.0
    EP 659 learning finished: 0.008893387660933172, 0.003989611329811164
    660
    EP 660 simulation finished, start learning
    5.0
    EP 660 learning finished: 0.015206113226998497, 0.0041405427273435436
    661
    EP 661 simulation finished, start learning
    5.0
    EP 661 learning finished: 0.014312666117263007, 0.004119466915916803
    662
    EP 662 simulation finished, start learning
    1.0
    EP 662 learning finished: 0.01589195758180395, 0.005137506739950837
    663
    EP 663 simulation finished, start learning
    3.0
    EP 663 learning finished: 0.00872614312863007, 0.003955825633485806
    664
    EP 664 simulation finished, start learning
    2.1
    EP 664 learning finished: 0.011985265500360498, 0.005535962771229919
    665
    EP 665 simulation finished, start learning
    3.1
    EP 665 learning finished: 0.018485742833071583, 0.007995402165072107
    666
    EP 666 simulation finished, start learning
    2.0
    EP 666 learning finished: 0.018426020119232334, 0.008572223485009781
    667
    EP 667 simulation finished, start learning
    4.0
    EP 667 learning finished: 0.018338018992206884, 0.009365121958043806
    668
    EP 668 simulation finished, start learning
    3.0
    EP 668 learning finished: 0.017654801699247713, 0.008298182166925608
    669
    EP 669 simulation finished, start learning
    5.0
    EP 669 learning finished: 0.008859640253895407, 0.0068249527199188975
    670
    EP 670 simulation finished, start learning
    5.0
    EP 670 learning finished: 0.008772075420022674, 0.005211377708398857
    671
    EP 671 simulation finished, start learning
    5.0
    EP 671 learning finished: 0.008752796288117537, 0.004377293911102716
    672
    EP 672 simulation finished, start learning
    3.0
    EP 672 learning finished: 0.00698002405209408, 0.0029595422492606125
    673
    EP 673 simulation finished, start learning
    3.0
    EP 673 learning finished: 0.005222467927879393, 0.0019811494571063313
    674
    EP 674 simulation finished, start learning
    3.0
    EP 674 learning finished: 0.008867177572199125, 0.0034948798877714182
    675
    EP 675 simulation finished, start learning
    2.0
    EP 675 learning finished: 0.00885613152334841, 0.0035863752137614093
    676
    EP 676 simulation finished, start learning
    5.0
    EP 676 learning finished: 0.00931261301384012, 0.0044727136158731655
    677
    EP 677 simulation finished, start learning
    5.0
    EP 677 learning finished: 0.008832904418920151, 0.004454717658375647
    678
    EP 678 simulation finished, start learning
    4.1
    EP 678 learning finished: 0.013783085041090715, 0.004932732460527944
    679
    EP 679 simulation finished, start learning
    5.0
    EP 679 learning finished: 0.012184052228800368, 0.00483071212904503
    680
    EP 680 simulation finished, start learning
    3.0
    EP 680 learning finished: 0.006856755380262314, 0.0038622754562922523
    681
    EP 681 simulation finished, start learning
    3.0
    EP 681 learning finished: 0.007616389701873061, 0.0038633329609260394
    682
    EP 682 simulation finished, start learning
    3.1
    EP 682 learning finished: 0.008801853751979394, 0.004312728616197248
    683
    EP 683 simulation finished, start learning
    4.0
    EP 683 learning finished: 0.01754276843967517, 0.005718275702601379
    684
    EP 684 simulation finished, start learning
    3.0
    EP 684 learning finished: 0.01744302266474307, 0.0071924127914027025
    685
    EP 685 simulation finished, start learning
    3.0
    EP 685 learning finished: 0.01735371187476531, 0.007171394329128944
    686
    EP 686 simulation finished, start learning
    3.0
    EP 686 learning finished: 0.01775244209245841, 0.0062356069403914915
    687
    EP 687 simulation finished, start learning
    4.0
    EP 687 learning finished: 0.01768305797945402, 0.005650719067370839
    688
    EP 688 simulation finished, start learning
    4.0
    EP 688 learning finished: 0.00892410253898035, 0.005542717270102433
    689
    EP 689 simulation finished, start learning
    3.1
    EP 689 learning finished: 0.0171951069783276, 0.00791399884545522
    690
    EP 690 simulation finished, start learning
    2.1
    EP 690 learning finished: 0.01783598121966179, 0.008893911777184016
    691
    EP 691 simulation finished, start learning
    3.0
    EP 691 learning finished: 0.01769977156221003, 0.009519056299286177
    692
    EP 692 simulation finished, start learning
    5.0
    EP 692 learning finished: 0.017604892403067396, 0.009075248112612991
    693
    EP 693 simulation finished, start learning
    5.0
    EP 693 learning finished: 0.01731714640450477, 0.007651153597777238
    694
    EP 694 simulation finished, start learning
    3.0
    EP 694 learning finished: 0.013488061943863983, 0.005268727939256479
    695
    EP 695 simulation finished, start learning
    3.0
    EP 695 learning finished: 0.013440985227621383, 0.005257076927326428
    696
    EP 696 simulation finished, start learning
    3.0
    EP 696 learning finished: 0.008508800395180316, 0.004304834345830955
    697
    EP 697 simulation finished, start learning
    3.0
    EP 697 learning finished: 0.01698440559696479, 0.004801100932818159
    698
    EP 698 simulation finished, start learning
    3.0
    EP 698 learning finished: 0.016935731771798856, 0.004717441431124551
    699
    EP 699 simulation finished, start learning
    3.0
    EP 699 learning finished: 0.016891755602742364, 0.004707531282537758
    700
    EP 700 simulation finished, start learning
    3.0
    EP 700 learning finished: 0.025249260467386847, 0.0061258990811582
    701
    EP 701 simulation finished, start learning
    0.0
    EP 701 learning finished: 0.025139074267659822, 0.006262162365673045
    702
    EP 702 simulation finished, start learning
    3.0
    EP 702 learning finished: 0.025030839775148195, 0.007687230580158892
    703
    EP 703 simulation finished, start learning
    5.0
    EP 703 learning finished: 0.0166300477013408, 0.007175890986651723
    704
    EP 704 simulation finished, start learning
    3.0
    EP 704 learning finished: 0.00949421249733362, 0.005745252304216446
    705
    EP 705 simulation finished, start learning
    5.0
    EP 705 learning finished: 0.008453719121696679, 0.005503432745642911
    706
    EP 706 simulation finished, start learning
    4.1
    EP 706 learning finished: 0.012324801936564711, 0.005950775513804629
    707
    EP 707 simulation finished, start learning
    3.0
    EP 707 learning finished: 0.01301199398458408, 0.006012064718906511
    708
    EP 708 simulation finished, start learning
    2.0
    EP 708 learning finished: 0.016532744209325045, 0.007472826220407569
    709
    EP 709 simulation finished, start learning
    4.1
    EP 709 learning finished: 0.013619353585532679, 0.007088315565224453
    710
    EP 710 simulation finished, start learning
    5.0
    EP 710 learning finished: 0.01356237693323692, 0.006106273769556599
    711
    EP 711 simulation finished, start learning
    4.1
    EP 711 learning finished: 0.01246851604392063, 0.0060708084406779035
    712
    EP 712 simulation finished, start learning
    4.0
    EP 712 learning finished: 0.017471115284685368, 0.005393967786782028
    713
    EP 713 simulation finished, start learning
    3.0
    EP 713 learning finished: 0.01120923880170821, 0.00581348624448393
    714
    EP 714 simulation finished, start learning
    4.0
    EP 714 learning finished: 0.011167302736723173, 0.006262647459144806
    715
    EP 715 simulation finished, start learning
    3.0
    EP 715 learning finished: 0.008465372334796939, 0.005375970800644225
    716
    EP 716 simulation finished, start learning
    3.0
    EP 716 learning finished: 0.008445762465143503, 0.00458015591891107
    717
    EP 717 simulation finished, start learning
    2.1
    EP 717 learning finished: 0.00842907558353545, 0.004140864916648705
    718
    EP 718 simulation finished, start learning
    4.0
    EP 718 learning finished: 0.008407127137108006, 0.004133786882505912
    719
    EP 719 simulation finished, start learning
    3.0
    EP 719 learning finished: 0.008391993041911405, 0.005434080733296653
    720
    EP 720 simulation finished, start learning
    3.0
    EP 720 learning finished: 0.008374788119610344, 0.0054243986066412715
    721
    EP 721 simulation finished, start learning
    3.0
    EP 721 learning finished: 0.016684892229434922, 0.005890712318231313
    722
    EP 722 simulation finished, start learning
    4.0
    EP 722 learning finished: 0.00831483177788428, 0.0053212789861719996
    723
    EP 723 simulation finished, start learning
    2.0
    EP 723 learning finished: 0.008292682760448145, 0.004016766324224245
    724
    EP 724 simulation finished, start learning
    4.1
    EP 724 learning finished: 0.0110323863361117, 0.005840396691856024
    725
    EP 725 simulation finished, start learning
    5.0
    EP 725 learning finished: 0.01099594612367838, 0.0044230408890980994
    726
    EP 726 simulation finished, start learning
    2.2
    EP 726 learning finished: 0.022968526445175996, 0.006272719894793018
    727
    EP 727 simulation finished, start learning
    3.0
    EP 727 learning finished: 0.022783961464863722, 0.007535463473783939
    728
    EP 728 simulation finished, start learning
    3.0
    EP 728 learning finished: 0.016689740695614574, 0.005696040403268654
    729
    EP 729 simulation finished, start learning
    3.0
    EP 729 learning finished: 0.016592778188261548, 0.007028439652678149
    730
    EP 730 simulation finished, start learning
    2.1
    EP 730 learning finished: 0.015975715357800624, 0.006750480911989598
    731
    EP 731 simulation finished, start learning
    4.0
    EP 731 learning finished: 0.010623044270843707, 0.006138746294748643
    732
    EP 732 simulation finished, start learning
    2.1
    EP 732 learning finished: 0.016412686160421686, 0.007026768311893861
    733
    EP 733 simulation finished, start learning
    1.1
    EP 733 learning finished: 0.022090102613461227, 0.007971639191741412
    734
    EP 734 simulation finished, start learning
    3.1
    EP 734 learning finished: 0.026553216738196994, 0.007269431453001
    735
    EP 735 simulation finished, start learning
    3.0
    EP 735 learning finished: 0.02169146200950678, 0.007901285812780046
    736
    EP 736 simulation finished, start learning
    2.1
    EP 736 learning finished: 0.02152316908312102, 0.007932334213631265
    737
    EP 737 simulation finished, start learning
    2.0
    EP 737 learning finished: 0.01575865296645418, 0.005709716929594914
    738
    EP 738 simulation finished, start learning
    5.0
    EP 738 learning finished: 0.008156307563023546, 0.004114116417142692
    739
    EP 739 simulation finished, start learning
    4.0
    EP 739 learning finished: 0.008139265773568383, 0.0040378845772490735
    740
    EP 740 simulation finished, start learning
    3.0
    EP 740 learning finished: 0.008122625045928089, 0.0030916776609786942
    741
    EP 741 simulation finished, start learning
    4.0
    EP 741 learning finished: 0.016187216157188544, 0.0043042259375264775
    742
    EP 742 simulation finished, start learning
    4.1
    EP 742 learning finished: 0.016104811020554166, 0.0060327544148227235
    743
    EP 743 simulation finished, start learning
    5.0
    EP 743 learning finished: 0.011606416114737783, 0.005596124592726663
    744
    EP 744 simulation finished, start learning
    4.0
    EP 744 learning finished: 0.010910850382177983, 0.006291917530779993
    745
    EP 745 simulation finished, start learning
    3.0
    EP 745 learning finished: 0.011556932437066791, 0.005003340583280409
    746
    EP 746 simulation finished, start learning
    3.0
    EP 746 learning finished: 0.006759430028647984, 0.0032486893718003764
    747
    EP 747 simulation finished, start learning
    4.0
    EP 747 learning finished: 0.008918981588394104, 0.0030804341131596877
    748
    EP 748 simulation finished, start learning
    3.0
    EP 748 learning finished: 0.004452158905035475, 0.0023774377800519866
    749
    EP 749 simulation finished, start learning
    3.1
    EP 749 learning finished: 0.009939604277509062, 0.004532574116535552
    750
    EP 750 simulation finished, start learning
    5.0
    EP 750 learning finished: 0.009904084723949325, 0.004531667041915692
    751
    EP 751 simulation finished, start learning
    4.1
    EP 751 learning finished: 0.010921905500495654, 0.00555374228190549
    752
    EP 752 simulation finished, start learning
    5.0
    EP 752 learning finished: 0.010886314987782697, 0.006403085510540289
    753
    EP 753 simulation finished, start learning
    3.0
    EP 753 learning finished: 0.01683044227524988, 0.005176245529791383
    754
    EP 754 simulation finished, start learning
    3.0
    EP 754 learning finished: 0.017417186053698686, 0.005160899585768301
    755
    EP 755 simulation finished, start learning
    3.0
    EP 755 learning finished: 0.012630467286228964, 0.0034339565135231473
    756
    EP 756 simulation finished, start learning
    3.0
    EP 756 learning finished: 0.007878951998442574, 0.0038791278384427605
    757
    EP 757 simulation finished, start learning
    3.0
    EP 757 learning finished: 0.015354698053819504, 0.00426309343704349
    758
    EP 758 simulation finished, start learning
    4.0
    EP 758 learning finished: 0.01531488616322818, 0.00554428081686531
    759
    EP 759 simulation finished, start learning
    2.1
    EP 759 learning finished: 0.023080340061678584, 0.007681999321561032
    760
    EP 760 simulation finished, start learning
    3.0
    EP 760 learning finished: 0.01585400952903626, 0.0077129193203410535
    761
    EP 761 simulation finished, start learning
    4.0
    EP 761 learning finished: 0.023632387664999754, 0.007670811196949974
    762
    EP 762 simulation finished, start learning
    4.0
    EP 762 learning finished: 0.023453816819717588, 0.007623749451342476
    763
    EP 763 simulation finished, start learning
    4.0
    EP 763 learning finished: 0.023268293303409848, 0.0061330189146418706
    764
    EP 764 simulation finished, start learning
    3.1
    EP 764 learning finished: 0.015422973359907388, 0.006326200812454275
    765
    EP 765 simulation finished, start learning
    3.1
    EP 765 learning finished: 0.01933049982157442, 0.007143580918080872
    766
    EP 766 simulation finished, start learning
    3.0
    EP 766 learning finished: 0.01918736118991049, 0.005889329908806455
    767
    EP 767 simulation finished, start learning
    3.0
    EP 767 learning finished: 0.014808858496486721, 0.005206780943934082
    768
    EP 768 simulation finished, start learning
    5.0
    EP 768 learning finished: 0.007614695465527177, 0.003699684173490425
    769
    EP 769 simulation finished, start learning
    5.0
    EP 769 learning finished: 0.0065612837511679345, 0.0024795368883422006
    770
    EP 770 simulation finished, start learning
    4.0
    EP 770 learning finished: 0.005914469498261354, 0.0031408093129344414
    771
    EP 771 simulation finished, start learning
    4.0
    EP 771 learning finished: 0.008449706062929607, 0.0038003771495513186
    772
    EP 772 simulation finished, start learning
    3.0
    EP 772 learning finished: 0.00842700639925198, 0.005066310392245181
    773
    EP 773 simulation finished, start learning
    -0.9
    EP 773 learning finished: 0.015188165648680751, 0.006442599037290719
    774
    EP 774 simulation finished, start learning
    3.0
    EP 774 learning finished: 0.022568650650999444, 0.007030477187641419
    775
    EP 775 simulation finished, start learning
    3.0
    EP 775 learning finished: 0.029839288415104193, 0.007606848973540655
    776
    EP 776 simulation finished, start learning
    3.0
    EP 776 learning finished: 0.022301403682245535, 0.006319968858027991
    777
    EP 777 simulation finished, start learning
    3.0
    EP 777 learning finished: 0.014685990230720591, 0.005412822100285079
    778
    EP 778 simulation finished, start learning
    3.0
    EP 778 learning finished: 0.007699975407902571, 0.005048227042318369
    779
    EP 779 simulation finished, start learning
    0.0
    EP 779 learning finished: 0.008911443078886805, 0.003917303446436075
    780
    EP 780 simulation finished, start learning
    3.0
    EP 780 learning finished: 0.008866001720260641, 0.0051403729168129075
    781
    EP 781 simulation finished, start learning
    3.0
    EP 781 learning finished: 0.008818473880069178, 0.0045573451342999505
    782
    EP 782 simulation finished, start learning
    4.1
    EP 782 learning finished: 0.00932681234386834, 0.005293099569320485
    783
    EP 783 simulation finished, start learning
    4.0
    EP 783 learning finished: 0.013425149829426886, 0.005811571081894229
    784
    EP 784 simulation finished, start learning
    3.0
    EP 784 learning finished: 0.013355207362359067, 0.0046350915886896145
    785
    EP 785 simulation finished, start learning
    4.0
    EP 785 learning finished: 0.009201744844389736, 0.005144225925675983
    786
    EP 786 simulation finished, start learning
    2.0
    EP 786 learning finished: 0.007577626657404579, 0.004817022077043968
    787
    EP 787 simulation finished, start learning
    3.0
    EP 787 learning finished: 0.007562645601804974, 0.004230083714286792
    788
    EP 788 simulation finished, start learning
    3.0
    EP 788 learning finished: 0.014476840406288716, 0.005391043361129277
    789
    EP 789 simulation finished, start learning
    2.1
    EP 789 learning finished: 0.02180036765630433, 0.006180954061909105
    790
    EP 790 simulation finished, start learning
    3.0
    EP 790 learning finished: 0.014521177779400318, 0.004879022650412661
    791
    EP 791 simulation finished, start learning
    4.0
    EP 791 learning finished: 0.01461218188709168, 0.005430763911016119
    792
    EP 792 simulation finished, start learning
    4.0
    EP 792 learning finished: 0.013172647904553969, 0.004840832416556751
    793
    EP 793 simulation finished, start learning
    3.0
    EP 793 learning finished: 0.00808322082183821, 0.002832684707184304
    794
    EP 794 simulation finished, start learning
    4.0
    EP 794 learning finished: 0.008059482336108204, 0.0040519218722619614
    795
    EP 795 simulation finished, start learning
    3.0
    EP 795 learning finished: 0.007542225183126305, 0.0046354513953007865
    796
    EP 796 simulation finished, start learning
    5.0
    EP 796 learning finished: 0.007528230498213162, 0.00393709224002355
    797
    EP 797 simulation finished, start learning
    5.0
    EP 797 learning finished: 0.007513696036543394, 0.0039361649355917905
    798
    EP 798 simulation finished, start learning
    4.0
    EP 798 learning finished: 0.0071009740070806515, 0.00335179615520372
    799
    EP 799 simulation finished, start learning
    3.0
    EP 799 learning finished: 0.004073502086583858, 0.002126161715652941
    800
    EP 800 simulation finished, start learning
    3.0
    EP 800 learning finished: 0.006237652365499002, 0.003047438224684266
    801
    EP 801 simulation finished, start learning
    4.0
    EP 801 learning finished: 0.007515527913700026, 0.004254892714330438
    802
    EP 802 simulation finished, start learning
    3.0
    EP 802 learning finished: 0.007496384004685552, 0.00361598850119312
    803
    EP 803 simulation finished, start learning
    4.0
    EP 803 learning finished: 0.007483684327539208, 0.00425543110199141
    804
    EP 804 simulation finished, start learning
    3.0
    EP 804 learning finished: 0.007466694428167386, 0.004597069435541575
    805
    EP 805 simulation finished, start learning
    4.1
    EP 805 learning finished: 0.014284070423144613, 0.005303811695820097
    806
    EP 806 simulation finished, start learning
    4.0
    EP 806 learning finished: 0.014246373633959308, 0.005916902947497841
    807
    EP 807 simulation finished, start learning
    3.1
    EP 807 learning finished: 0.014326326424473734, 0.007246536369923007
    808
    EP 808 simulation finished, start learning
    3.0
    EP 808 learning finished: 0.013766018776880884, 0.006033168376232467
    809
    EP 809 simulation finished, start learning
    4.1
    EP 809 learning finished: 0.013699660060840557, 0.004826686844947285
    810
    EP 810 simulation finished, start learning
    5.0
    EP 810 learning finished: 0.00974257855806826, 0.004206470906741681
    811
    EP 811 simulation finished, start learning
    3.1
    EP 811 learning finished: 0.01358280542151391, 0.0036119629811626197
    812
    EP 812 simulation finished, start learning
    3.0
    EP 812 learning finished: 0.013513614511282412, 0.0047847280610623
    813
    EP 813 simulation finished, start learning
    4.0
    EP 813 learning finished: 0.008652203741991826, 0.005279548211121347
    814
    EP 814 simulation finished, start learning
    5.0
    EP 814 learning finished: 0.008628867407956875, 0.005269423675226648
    815
    EP 815 simulation finished, start learning
    4.0
    EP 815 learning finished: 0.007384196163922496, 0.004512552603278171
    816
    EP 816 simulation finished, start learning
    3.0
    EP 816 learning finished: 0.007371247438772599, 0.00450730992813142
    817
    EP 817 simulation finished, start learning
    3.0
    EP 817 learning finished: 0.013828138459917483, 0.004490541643677422
    818
    EP 818 simulation finished, start learning
    3.0
    EP 818 learning finished: 0.013796835980652844, 0.005734431393725241
    819
    EP 819 simulation finished, start learning
    3.0
    EP 819 learning finished: 0.013762089667448407, 0.005979556082164568
    820
    EP 820 simulation finished, start learning
    3.0
    EP 820 learning finished: 0.013733105651105859, 0.005966511170090764
    821
    EP 821 simulation finished, start learning
    3.0
    EP 821 learning finished: 0.007351447375242515, 0.004772394220150152
    822
    EP 822 simulation finished, start learning
    5.0
    EP 822 learning finished: 0.011652847479144413, 0.0043056181685473165
    823
    EP 823 simulation finished, start learning
    2.1
    EP 823 learning finished: 0.007200556602750652, 0.004237110728649777
    824
    EP 824 simulation finished, start learning
    4.0
    EP 824 learning finished: 0.008582930000823552, 0.003668288778971037
    825
    EP 825 simulation finished, start learning
    2.1
    EP 825 learning finished: 0.013298337223474732, 0.005569971679124801
    826
    EP 826 simulation finished, start learning
    3.0
    EP 826 learning finished: 0.013927684945461039, 0.005924546124321732
    827
    EP 827 simulation finished, start learning
    3.0
    EP 827 learning finished: 0.020347796000891094, 0.006275234514795021
    828
    EP 828 simulation finished, start learning
    3.0
    EP 828 learning finished: 0.02025221468995042, 0.006910590535030181
    829
    EP 829 simulation finished, start learning
    5.0
    EP 829 learning finished: 0.013381450026916042, 0.005807024603048679
    830
    EP 830 simulation finished, start learning
    5.0
    EP 830 learning finished: 0.007294289960125155, 0.004662162455474572
    831
    EP 831 simulation finished, start learning
    5.0
    EP 831 learning finished: 0.01029796950703954, 0.004293923214959842
    832
    EP 832 simulation finished, start learning
    5.0
    EP 832 learning finished: 0.009699857156430625, 0.003054535172899738
    833
    EP 833 simulation finished, start learning
    4.1
    EP 833 learning finished: 0.004844192678692849, 0.0030053196485943325
    834
    EP 834 simulation finished, start learning
    4.0
    EP 834 learning finished: 0.008449013353811097, 0.003588344445101773
    835
    EP 835 simulation finished, start learning
    3.0
    EP 835 learning finished: 0.00842744205707291, 0.002810618522398537
    836
    EP 836 simulation finished, start learning
    2.0
    EP 836 learning finished: 0.00839951233932723, 0.003997610473727302
    837
    EP 837 simulation finished, start learning
    5.0
    EP 837 learning finished: 0.006664339687670662, 0.0032720545188631434
    838
    EP 838 simulation finished, start learning
    4.1
    EP 838 learning finished: 0.006763056181984695, 0.004172868591923044
    839
    EP 839 simulation finished, start learning
    3.1
    EP 839 learning finished: 0.012988767378654842, 0.005464439551171078
    840
    EP 840 simulation finished, start learning
    4.0
    EP 840 learning finished: 0.01292833358009915, 0.005446772928320816
    841
    EP 841 simulation finished, start learning
    3.0
    EP 841 learning finished: 0.012861255880768141, 0.005429749352759454
    842
    EP 842 simulation finished, start learning
    3.0
    EP 842 learning finished: 0.008232939095975667, 0.005066756510717123
    843
    EP 843 simulation finished, start learning
    3.0
    EP 843 learning finished: 0.007262536375328921, 0.0037788392700885964
    844
    EP 844 simulation finished, start learning
    3.0
    EP 844 learning finished: 0.013193931076674659, 0.003723300684371248
    845
    EP 845 simulation finished, start learning
    1.2
    EP 845 learning finished: 0.019940129086919, 0.006245423661937366
    846
    EP 846 simulation finished, start learning
    3.0
    EP 846 learning finished: 0.013666568917976463, 0.006776434968765936
    847
    EP 847 simulation finished, start learning
    5.0
    EP 847 learning finished: 0.013597130490108356, 0.007515400823540001
    848
    EP 848 simulation finished, start learning
    5.0
    EP 848 learning finished: 0.013529961007723301, 0.007142555045470307
    849
    EP 849 simulation finished, start learning
    5.0
    EP 849 learning finished: 0.009513767678715368, 0.004644055650401195
    850
    EP 850 simulation finished, start learning
    4.1
    EP 850 learning finished: 0.014244583297511326, 0.00443711254115738
    851
    EP 851 simulation finished, start learning
    3.0
    EP 851 learning finished: 0.010034475016340139, 0.003732129145400146
    852
    EP 852 simulation finished, start learning
    4.0
    EP 852 learning finished: 0.007174212302121674, 0.004131966990704801
    853
    EP 853 simulation finished, start learning
    3.0
    EP 853 learning finished: 0.0071603270957254495, 0.004121363357622106
    854
    EP 854 simulation finished, start learning
    2.2
    EP 854 learning finished: 0.014269829555438207, 0.005205105685982166
    855
    EP 855 simulation finished, start learning
    4.0
    EP 855 learning finished: 0.021283043561869164, 0.00627888244216616
    856
    EP 856 simulation finished, start learning
    4.0
    EP 856 learning finished: 0.021134229753169365, 0.006249683824169831
    857
    EP 857 simulation finished, start learning
    3.0
    EP 857 learning finished: 0.02098913204785191, 0.006220923055162568
    858
    EP 858 simulation finished, start learning
    4.0
    EP 858 learning finished: 0.02084462794077524, 0.00482064323085396
    859
    EP 859 simulation finished, start learning
    3.0
    EP 859 learning finished: 0.01382079815111581, 0.004780334849828686
    860
    EP 860 simulation finished, start learning
    1.1
    EP 860 learning finished: 0.006888282256323208, 0.004399688622258442
    861
    EP 861 simulation finished, start learning
    4.0
    EP 861 learning finished: 0.007928491420443606, 0.00494389469775965
    862
    EP 862 simulation finished, start learning
    3.0
    EP 862 learning finished: 0.007905681567564216, 0.004992130047466385
    863
    EP 863 simulation finished, start learning
    5.0
    EP 863 learning finished: 0.00788080641142841, 0.0038946282610510252
    864
    EP 864 simulation finished, start learning
    5.0
    EP 864 learning finished: 0.0068359131826198904, 0.003900205253725512
    865
    EP 865 simulation finished, start learning
    5.0
    EP 865 learning finished: 0.00682442922635218, 0.0033525735509947197
    866
    EP 866 simulation finished, start learning
    5.0
    EP 866 learning finished: 0.009352100430845667, 0.0029370534402749135
    867
    EP 867 simulation finished, start learning
    3.0
    EP 867 learning finished: 0.014546744872330208, 0.003728583586791364
    868
    EP 868 simulation finished, start learning
    2.1
    EP 868 learning finished: 0.010395627884239644, 0.004750750319605738
    869
    EP 869 simulation finished, start learning
    3.0
    EP 869 learning finished: 0.012947048183015397, 0.005825269017286416
    870
    EP 870 simulation finished, start learning
    2.1
    EP 870 learning finished: 0.019409372403977443, 0.006830802534147769
    871
    EP 871 simulation finished, start learning
    3.0
    EP 871 learning finished: 0.019306313557205357, 0.006012028401499947
    872
    EP 872 simulation finished, start learning
    3.0
    EP 872 learning finished: 0.012773732224967915, 0.004242231575108946
    873
    EP 873 simulation finished, start learning
    4.0
    EP 873 learning finished: 0.006837320751067466, 0.004267790659105468
    874
    EP 874 simulation finished, start learning
    0.0
    EP 874 learning finished: 0.006825808151269541, 0.002622826188182733
    875
    EP 875 simulation finished, start learning
    4.0
    EP 875 learning finished: 0.013603944812743407, 0.003720528360002954
    876
    EP 876 simulation finished, start learning
    4.0
    EP 876 learning finished: 0.020296670602558464, 0.004803188167035917
    877
    EP 877 simulation finished, start learning
    4.0
    EP 877 learning finished: 0.020156745547957655, 0.004778596056029372
    878
    EP 878 simulation finished, start learning
    3.0
    EP 878 learning finished: 0.02002319656466156, 0.004649624343175063
    879
    EP 879 simulation finished, start learning
    4.0
    EP 879 learning finished: 0.019892050074798107, 0.004626115537220699
    880
    EP 880 simulation finished, start learning
    4.0
    EP 880 learning finished: 0.01976230917634325, 0.004603081506716696
    881
    EP 881 simulation finished, start learning
    3.0
    EP 881 learning finished: 0.013104433142057559, 0.00459940897599805
    882
    EP 882 simulation finished, start learning
    3.0
    EP 882 learning finished: 0.013049927702631692, 0.0056616833137492035
    883
    EP 883 simulation finished, start learning
    5.0
    EP 883 learning finished: 0.01887620104622556, 0.005614597720477714
    884
    EP 884 simulation finished, start learning
    5.0
    EP 884 learning finished: 0.018813688424124564, 0.004555029148978207
    885
    EP 885 simulation finished, start learning
    4.1
    EP 885 learning finished: 0.012580778152138095, 0.004881592487524363
    886
    EP 886 simulation finished, start learning
    5.0
    EP 886 learning finished: 0.009120179010924545, 0.0045407454745761315
    887
    EP 887 simulation finished, start learning
    4.0
    EP 887 learning finished: 0.009108998330109586, 0.004073040205616015
    888
    EP 888 simulation finished, start learning
    3.0
    EP 888 learning finished: 0.009622405215836238, 0.004060651846331235
    889
    EP 889 simulation finished, start learning
    3.0
    EP 889 learning finished: 0.006513788303678325, 0.003768117842290882
    890
    EP 890 simulation finished, start learning
    4.0
    EP 890 learning finished: 0.006898315383736531, 0.0035796457645940194
    891
    EP 891 simulation finished, start learning
    2.0
    EP 891 learning finished: 0.006488815602679665, 0.0030835521045181244
    892
    EP 892 simulation finished, start learning
    5.0
    EP 892 learning finished: 0.006477444327931181, 0.003804316756015188
    893
    EP 893 simulation finished, start learning
    4.0
    EP 893 learning finished: 0.006868141577150766, 0.003240217517203566
    894
    EP 894 simulation finished, start learning
    3.0
    EP 894 learning finished: 0.006068819487627053, 0.0026934834148324427
    895
    EP 895 simulation finished, start learning
    4.0
    EP 895 learning finished: 0.006473559629220915, 0.0036865089040395795
    896
    EP 896 simulation finished, start learning
    4.0
    EP 896 learning finished: 0.0068356567722922534, 0.003499312527299736
    897
    EP 897 simulation finished, start learning
    2.0
    EP 897 learning finished: 0.007696563375116483, 0.004452924234851589
    898
    EP 898 simulation finished, start learning
    3.0
    EP 898 learning finished: 0.007672929307304344, 0.0052144188573066685
    899
    EP 899 simulation finished, start learning
    3.0
    EP 899 learning finished: 0.007653307820124411, 0.005256404790948466
    900
    EP 900 simulation finished, start learning
    5.0
    EP 900 learning finished: 0.0063170256722600016, 0.004717697965769929
    901
    EP 901 simulation finished, start learning
    5.0
    EP 901 learning finished: 0.009961893009922562, 0.0039393138370199905
    902
    EP 902 simulation finished, start learning
    3.1
    EP 902 learning finished: 0.007630262244230579, 0.004358332819663839
    903
    EP 903 simulation finished, start learning
    3.0
    EP 903 learning finished: 0.00760882898484283, 0.004325146164141221
    904
    EP 904 simulation finished, start learning
    3.1
    EP 904 learning finished: 0.01180700974692428, 0.005993945197290647
    905
    EP 905 simulation finished, start learning
    3.0
    EP 905 learning finished: 0.012668401999171408, 0.006298827812724876
    906
    EP 906 simulation finished, start learning
    4.1
    EP 906 learning finished: 0.012623982339729167, 0.005761927748005509
    907
    EP 907 simulation finished, start learning
    4.1
    EP 907 learning finished: 0.012527214010567889, 0.005354885890981838
    908
    EP 908 simulation finished, start learning
    3.1
    EP 908 learning finished: 0.015749812902324578, 0.004835678588331523
    909
    EP 909 simulation finished, start learning
    4.0
    EP 909 learning finished: 0.01565248238365058, 0.004820431953016069
    910
    EP 910 simulation finished, start learning
    2.1
    EP 910 learning finished: 0.01555858392054077, 0.004797650222008916
    911
    EP 911 simulation finished, start learning
    3.0
    EP 911 learning finished: 0.011393287294570342, 0.0041460406194886446
    912
    EP 912 simulation finished, start learning
    3.0
    EP 912 learning finished: 0.006285219568841427, 0.0030002184918587935
    913
    EP 913 simulation finished, start learning
    3.0
    EP 913 learning finished: 0.006204334064584316, 0.0030137107050910807
    914
    EP 914 simulation finished, start learning
    3.0
    EP 914 learning finished: 0.012110461989177423, 0.003409568606347335
    915
    EP 915 simulation finished, start learning
    2.0
    EP 915 learning finished: 0.012085926542948044, 0.003455480856236074
    916
    EP 916 simulation finished, start learning
    5.0
    EP 916 learning finished: 0.012061222670521232, 0.004156267704197821
    917
    EP 917 simulation finished, start learning
    5.0
    EP 917 learning finished: 0.01031718158444761, 0.003829309092003186
    918
    EP 918 simulation finished, start learning
    3.1
    EP 918 learning finished: 0.009784530508913036, 0.00393343641315802
    919
    EP 919 simulation finished, start learning
    3.0
    EP 919 learning finished: 0.009261106458865243, 0.004899411496542901
    920
    EP 920 simulation finished, start learning
    3.0
    EP 920 learning finished: 0.00724829853795228, 0.004187712456908225
    921
    EP 921 simulation finished, start learning
    3.0
    EP 921 learning finished: 0.0072291921009224325, 0.004550515217282817
    922
    EP 922 simulation finished, start learning
    4.0
    EP 922 learning finished: 0.006290300264954141, 0.003921383892025195
    923
    EP 923 simulation finished, start learning
    3.0
    EP 923 learning finished: 0.0062762625081305146, 0.0028972655014000517
    924
    EP 924 simulation finished, start learning
    3.0
    EP 924 learning finished: 0.006267534563349386, 0.002893909171547193
    925
    EP 925 simulation finished, start learning
    3.0
    EP 925 learning finished: 0.005987214173650557, 0.0028453873554716105
    926
    EP 926 simulation finished, start learning
    0.0
    EP 926 learning finished: 0.011955069382793226, 0.003456311242845326
    927
    EP 927 simulation finished, start learning
    3.0
    EP 927 learning finished: 0.011931255927454032, 0.003450269143285093
    928
    EP 928 simulation finished, start learning
    3.0
    EP 928 learning finished: 0.011904947051398551, 0.003443134886527588
    929
    EP 929 simulation finished, start learning
    3.0
    EP 929 learning finished: 0.011881212812109965, 0.0034370648790655878
    930
    EP 930 simulation finished, start learning
    4.0
    EP 930 learning finished: 0.006272764994026403, 0.0033438651567951894
    931
    EP 931 simulation finished, start learning
    3.0
    EP 931 learning finished: 0.011845975910446782, 0.004348040700289009
    932
    EP 932 simulation finished, start learning
    3.0
    EP 932 learning finished: 0.01182193528105202, 0.004341121895167888
    933
    EP 933 simulation finished, start learning
    3.0
    EP 933 learning finished: 0.006238820946094847, 0.0033294467955589844
    934
    EP 934 simulation finished, start learning
    3.0
    EP 934 learning finished: 0.005898001994829372, 0.002315220426507262
    935
    EP 935 simulation finished, start learning
    3.0
    EP 935 learning finished: 0.003120413665783417, 0.0013091281909520802
    936
    EP 936 simulation finished, start learning
    3.0
    EP 936 learning finished: 0.003116746942737336, 0.0013074859372507557
    937
    EP 937 simulation finished, start learning
    3.0
    EP 937 learning finished: 0.0031127032623814443, 0.001305841600463727
    938
    EP 938 simulation finished, start learning
    4.0
    EP 938 learning finished: 0.006241540682871971, 0.00231741393791296
    939
    EP 939 simulation finished, start learning
    3.0
    EP 939 learning finished: 0.006234344505500644, 0.0023149284992882713
    940
    EP 940 simulation finished, start learning
    4.0
    EP 940 learning finished: 0.012431618587008586, 0.0033188357100311094
    941
    EP 941 simulation finished, start learning
    4.0
    EP 941 learning finished: 0.018544993210408478, 0.004308739132460104
    942
    EP 942 simulation finished, start learning
    3.0
    EP 942 learning finished: 0.012302062903415845, 0.0043003469508064995
    943
    EP 943 simulation finished, start learning
    2.1
    EP 943 learning finished: 0.012252576955788186, 0.00591310058175291
    944
    EP 944 simulation finished, start learning
    2.0
    EP 944 learning finished: 0.011806580161632496, 0.004958658768916398
    945
    EP 945 simulation finished, start learning
    5.0
    EP 945 learning finished: 0.011769774702678149, 0.004640235357180463
    946
    EP 946 simulation finished, start learning
    5.0
    EP 946 learning finished: 0.00991726150954264, 0.004313555497454639
    947
    EP 947 simulation finished, start learning
    4.0
    EP 947 learning finished: 0.009405261988455038, 0.0032016288827924435
    948
    EP 948 simulation finished, start learning
    4.0
    EP 948 learning finished: 0.008426070496259541, 0.004140436326505181
    949
    EP 949 simulation finished, start learning
    3.0
    EP 949 learning finished: 0.006096752366289593, 0.004454178515588936
    950
    EP 950 simulation finished, start learning
    2.0
    EP 950 learning finished: 0.006086784612083562, 0.0038207524679041427
    951
    EP 951 simulation finished, start learning
    3.1
    EP 951 learning finished: 0.0072152286684747935, 0.004428814485262222
    952
    EP 952 simulation finished, start learning
    3.0
    EP 952 learning finished: 0.007195385799441567, 0.0044692795955381115
    953
    EP 953 simulation finished, start learning
    3.0
    EP 953 learning finished: 0.007175335593142762, 0.00352370925962416
    954
    EP 954 simulation finished, start learning
    4.0
    EP 954 learning finished: 0.012074922439186508, 0.004449140005323334
    955
    EP 955 simulation finished, start learning
    3.0
    EP 955 learning finished: 0.01202863349027794, 0.004321591974505878
    956
    EP 956 simulation finished, start learning
    2.0
    EP 956 learning finished: 0.005996669195653226, 0.0033409317623848502
    957
    EP 957 simulation finished, start learning
    5.0
    EP 957 learning finished: 0.0059865897145279234, 0.0032910655457486135
    958
    EP 958 simulation finished, start learning
    2.0
    EP 958 learning finished: 0.006549289063304542, 0.0030805985567473235
    959
    EP 959 simulation finished, start learning
    5.0
    EP 959 learning finished: 0.0060546076139971206, 0.002094732234778481
    960
    EP 960 simulation finished, start learning
    2.0
    EP 960 learning finished: 0.010178321545236457, 0.002801815351750392
    961
    EP 961 simulation finished, start learning
    3.0
    EP 961 learning finished: 0.010605395789607286, 0.0028384692386615265
    962
    EP 962 simulation finished, start learning
    3.0
    EP 962 learning finished: 0.005982394976225436, 0.002077323099843061
    963
    EP 963 simulation finished, start learning
    3.1
    EP 963 learning finished: 0.0059934018708673875, 0.0036491555398428708
    964
    EP 964 simulation finished, start learning
    4.0
    EP 964 learning finished: 0.011952975190929749, 0.003863800339946266
    965
    EP 965 simulation finished, start learning
    3.1
    EP 965 learning finished: 0.017839207289661374, 0.005374899145258165
    966
    EP 966 simulation finished, start learning
    4.0
    EP 966 learning finished: 0.023617438216131603, 0.006310062373475016
    967
    EP 967 simulation finished, start learning
    3.0
    EP 967 learning finished: 0.01758415629847576, 0.00469679597476258
    968
    EP 968 simulation finished, start learning
    3.2
    EP 968 learning finished: 0.01190151258260741, 0.0056086440547898535
    969
    EP 969 simulation finished, start learning
    4.0
    EP 969 learning finished: 0.01102937818778571, 0.004541969015035008
    970
    EP 970 simulation finished, start learning
    3.0
    EP 970 learning finished: 0.010980602212313784, 0.004578721551207347
    971
    EP 971 simulation finished, start learning
    5.0
    EP 971 learning finished: 0.010931015972811764, 0.005247679974533644
    972
    EP 972 simulation finished, start learning
    5.0
    EP 972 learning finished: 0.0057851837004783135, 0.003390298247938498
    973
    EP 973 simulation finished, start learning
    4.0
    EP 973 learning finished: 0.0057765206218043375, 0.003386069840276562
    974
    EP 974 simulation finished, start learning
    3.0
    EP 974 learning finished: 0.0045357345176775254, 0.0024004937303665486
    975
    EP 975 simulation finished, start learning
    2.1
    EP 975 learning finished: 0.006972828114374607, 0.002340399530943079
    976
    EP 976 simulation finished, start learning
    3.0
    EP 976 learning finished: 0.006954728379323427, 0.0033106095368093946
    977
    EP 977 simulation finished, start learning
    3.0
    EP 977 learning finished: 0.0057645225161735425, 0.002872251384033575
    978
    EP 978 simulation finished, start learning
    3.1
    EP 978 learning finished: 0.011495851679409697, 0.004398469389785238
    979
    EP 979 simulation finished, start learning
    4.0
    EP 979 learning finished: 0.017159780216381648, 0.004718989742061197
    980
    EP 980 simulation finished, start learning
    3.0
    EP 980 learning finished: 0.011386457394685578, 0.0037346910043171513
    981
    EP 981 simulation finished, start learning
    4.0
    EP 981 learning finished: 0.016992025050019066, 0.004591571970889182
    982
    EP 982 simulation finished, start learning
    2.1
    EP 982 learning finished: 0.011273267811336751, 0.003662037260191653
    983
    EP 983 simulation finished, start learning
    3.0
    EP 983 learning finished: 0.005622068259462765, 0.0027436709516539537
    984
    EP 984 simulation finished, start learning
    3.0
    EP 984 learning finished: 0.00561404461804571, 0.002740944555871039
    985
    EP 985 simulation finished, start learning
    3.0
    EP 985 learning finished: 0.0038407130832567304, 0.001828868873449352
    986
    EP 986 simulation finished, start learning
    3.0
    EP 986 learning finished: 0.0057350570575991355, 0.002208647252389303
    987
    EP 987 simulation finished, start learning
    4.0
    EP 987 learning finished: 0.00572996355055011, 0.003116923187494199
    988
    EP 988 simulation finished, start learning
    3.1
    EP 988 learning finished: 0.011190591195404664, 0.004613762387172334
    989
    EP 989 simulation finished, start learning
    3.0
    EP 989 learning finished: 0.011620228412372313, 0.005560545705275051
    990
    EP 990 simulation finished, start learning
    4.0
    EP 990 learning finished: 0.011105662149127227, 0.0050556620504623355
    991
    EP 991 simulation finished, start learning
    3.0
    EP 991 learning finished: 0.011066866439918499, 0.005088988170959041
    992
    EP 992 simulation finished, start learning
    5.0
    EP 992 learning finished: 0.005770978315965181, 0.004242369828537207
    993
    EP 993 simulation finished, start learning
    3.0
    EP 993 learning finished: 0.008939113184890601, 0.003974606141468347
    994
    EP 994 simulation finished, start learning
    3.0
    EP 994 learning finished: 0.009370378050629569, 0.004442310134903001
    995
    EP 995 simulation finished, start learning
    5.0
    EP 995 learning finished: 0.01288557052973971, 0.004140905506214783
    996
    EP 996 simulation finished, start learning
    2.1
    EP 996 learning finished: 0.009310059594799469, 0.004599311963909021
    997
    EP 997 simulation finished, start learning
    4.0
    EP 997 learning finished: 0.009793158297537019, 0.00438087200233106
    998
    EP 998 simulation finished, start learning
    3.0
    EP 998 learning finished: 0.009752225713601348, 0.0034283705445273447
    999
    EP 999 simulation finished, start learning
    3.0
    EP 999 learning finished: 0.009713502609681512, 0.0037385371966050704



```python
v_model, pi_model= q_model.q.yield_v_pi()
```


```python
v_model
```




    0    30.251043
    1    35.767538
    2    40.763216
    dtype: float64




```python
pi_model
```




    0    1
    1    1
    2   -1
    dtype: int64




```python

```


```python
for s in state_space:
    for a in control_space:
        print(f'state: {s}, action_buffer: {[a]}, action to take: {q_model.q.max_value_act(model_model(s, [a]))[1]}')
        
def pi_model(i):
    s, a = i
    return q_model.q.max_value_act(model_model(s, [a]))[1]
```

    state: 0, action_buffer: [-1], action to take: 1
    state: 0, action_buffer: [1], action to take: 1
    state: 1, action_buffer: [-1], action to take: 1
    state: 1, action_buffer: [1], action to take: -1
    state: 2, action_buffer: [-1], action to take: 1
    state: 2, action_buffer: [1], action to take: -1



```python
def pi_wrapper(pi):
    def ret(x):
        return pi[x]
    return ret

def eval_pi_model(env, pi, model, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = pi(model(obs, action_buffer))
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        
    return np.mean(r_ls), r_ls.mean(axis = 1)

def cummean(arr):
    return np.cumsum(arr)/np.arange(1, len(arr)+1)
```


```python
val, v_samples_lin = eval_pi_model(env, pi_wrapper(pi_l), model_model, 0.95)
val
```




    4.875468543105966




```python
exp2_model = v_samples_lin
```


```python
sns.distplot(v_samples_lin, label = 'model')
plt.legend()
```


```python
val, v_samples_lin = eval_pi_model(env, lambda x: np.random.choice([-1,1]), model_model, 0.95)
val
```




    -0.0023362149317737234




```python
exp2_random = v_samples_lin
```


```python
sns.distplot(v_samples_lin, label = 'random benchmark')
plt.legend()
```


```python
# Q-learning using model, async reward collection and optimal action selection



class Q_learning:
    def __init__(self, q, env, delay_step, model, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(400)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        model = self.model
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            for _ in range(100):
                
                
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (get_opt_act(q, model, obs, self.action_memory), False)
                last_action = self.action_memory.pop(0)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q

def get_opt_act(q, model, x, acts):
    act_idx = np.argmax([np.sum([q[s, u]*model(s, x, acts) for s in q.state_space]) for u in q.control_space])
    return q.control_space[act_idx]
```


```python
# P(s|x, a_1, a_2, ...)
def model(s, x, acts):
    return trans_prob(s, x, acts[0])
```


```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
# lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_opt = Q_learning(q, env, 1, model, MAX_ITER = 1000)
learned_q = q_opt.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    -0.42
    EP 3 learning finished: 14.605154166666667, 7.925828873526876
    4
    EP 4 simulation finished, start learning
    4.23
    EP 4 learning finished: 11.174326548378074, 6.106331593288012
    5
    EP 5 simulation finished, start learning
    4.22
    EP 5 learning finished: 3.9503093926619712, 2.67502396764157
    6
    EP 6 simulation finished, start learning
    4.02
    EP 6 learning finished: 2.3821913968352426, 1.7846461414480237
    7
    EP 7 simulation finished, start learning
    3.63
    EP 7 learning finished: 1.7540001958894287, 1.1860228996930828
    8
    EP 8 simulation finished, start learning
    4.91
    EP 8 learning finished: 1.155232159564413, 0.7591389838163316
    9
    EP 9 simulation finished, start learning
    4.53
    EP 9 learning finished: 0.9667904691155087, 0.6433414374229048
    10
    EP 10 simulation finished, start learning
    4.91
    EP 10 learning finished: 0.7630439542378724, 0.49595292183658984
    11
    EP 11 simulation finished, start learning
    4.14
    EP 11 learning finished: 0.8155722903675944, 0.44790107469077495
    12
    EP 12 simulation finished, start learning
    4.33
    EP 12 learning finished: 0.8847894074116383, 0.45801466354117054
    13
    EP 13 simulation finished, start learning
    4.13
    EP 13 learning finished: 0.782381074201842, 0.45024093356765427
    14
    EP 14 simulation finished, start learning
    4.61
    EP 14 learning finished: 0.7098535856719099, 0.4335257724939714
    15
    EP 15 simulation finished, start learning
    4.53
    EP 15 learning finished: 0.6020613050517056, 0.37757019328716207
    16
    EP 16 simulation finished, start learning
    4.52
    EP 16 learning finished: 0.6087425444809078, 0.34088763950052847
    17
    EP 17 simulation finished, start learning
    4.02
    EP 17 learning finished: 0.5368654990832518, 0.29311379598029674
    18
    EP 18 simulation finished, start learning
    4.52
    EP 18 learning finished: 0.488609247320408, 0.27951741520122003
    19
    EP 19 simulation finished, start learning
    4.23
    EP 19 learning finished: 0.45661023224871755, 0.2630316866339797
    20
    EP 20 simulation finished, start learning
    3.84
    EP 20 learning finished: 0.47746765240248834, 0.2669158906321094
    21
    EP 21 simulation finished, start learning
    4.23
    EP 21 learning finished: 0.537111702900873, 0.24994761353942074
    22
    EP 22 simulation finished, start learning
    4.11
    EP 22 learning finished: 0.4684700878881465, 0.22870415387506782
    23
    EP 23 simulation finished, start learning
    3.62
    EP 23 learning finished: 0.4520196956381106, 0.22013145129384468
    24
    EP 24 simulation finished, start learning
    4.33
    EP 24 learning finished: 0.3374945592392713, 0.18752488633895018
    25
    EP 25 simulation finished, start learning
    4.71
    EP 25 learning finished: 0.228639494019113, 0.15736064447635645
    26
    EP 26 simulation finished, start learning
    4.3
    EP 26 learning finished: 0.21352417657148948, 0.14462990209773294
    27
    EP 27 simulation finished, start learning
    4.11
    EP 27 learning finished: 0.14744006045243196, 0.11747570458825862
    28
    EP 28 simulation finished, start learning
    4.62
    EP 28 learning finished: 0.14012099588176596, 0.10429874915820214
    29
    EP 29 simulation finished, start learning
    4.53
    EP 29 learning finished: 0.14394901283075967, 0.11262313228299654
    30
    EP 30 simulation finished, start learning
    4.12
    EP 30 learning finished: 0.16763865551578405, 0.11550477692756278
    31
    EP 31 simulation finished, start learning
    4.34
    EP 31 learning finished: 0.2011972376210025, 0.12102953975576784
    32
    EP 32 simulation finished, start learning
    4.35
    EP 32 learning finished: 0.2367990023066504, 0.1314615878598445
    33
    EP 33 simulation finished, start learning
    4.14
    EP 33 learning finished: 0.2938225219223405, 0.14176885196251496
    34
    EP 34 simulation finished, start learning
    4.23
    EP 34 learning finished: 0.31638789848907223, 0.14017648848876232
    35
    EP 35 simulation finished, start learning
    4.12
    EP 35 learning finished: 0.27173149132472574, 0.13184346431470098
    36
    EP 36 simulation finished, start learning
    4.63
    EP 36 learning finished: 0.23317556820136076, 0.12182111725799889
    37
    EP 37 simulation finished, start learning
    4.02
    EP 37 learning finished: 0.18738618365974702, 0.10678574340427449
    38
    EP 38 simulation finished, start learning
    4.41
    EP 38 learning finished: 0.12446644323044254, 0.09523574990905015
    39
    EP 39 simulation finished, start learning
    4.7
    EP 39 learning finished: 0.10286978355395604, 0.08483127290536825
    40
    EP 40 simulation finished, start learning
    4.52
    EP 40 learning finished: 0.1124129338916866, 0.07700211050812762
    41
    EP 41 simulation finished, start learning
    4.71
    EP 41 learning finished: 0.0893013452778817, 0.062488289888361415
    42
    EP 42 simulation finished, start learning
    4.13
    EP 42 learning finished: 0.08758743516724721, 0.06454441585929101
    43
    EP 43 simulation finished, start learning
    4.31
    EP 43 learning finished: 0.08217080685509615, 0.06376816629844602
    44
    EP 44 simulation finished, start learning
    4.42
    EP 44 learning finished: 0.08373719562053594, 0.06372267322718746
    45
    EP 45 simulation finished, start learning
    4.51
    EP 45 learning finished: 0.08205859708432328, 0.06766327467023074
    46
    EP 46 simulation finished, start learning
    4.02
    EP 46 learning finished: 0.0761958854961442, 0.0639798660269939
    47
    EP 47 simulation finished, start learning
    4.6
    EP 47 learning finished: 0.07426978908698345, 0.06219301747716669
    48
    EP 48 simulation finished, start learning
    4.34
    EP 48 learning finished: 0.09772747065343168, 0.0710952441455482
    49
    EP 49 simulation finished, start learning
    4.44
    EP 49 learning finished: 0.13339404610796635, 0.08329502355270553
    50
    EP 50 simulation finished, start learning
    3.91
    EP 50 learning finished: 0.1204286611024088, 0.08364994808970483
    51
    EP 51 simulation finished, start learning
    4.22
    EP 51 learning finished: 0.1710374484545767, 0.0946057131039882
    52
    EP 52 simulation finished, start learning
    4.05
    EP 52 learning finished: 0.1472835982046412, 0.09118179068161177
    53
    EP 53 simulation finished, start learning
    4.53
    EP 53 learning finished: 0.12571605314133905, 0.07704712819335609
    54
    EP 54 simulation finished, start learning
    4.7
    EP 54 learning finished: 0.11469795933709293, 0.06534288329428446
    55
    EP 55 simulation finished, start learning
    3.76
    EP 55 learning finished: 0.12691909669747403, 0.07098991951901024
    56
    EP 56 simulation finished, start learning
    4.53
    EP 56 learning finished: 0.10661696551563438, 0.06725795230764753
    57
    EP 57 simulation finished, start learning
    4.41
    EP 57 learning finished: 0.08819542158638427, 0.06255577168698483
    58
    EP 58 simulation finished, start learning
    3.93
    EP 58 learning finished: 0.1263650894785151, 0.07431609743597445
    59
    EP 59 simulation finished, start learning
    3.92
    EP 59 learning finished: 0.1092811757877108, 0.06290777591241466
    60
    EP 60 simulation finished, start learning
    4.32
    EP 60 learning finished: 0.09978921355174464, 0.0563745736534725
    61
    EP 61 simulation finished, start learning
    4.15
    EP 61 learning finished: 0.12478829631007926, 0.0674895645422063
    62
    EP 62 simulation finished, start learning
    4.41
    EP 62 learning finished: 0.08731073439176384, 0.06141239024883512
    63
    EP 63 simulation finished, start learning
    4.33
    EP 63 learning finished: 0.09292415694160638, 0.06687147047260744
    64
    EP 64 simulation finished, start learning
    4.32
    EP 64 learning finished: 0.11850349366281065, 0.07002589728440374
    65
    EP 65 simulation finished, start learning
    3.73
    EP 65 learning finished: 0.09167328356666182, 0.06387229915942043
    66
    EP 66 simulation finished, start learning
    3.92
    EP 66 learning finished: 0.09370969595432399, 0.06547324856370966
    67
    EP 67 simulation finished, start learning
    4.62
    EP 67 learning finished: 0.07353835604142844, 0.052727674531412916
    68
    EP 68 simulation finished, start learning
    4.51
    EP 68 learning finished: 0.06651935098280504, 0.045807284220751164
    69
    EP 69 simulation finished, start learning
    4.22
    EP 69 learning finished: 0.0778256238017967, 0.04445462809955222
    70
    EP 70 simulation finished, start learning
    4.2
    EP 70 learning finished: 0.047196062986614606, 0.03666204205958875
    71
    EP 71 simulation finished, start learning
    4.33
    EP 71 learning finished: 0.05238437787800265, 0.042115942903828106
    72
    EP 72 simulation finished, start learning
    4.11
    EP 72 learning finished: 0.05459340778898181, 0.04240602698275442
    73
    EP 73 simulation finished, start learning
    4.62
    EP 73 learning finished: 0.04449007212919298, 0.037350191752802665
    74
    EP 74 simulation finished, start learning
    4.41
    EP 74 learning finished: 0.05616469030802662, 0.04052089271607843
    75
    EP 75 simulation finished, start learning
    4.5
    EP 75 learning finished: 0.05661780234801128, 0.03401165925342475
    76
    EP 76 simulation finished, start learning
    4.42
    EP 76 learning finished: 0.06657241189396501, 0.04261786547414026
    77
    EP 77 simulation finished, start learning
    4.14
    EP 77 learning finished: 0.07590044417073116, 0.050808843178911246
    78
    EP 78 simulation finished, start learning
    4.03
    EP 78 learning finished: 0.07108217014070739, 0.05189280017975436
    79
    EP 79 simulation finished, start learning
    4.43
    EP 79 learning finished: 0.08226004566533263, 0.05557112172737474
    80
    EP 80 simulation finished, start learning
    4.31
    EP 80 learning finished: 0.07551107063166995, 0.049890568746232176
    81
    EP 81 simulation finished, start learning
    4.24
    EP 81 learning finished: 0.07414513088856367, 0.045049600833237946
    82
    EP 82 simulation finished, start learning
    4.02
    EP 82 learning finished: 0.08367388607892323, 0.04529635367656374
    83
    EP 83 simulation finished, start learning
    4.23
    EP 83 learning finished: 0.06675198214080069, 0.04480401077490548
    84
    EP 84 simulation finished, start learning
    4.71
    EP 84 learning finished: 0.07616326935805873, 0.041019309987472496
    85
    EP 85 simulation finished, start learning
    4.62
    EP 85 learning finished: 0.06578508602748201, 0.03874229016636536
    86
    EP 86 simulation finished, start learning
    4.11
    EP 86 learning finished: 0.054682174798085015, 0.03431911959683968
    87
    EP 87 simulation finished, start learning
    4.22
    EP 87 learning finished: 0.043513249470766624, 0.030705923922750362
    88
    EP 88 simulation finished, start learning
    4.32
    EP 88 learning finished: 0.048206664119991416, 0.030723533245180807
    89
    EP 89 simulation finished, start learning
    4.11
    EP 89 learning finished: 0.04254792024085319, 0.03274408912646365
    90
    EP 90 simulation finished, start learning
    4.23
    EP 90 learning finished: 0.05695835252033277, 0.040939858470321035
    91
    EP 91 simulation finished, start learning
    4.34
    EP 91 learning finished: 0.07825610872453836, 0.04466058178319502
    92
    EP 92 simulation finished, start learning
    4.52
    EP 92 learning finished: 0.07674990522045633, 0.04318095055375526
    93
    EP 93 simulation finished, start learning
    4.26
    EP 93 learning finished: 0.09096252144045636, 0.045272786970867195
    94
    EP 94 simulation finished, start learning
    4.31
    EP 94 learning finished: 0.07720950531110304, 0.03826046828024291
    95
    EP 95 simulation finished, start learning
    4.52
    EP 95 learning finished: 0.055255659128398804, 0.033096571022261045
    96
    EP 96 simulation finished, start learning
    3.93
    EP 96 learning finished: 0.05828192558156431, 0.035466515164213654
    97
    EP 97 simulation finished, start learning
    4.25
    EP 97 learning finished: 0.05364945027787016, 0.032230145466742854
    98
    EP 98 simulation finished, start learning
    4.91
    EP 98 learning finished: 0.04424833052260624, 0.030175688357499435
    99
    EP 99 simulation finished, start learning
    4.31
    EP 99 learning finished: 0.03991785521754565, 0.029387789445065593
    100
    EP 100 simulation finished, start learning
    4.31
    EP 100 learning finished: 0.03471349732905793, 0.02597115907403591
    101
    EP 101 simulation finished, start learning
    4.6
    EP 101 learning finished: 0.03426585985100772, 0.020386340415869952
    102
    EP 102 simulation finished, start learning
    4.14
    EP 102 learning finished: 0.030661215517163498, 0.02285635206309659
    103
    EP 103 simulation finished, start learning
    4.44
    EP 103 learning finished: 0.035300244073155795, 0.02502743683233118
    104
    EP 104 simulation finished, start learning
    3.93
    EP 104 learning finished: 0.050672988844297606, 0.02932741357616327
    105
    EP 105 simulation finished, start learning
    4.2
    EP 105 learning finished: 0.05010797046242743, 0.03387214467102299
    106
    EP 106 simulation finished, start learning
    4.53
    EP 106 learning finished: 0.04603865530667406, 0.033514088582694924
    107
    EP 107 simulation finished, start learning
    3.74
    EP 107 learning finished: 0.047073968351234896, 0.034767525718254554
    108
    EP 108 simulation finished, start learning
    4.13
    EP 108 learning finished: 0.04398863981790413, 0.03214655583066328
    109
    EP 109 simulation finished, start learning
    4.1
    EP 109 learning finished: 0.05518849744190035, 0.02950596371152751
    110
    EP 110 simulation finished, start learning
    4.55
    EP 110 learning finished: 0.052009465552103507, 0.03089600004900639
    111
    EP 111 simulation finished, start learning
    4.31
    EP 111 learning finished: 0.04711224169669137, 0.026821163124086667
    112
    EP 112 simulation finished, start learning
    4.44
    EP 112 learning finished: 0.04053861506891465, 0.02703052820398888
    113
    EP 113 simulation finished, start learning
    4.53
    EP 113 learning finished: 0.04637813011793668, 0.027122526862989698
    114
    EP 114 simulation finished, start learning
    4.5
    EP 114 learning finished: 0.032236996489501735, 0.021480606562272204
    115
    EP 115 simulation finished, start learning
    4.13
    EP 115 learning finished: 0.03800801344292637, 0.024840477925164233
    116
    EP 116 simulation finished, start learning
    4.62
    EP 116 learning finished: 0.035640937784943816, 0.023815853574454593
    117
    EP 117 simulation finished, start learning
    3.72
    EP 117 learning finished: 0.04618678167292245, 0.028247205991622337
    118
    EP 118 simulation finished, start learning
    4.31
    EP 118 learning finished: 0.05438712632939513, 0.03440708921658923
    119
    EP 119 simulation finished, start learning
    4.32
    EP 119 learning finished: 0.05155685530669629, 0.03281853048153641
    120
    EP 120 simulation finished, start learning
    4.41
    EP 120 learning finished: 0.07134442794681028, 0.034864394513417864
    121
    EP 121 simulation finished, start learning
    4.32
    EP 121 learning finished: 0.045543732879133714, 0.029315977013021726
    122
    EP 122 simulation finished, start learning
    4.42
    EP 122 learning finished: 0.09062101523825028, 0.036307976030333755
    123
    EP 123 simulation finished, start learning
    4.52
    EP 123 learning finished: 0.0899878126897864, 0.034331405296888974
    124
    EP 124 simulation finished, start learning
    5.0
    EP 124 learning finished: 0.06438942288887795, 0.02825788691812973
    125
    EP 125 simulation finished, start learning
    4.55
    EP 125 learning finished: 0.06406244090947943, 0.031505945442890315
    126
    EP 126 simulation finished, start learning
    4.62
    EP 126 learning finished: 0.027848126781847782, 0.019457806161373863
    127
    EP 127 simulation finished, start learning
    4.82
    EP 127 learning finished: 0.02761793565823467, 0.020253251845143556
    128
    EP 128 simulation finished, start learning
    4.44
    EP 128 learning finished: 0.038498776665200296, 0.025350141356011473
    129
    EP 129 simulation finished, start learning
    4.5
    EP 129 learning finished: 0.025250878449131875, 0.01998685679062895
    130
    EP 130 simulation finished, start learning
    4.23
    EP 130 learning finished: 0.028379345748348328, 0.020884266235423016
    131
    EP 131 simulation finished, start learning
    3.81
    EP 131 learning finished: 0.03106272895433193, 0.0243439350119532
    132
    EP 132 simulation finished, start learning
    4.73
    EP 132 learning finished: 0.029061025568104526, 0.023044615415982978
    133
    EP 133 simulation finished, start learning
    4.53
    EP 133 learning finished: 0.027874606568651927, 0.025051907540571367
    134
    EP 134 simulation finished, start learning
    3.82
    EP 134 learning finished: 0.049795009946372204, 0.028884881378624883
    135
    EP 135 simulation finished, start learning
    4.22
    EP 135 learning finished: 0.0275308890121444, 0.024689713366642902
    136
    EP 136 simulation finished, start learning
    4.34
    EP 136 learning finished: 0.030027640821508328, 0.024519118015289425
    137
    EP 137 simulation finished, start learning
    3.83
    EP 137 learning finished: 0.029759277807386297, 0.025448653041358977
    138
    EP 138 simulation finished, start learning
    3.96
    EP 138 learning finished: 0.040175120352920146, 0.027881237488170285
    139
    EP 139 simulation finished, start learning
    4.04
    EP 139 learning finished: 0.045008903933705824, 0.02928530187468296
    140
    EP 140 simulation finished, start learning
    4.34
    EP 140 learning finished: 0.04448650880727456, 0.030222212270622606
    141
    EP 141 simulation finished, start learning
    4.53
    EP 141 learning finished: 0.04393941436586957, 0.028822618542883482
    142
    EP 142 simulation finished, start learning
    4.41
    EP 142 learning finished: 0.03925379204761015, 0.021941946434642052
    143
    EP 143 simulation finished, start learning
    4.01
    EP 143 learning finished: 0.03983524425505536, 0.02125311228323845
    144
    EP 144 simulation finished, start learning
    4.41
    EP 144 learning finished: 0.038914821069514716, 0.026529387827996704
    145
    EP 145 simulation finished, start learning
    4.62
    EP 145 learning finished: 0.03879322111509609, 0.025948240213740686
    146
    EP 146 simulation finished, start learning
    4.42
    EP 146 learning finished: 0.038671625302413304, 0.025207939331282365
    147
    EP 147 simulation finished, start learning
    4.52
    EP 147 learning finished: 0.03856506139587168, 0.02496160085688892
    148
    EP 148 simulation finished, start learning
    4.71
    EP 148 learning finished: 0.027372985933187266, 0.016015482938049885
    149
    EP 149 simulation finished, start learning
    4.23
    EP 149 learning finished: 0.027161456992555344, 0.016988377225101676
    150
    EP 150 simulation finished, start learning
    4.13
    EP 150 learning finished: 0.029313419487728254, 0.01776704611978097
    151
    EP 151 simulation finished, start learning
    4.14
    EP 151 learning finished: 0.028063901627305654, 0.017976808341003903
    152
    EP 152 simulation finished, start learning
    4.53
    EP 152 learning finished: 0.030577505301856434, 0.02308962320313886
    153
    EP 153 simulation finished, start learning
    4.32
    EP 153 learning finished: 0.028047146797000266, 0.02222544994507854
    154
    EP 154 simulation finished, start learning
    4.17
    EP 154 learning finished: 0.03653130779138536, 0.025423113520561813
    155
    EP 155 simulation finished, start learning
    4.44
    EP 155 learning finished: 0.04387533458846704, 0.026061498635592706
    156
    EP 156 simulation finished, start learning
    4.23
    EP 156 learning finished: 0.043310376487518454, 0.022151372791037478
    157
    EP 157 simulation finished, start learning
    4.61
    EP 157 learning finished: 0.040579356052383275, 0.020782525530622802
    158
    EP 158 simulation finished, start learning
    4.22
    EP 158 learning finished: 0.02924183005770331, 0.017854098877987685
    159
    EP 159 simulation finished, start learning
    4.22
    EP 159 learning finished: 0.022604555875062715, 0.01944785253497668
    160
    EP 160 simulation finished, start learning
    4.22
    EP 160 learning finished: 0.030466271186050164, 0.020015030274335228
    161
    EP 161 simulation finished, start learning
    4.61
    EP 161 learning finished: 0.030090886961481544, 0.019907386940594424
    162
    EP 162 simulation finished, start learning
    4.55
    EP 162 learning finished: 0.024773703066991004, 0.020979284103246936
    163
    EP 163 simulation finished, start learning
    4.11
    EP 163 learning finished: 0.027059062679850143, 0.01673626217867093
    164
    EP 164 simulation finished, start learning
    4.14
    EP 164 learning finished: 0.030258155948484244, 0.017828537709335162
    165
    EP 165 simulation finished, start learning
    4.52
    EP 165 learning finished: 0.03206800298069368, 0.018306374065093205
    166
    EP 166 simulation finished, start learning
    4.53
    EP 166 learning finished: 0.027675693272467328, 0.017082726161604
    167
    EP 167 simulation finished, start learning
    4.72
    EP 167 learning finished: 0.02944982556930853, 0.0207051947724608
    168
    EP 168 simulation finished, start learning
    4.32
    EP 168 learning finished: 0.024349002643006656, 0.018159635268591618
    169
    EP 169 simulation finished, start learning
    4.22
    EP 169 learning finished: 0.025050256631541146, 0.019627142579545048
    170
    EP 170 simulation finished, start learning
    4.14
    EP 170 learning finished: 0.02685739179899116, 0.019457672479300452
    171
    EP 171 simulation finished, start learning
    4.41
    EP 171 learning finished: 0.026656713229474605, 0.018860273259935074
    172
    EP 172 simulation finished, start learning
    4.8
    EP 172 learning finished: 0.029388335691368894, 0.018330578762871436
    173
    EP 173 simulation finished, start learning
    4.42
    EP 173 learning finished: 0.04338097656722084, 0.020763900456911344
    174
    EP 174 simulation finished, start learning
    4.23
    EP 174 learning finished: 0.04324350201357596, 0.020041589119200804
    175
    EP 175 simulation finished, start learning
    4.42
    EP 175 learning finished: 0.04807276723172116, 0.020807345641947528
    176
    EP 176 simulation finished, start learning
    3.93
    EP 176 learning finished: 0.047928824272688075, 0.022096300122839807
    177
    EP 177 simulation finished, start learning
    4.62
    EP 177 learning finished: 0.04776691819769319, 0.021806253390414494
    178
    EP 178 simulation finished, start learning
    4.73
    EP 178 learning finished: 0.04761965862365081, 0.021192204041485436
    179
    EP 179 simulation finished, start learning
    4.53
    EP 179 learning finished: 0.023761261672174072, 0.018567308737908423
    180
    EP 180 simulation finished, start learning
    4.23
    EP 180 learning finished: 0.023731805275881257, 0.01811626488235074
    181
    EP 181 simulation finished, start learning
    4.32
    EP 181 learning finished: 0.02031085548821565, 0.014425651544000218
    182
    EP 182 simulation finished, start learning
    4.2
    EP 182 learning finished: 0.023707454383764315, 0.017452308700061536
    183
    EP 183 simulation finished, start learning
    3.97
    EP 183 learning finished: 0.02368533839589304, 0.019031419597473825
    184
    EP 184 simulation finished, start learning
    4.52
    EP 184 learning finished: 0.0263768194533327, 0.019256457014673007
    185
    EP 185 simulation finished, start learning
    4.23
    EP 185 learning finished: 0.0279879826502345, 0.01973641221530868
    186
    EP 186 simulation finished, start learning
    4.33
    EP 186 learning finished: 0.033091333267954326, 0.02075853432077235
    187
    EP 187 simulation finished, start learning
    4.53
    EP 187 learning finished: 0.047069097827542095, 0.0219673372378922
    188
    EP 188 simulation finished, start learning
    4.4
    EP 188 learning finished: 0.04692070097314094, 0.021056413349859266
    189
    EP 189 simulation finished, start learning
    4.71
    EP 189 learning finished: 0.04677245081263237, 0.020487974484947696
    190
    EP 190 simulation finished, start learning
    4.71
    EP 190 learning finished: 0.023331442128922575, 0.016293040703706
    191
    EP 191 simulation finished, start learning
    4.52
    EP 191 learning finished: 0.02043505229072551, 0.01196138896431916
    192
    EP 192 simulation finished, start learning
    3.84
    EP 192 learning finished: 0.019939199303713906, 0.013469602045035481
    193
    EP 193 simulation finished, start learning
    4.91
    EP 193 learning finished: 0.01982430094744103, 0.012288341196339777
    194
    EP 194 simulation finished, start learning
    4.52
    EP 194 learning finished: 0.016218094095023616, 0.011928523480844907
    195
    EP 195 simulation finished, start learning
    4.51
    EP 195 learning finished: 0.0156246115822114, 0.011545375056188098
    196
    EP 196 simulation finished, start learning
    4.31
    EP 196 learning finished: 0.015310353140570498, 0.009533138926089535
    197
    EP 197 simulation finished, start learning
    4.31
    EP 197 learning finished: 0.03700245521022616, 0.017958060682625582
    198
    EP 198 simulation finished, start learning
    4.44
    EP 198 learning finished: 0.03691066863033399, 0.01877122335133959
    199
    EP 199 simulation finished, start learning
    4.01
    EP 199 learning finished: 0.05513988939439329, 0.022543066657358796
    200
    EP 200 simulation finished, start learning
    4.23
    EP 200 learning finished: 0.054879852408364016, 0.023501378879168477
    201
    EP 201 simulation finished, start learning
    4.71
    EP 201 learning finished: 0.01823444488971404, 0.01613131990185816
    202
    EP 202 simulation finished, start learning
    3.8
    EP 202 learning finished: 0.02329163784219901, 0.01609276290045673
    203
    EP 203 simulation finished, start learning
    4.43
    EP 203 learning finished: 0.0199475566794014, 0.013045616262507664
    204
    EP 204 simulation finished, start learning
    4.52
    EP 204 learning finished: 0.024177486334505005, 0.013019850592730956
    205
    EP 205 simulation finished, start learning
    4.32
    EP 205 learning finished: 0.025958366846332126, 0.013760915940011978
    206
    EP 206 simulation finished, start learning
    4.42
    EP 206 learning finished: 0.023031344544730104, 0.01663602898638968
    207
    EP 207 simulation finished, start learning
    3.14
    EP 207 learning finished: 0.02746989505083519, 0.01923024561671542
    208
    EP 208 simulation finished, start learning
    4.04
    EP 208 learning finished: 0.02484010922002966, 0.01954416068318861
    209
    EP 209 simulation finished, start learning
    4.8
    EP 209 learning finished: 0.022941672636935806, 0.017445735225452452
    210
    EP 210 simulation finished, start learning
    4.32
    EP 210 learning finished: 0.021373638749313528, 0.013693074268511888
    211
    EP 211 simulation finished, start learning
    4.41
    EP 211 learning finished: 0.022929674271727762, 0.013998959662716063
    212
    EP 212 simulation finished, start learning
    4.41
    EP 212 learning finished: 0.022899056195488043, 0.012337622879205293
    213
    EP 213 simulation finished, start learning
    4.24
    EP 213 learning finished: 0.04569274915664856, 0.01802795475542999
    214
    EP 214 simulation finished, start learning
    4.11
    EP 214 learning finished: 0.06349351396534608, 0.02106311321673271
    215
    EP 215 simulation finished, start learning
    4.14
    EP 215 learning finished: 0.04058172336661592, 0.01864345687252733
    216
    EP 216 simulation finished, start learning
    4.34
    EP 216 learning finished: 0.04043939603757174, 0.019831966051211996
    217
    EP 217 simulation finished, start learning
    4.12
    EP 217 learning finished: 0.019021784804955644, 0.015489939953575652
    218
    EP 218 simulation finished, start learning
    3.45
    EP 218 learning finished: 0.02330680863019552, 0.018067538733820204
    219
    EP 219 simulation finished, start learning
    4.53
    EP 219 learning finished: 0.02260745300519318, 0.017385290528391845
    220
    EP 220 simulation finished, start learning
    4.32
    EP 220 learning finished: 0.044888584860125036, 0.020243393442350783
    221
    EP 221 simulation finished, start learning
    4.13
    EP 221 learning finished: 0.04473743443851674, 0.021246913556707863
    222
    EP 222 simulation finished, start learning
    4.32
    EP 222 learning finished: 0.022318939807171034, 0.016295415956994574
    223
    EP 223 simulation finished, start learning
    4.71
    EP 223 learning finished: 0.044545258812098254, 0.018849157597222128
    224
    EP 224 simulation finished, start learning
    4.43
    EP 224 learning finished: 0.02221978995939189, 0.01540668613129951
    225
    EP 225 simulation finished, start learning
    4.62
    EP 225 learning finished: 0.022191085605077987, 0.013859802058607151
    226
    EP 226 simulation finished, start learning
    4.41
    EP 226 learning finished: 0.022160201913839828, 0.012578962295522503
    227
    EP 227 simulation finished, start learning
    4.23
    EP 227 learning finished: 0.019764233796276898, 0.014120933692738369
    228
    EP 228 simulation finished, start learning
    4.02
    EP 228 learning finished: 0.018197440734311954, 0.014790499033982663
    229
    EP 229 simulation finished, start learning
    4.02
    EP 229 learning finished: 0.018101155972324534, 0.015088848755198706
    230
    EP 230 simulation finished, start learning
    4.41
    EP 230 learning finished: 0.023165448610466655, 0.015866061733423347
    231
    EP 231 simulation finished, start learning
    4.41
    EP 231 learning finished: 0.018299278836941824, 0.010834702652759395
    232
    EP 232 simulation finished, start learning
    4.32
    EP 232 learning finished: 0.022042647795846193, 0.013426913359560663
    233
    EP 233 simulation finished, start learning
    4.32
    EP 233 learning finished: 0.02201962120788359, 0.013143524465868417
    234
    EP 234 simulation finished, start learning
    4.52
    EP 234 learning finished: 0.021993021813994318, 0.012557715768380126
    235
    EP 235 simulation finished, start learning
    4.8
    EP 235 learning finished: 0.021955066554600222, 0.011824035468257795
    236
    EP 236 simulation finished, start learning
    4.02
    EP 236 learning finished: 0.02192725438875165, 0.012049708516709204
    237
    EP 237 simulation finished, start learning
    4.5
    EP 237 learning finished: 0.021903706356438235, 0.011290419532262952
    238
    EP 238 simulation finished, start learning
    4.62
    EP 238 learning finished: 0.021871617785558684, 0.011300637999274166
    239
    EP 239 simulation finished, start learning
    4.32
    EP 239 learning finished: 0.021841699453178265, 0.012561959026505734
    240
    EP 240 simulation finished, start learning
    4.44
    EP 240 learning finished: 0.01719702485564767, 0.01257127644175308
    241
    EP 241 simulation finished, start learning
    4.71
    EP 241 learning finished: 0.017183283711514008, 0.012558218358960241
    242
    EP 242 simulation finished, start learning
    4.05
    EP 242 learning finished: 0.02279616493162706, 0.01472815566965154
    243
    EP 243 simulation finished, start learning
    4.33
    EP 243 learning finished: 0.03886415079231931, 0.0184116819141229
    244
    EP 244 simulation finished, start learning
    4.61
    EP 244 learning finished: 0.023345510431425964, 0.014725110169675778
    245
    EP 245 simulation finished, start learning
    3.83
    EP 245 learning finished: 0.025832531839434125, 0.016114819966682365
    246
    EP 246 simulation finished, start learning
    4.34
    EP 246 learning finished: 0.021620683740440683, 0.014695595950250606
    247
    EP 247 simulation finished, start learning
    3.36
    EP 247 learning finished: 0.023280059808644893, 0.013049780019123011
    248
    EP 248 simulation finished, start learning
    4.1
    EP 248 learning finished: 0.017590512081582688, 0.012261281815857691
    249
    EP 249 simulation finished, start learning
    4.61
    EP 249 learning finished: 0.017971673061282445, 0.011603627928624386
    250
    EP 250 simulation finished, start learning
    4.72
    EP 250 learning finished: 0.016229604304704992, 0.010616245938492122
    251
    EP 251 simulation finished, start learning
    4.62
    EP 251 learning finished: 0.011673269657137553, 0.008066272819904205
    252
    EP 252 simulation finished, start learning
    4.22
    EP 252 learning finished: 0.01377975809343468, 0.009385026409513367
    253
    EP 253 simulation finished, start learning
    4.2
    EP 253 learning finished: 0.012439967885377712, 0.00874904874073934
    254
    EP 254 simulation finished, start learning
    4.53
    EP 254 learning finished: 0.017107513668815244, 0.012919849122755641
    255
    EP 255 simulation finished, start learning
    4.13
    EP 255 learning finished: 0.01708635473118747, 0.013630292308957218
    256
    EP 256 simulation finished, start learning
    4.81
    EP 256 learning finished: 0.017068674042791443, 0.012070373689208047
    257
    EP 257 simulation finished, start learning
    4.11
    EP 257 learning finished: 0.01704513454327028, 0.013060033179190947
    258
    EP 258 simulation finished, start learning
    4.62
    EP 258 learning finished: 0.013445800533723684, 0.00881364740867241
    259
    EP 259 simulation finished, start learning
    4.14
    EP 259 learning finished: 0.021564394241117668, 0.012662051920070638
    260
    EP 260 simulation finished, start learning
    4.73
    EP 260 learning finished: 0.021540944738475787, 0.01395142774785422
    261
    EP 261 simulation finished, start learning
    3.53
    EP 261 learning finished: 0.03846222259460497, 0.01804293437736515
    262
    EP 262 simulation finished, start learning
    4.5
    EP 262 learning finished: 0.038350667525776316, 0.01811010245733513
    263
    EP 263 simulation finished, start learning
    4.03
    EP 263 learning finished: 0.022694985222308617, 0.014333214719667827
    264
    EP 264 simulation finished, start learning
    4.14
    EP 264 learning finished: 0.03818876680652217, 0.01786882422868441
    265
    EP 265 simulation finished, start learning
    4.41
    EP 265 learning finished: 0.02129749750098675, 0.01340217182477795
    266
    EP 266 simulation finished, start learning
    4.42
    EP 266 learning finished: 0.04250360358344274, 0.01687790360344395
    267
    EP 267 simulation finished, start learning
    4.23
    EP 267 learning finished: 0.059031932410896104, 0.02034448429589375
    268
    EP 268 simulation finished, start learning
    3.93
    EP 268 learning finished: 0.05430186351791555, 0.020347319903607126
    269
    EP 269 simulation finished, start learning
    4.35
    EP 269 learning finished: 0.05406102600782958, 0.02085555046892414
    270
    EP 270 simulation finished, start learning
    4.23
    EP 270 learning finished: 0.049387955253076044, 0.020723401094326317
    271
    EP 271 simulation finished, start learning
    4.31
    EP 271 learning finished: 0.03281461682137632, 0.01627087303241363
    272
    EP 272 simulation finished, start learning
    4.11
    EP 272 learning finished: 0.01711605864765886, 0.012500419515939498
    273
    EP 273 simulation finished, start learning
    4.71
    EP 273 learning finished: 0.01670681422089615, 0.011726283528746606
    274
    EP 274 simulation finished, start learning
    4.21
    EP 274 learning finished: 0.02072627434210972, 0.011570335824881539
    275
    EP 275 simulation finished, start learning
    4.13
    EP 275 learning finished: 0.03697646686996592, 0.015215046029294873
    276
    EP 276 simulation finished, start learning
    4.41
    EP 276 learning finished: 0.03686241882714292, 0.014228760108796775
    277
    EP 277 simulation finished, start learning
    3.71
    EP 277 learning finished: 0.08509660853467871, 0.023000293407036132
    278
    EP 278 simulation finished, start learning
    4.11
    EP 278 learning finished: 0.08008298452398499, 0.022486964202034738
    279
    EP 279 simulation finished, start learning
    4.54
    EP 279 learning finished: 0.08378241093527805, 0.02276322449919969
    280
    EP 280 simulation finished, start learning
    3.81
    EP 280 learning finished: 0.103093677821267, 0.026375844222726503
    281
    EP 281 simulation finished, start learning
    4.71
    EP 281 learning finished: 0.05546521522335013, 0.016965363144125927
    282
    EP 282 simulation finished, start learning
    4.32
    EP 282 learning finished: 0.059479946326455035, 0.017504207539371208
    283
    EP 283 simulation finished, start learning
    4.23
    EP 283 learning finished: 0.05920890381494459, 0.017400057339895508
    284
    EP 284 simulation finished, start learning
    3.85
    EP 284 learning finished: 0.03932025334972522, 0.015271181562881134
    285
    EP 285 simulation finished, start learning
    4.14
    EP 285 learning finished: 0.058757947850555325, 0.019716758060393264
    286
    EP 286 simulation finished, start learning
    4.82
    EP 286 learning finished: 0.03902439842433836, 0.0166858226574161
    287
    EP 287 simulation finished, start learning
    4.61
    EP 287 learning finished: 0.019472476715748144, 0.013201860679200669
    288
    EP 288 simulation finished, start learning
    4.23
    EP 288 learning finished: 0.019446750353726117, 0.012367896893971775
    289
    EP 289 simulation finished, start learning
    4.44
    EP 289 learning finished: 0.011450659809923991, 0.008897492925183789
    290
    EP 290 simulation finished, start learning
    4.02
    EP 290 learning finished: 0.01942936834808151, 0.01200428711602027
    291
    EP 291 simulation finished, start learning
    4.61
    EP 291 learning finished: 0.019402771585760092, 0.011964294801657616
    292
    EP 292 simulation finished, start learning
    4.61
    EP 292 learning finished: 0.019378799449889073, 0.011040551937776163
    293
    EP 293 simulation finished, start learning
    3.93
    EP 293 learning finished: 0.019358814206189123, 0.011737355060907007
    294
    EP 294 simulation finished, start learning
    3.84
    EP 294 learning finished: 0.01465108080473243, 0.009308175889512901
    295
    EP 295 simulation finished, start learning
    4.31
    EP 295 learning finished: 0.01457713340030864, 0.009464118865930496
    296
    EP 296 simulation finished, start learning
    4.34
    EP 296 learning finished: 0.017652540162863772, 0.010416663042402519
    297
    EP 297 simulation finished, start learning
    4.23
    EP 297 learning finished: 0.013619095291339534, 0.009520336996851361
    298
    EP 298 simulation finished, start learning
    4.52
    EP 298 learning finished: 0.010533701453660171, 0.00836519036445793
    299
    EP 299 simulation finished, start learning
    4.71
    EP 299 learning finished: 0.010392370348256463, 0.007466826047895836
    300
    EP 300 simulation finished, start learning
    4.51
    EP 300 learning finished: 0.009654921836997232, 0.006571614676741906
    301
    EP 301 simulation finished, start learning
    4.23
    EP 301 learning finished: 0.01938187979445516, 0.009774884702415795
    302
    EP 302 simulation finished, start learning
    4.22
    EP 302 learning finished: 0.01935651039868702, 0.009963316386199258
    303
    EP 303 simulation finished, start learning
    4.6
    EP 303 learning finished: 0.03863712198597469, 0.012920976276553494
    304
    EP 304 simulation finished, start learning
    4.41
    EP 304 learning finished: 0.038525176657124405, 0.012857905125366406
    305
    EP 305 simulation finished, start learning
    4.52
    EP 305 learning finished: 0.019224017071543642, 0.009848273170252497
    306
    EP 306 simulation finished, start learning
    4.53
    EP 306 learning finished: 0.019197516444567242, 0.009967616810888202
    307
    EP 307 simulation finished, start learning
    4.12
    EP 307 learning finished: 0.014999674539026842, 0.010418829651036058
    308
    EP 308 simulation finished, start learning
    4.5
    EP 308 learning finished: 0.06394564818884874, 0.018668859244616698
    309
    EP 309 simulation finished, start learning
    4.42
    EP 309 learning finished: 0.06355856878051114, 0.018054337905837176
    310
    EP 310 simulation finished, start learning
    4.34
    EP 310 learning finished: 0.06317494780979516, 0.018257540373420866
    311
    EP 311 simulation finished, start learning
    4.4
    EP 311 learning finished: 0.04815951967579224, 0.015338507541526289
    312
    EP 312 simulation finished, start learning
    4.41
    EP 312 learning finished: 0.010274706502045206, 0.007188792720241348
    313
    EP 313 simulation finished, start learning
    4.71
    EP 313 learning finished: 0.018763067731331518, 0.009770310044008449
    314
    EP 314 simulation finished, start learning
    4.32
    EP 314 learning finished: 0.018734597655502228, 0.00978137646340708
    315
    EP 315 simulation finished, start learning
    3.84
    EP 315 learning finished: 0.018716356835387415, 0.010540869407966724
    316
    EP 316 simulation finished, start learning
    4.04
    EP 316 learning finished: 0.018690596740224663, 0.012162054167800918
    317
    EP 317 simulation finished, start learning
    4.22
    EP 317 learning finished: 0.015590154965714476, 0.009608574347137022
    318
    EP 318 simulation finished, start learning
    4.35
    EP 318 learning finished: 0.018443132086922276, 0.009761274063241862
    319
    EP 319 simulation finished, start learning
    4.53
    EP 319 learning finished: 0.03323369698212986, 0.014787981242940637
    320
    EP 320 simulation finished, start learning
    4.53
    EP 320 learning finished: 0.033144831825769216, 0.014084437793874471
    321
    EP 321 simulation finished, start learning
    4.33
    EP 321 learning finished: 0.05157578288339337, 0.01723338019709845
    322
    EP 322 simulation finished, start learning
    4.61
    EP 322 learning finished: 0.051345091375665675, 0.016026512859936542
    323
    EP 323 simulation finished, start learning
    4.6
    EP 323 learning finished: 0.01841838162136966, 0.009606551008732941
    324
    EP 324 simulation finished, start learning
    4.22
    EP 324 learning finished: 0.01839503408326948, 0.009004369568926288
    325
    EP 325 simulation finished, start learning
    4.01
    EP 325 learning finished: 0.011655739507169471, 0.00623264639974271
    326
    EP 326 simulation finished, start learning
    4.61
    EP 326 learning finished: 0.014857253992353492, 0.006762734384680631
    327
    EP 327 simulation finished, start learning
    4.15
    EP 327 learning finished: 0.01476811901595454, 0.00824217420333421
    328
    EP 328 simulation finished, start learning
    3.93
    EP 328 learning finished: 0.015255260566519269, 0.008597586703142923
    329
    EP 329 simulation finished, start learning
    4.32
    EP 329 learning finished: 0.014009904711549837, 0.008654107754965187
    330
    EP 330 simulation finished, start learning
    4.32
    EP 330 learning finished: 0.018405073500769475, 0.011539572485109654
    331
    EP 331 simulation finished, start learning
    4.05
    EP 331 learning finished: 0.032690864971947065, 0.01417245215735837
    332
    EP 332 simulation finished, start learning
    4.82
    EP 332 learning finished: 0.0326102958909118, 0.013400576926740607
    333
    EP 333 simulation finished, start learning
    4.51
    EP 333 learning finished: 0.03251214619691467, 0.012490107349278171
    334
    EP 334 simulation finished, start learning
    4.32
    EP 334 learning finished: 0.014218589449537689, 0.010003553061521231
    335
    EP 335 simulation finished, start learning
    3.65
    EP 335 learning finished: 0.012720434937044445, 0.007566312311987862
    336
    EP 336 simulation finished, start learning
    4.42
    EP 336 learning finished: 0.014209300413689618, 0.010373078230388208
    337
    EP 337 simulation finished, start learning
    4.02
    EP 337 learning finished: 0.014438179312293187, 0.01078029798504924
    338
    EP 338 simulation finished, start learning
    4.71
    EP 338 learning finished: 0.03234316496489953, 0.012734456181379509
    339
    EP 339 simulation finished, start learning
    4.43
    EP 339 learning finished: 0.03224705490682567, 0.012303895829478032
    340
    EP 340 simulation finished, start learning
    4.35
    EP 340 learning finished: 0.01808505911514402, 0.010916800640805727
    341
    EP 341 simulation finished, start learning
    4.43
    EP 341 learning finished: 0.018062781585875598, 0.011488325177608042
    342
    EP 342 simulation finished, start learning
    4.71
    EP 342 learning finished: 0.014166633945386309, 0.008961002932936188
    343
    EP 343 simulation finished, start learning
    4.62
    EP 343 learning finished: 0.01804970340895551, 0.011055083089357728
    344
    EP 344 simulation finished, start learning
    4.61
    EP 344 learning finished: 0.018024895083339487, 0.009991551703492382
    345
    EP 345 simulation finished, start learning
    4.34
    EP 345 learning finished: 0.01800628158442308, 0.00969375943230529
    346
    EP 346 simulation finished, start learning
    3.93
    EP 346 learning finished: 0.017983559804260807, 0.01070728837814213
    347
    EP 347 simulation finished, start learning
    4.51
    EP 347 learning finished: 0.012216390185848525, 0.0075889835048315035
    348
    EP 348 simulation finished, start learning
    4.63
    EP 348 learning finished: 0.017961694934466976, 0.010433278513755676
    349
    EP 349 simulation finished, start learning
    4.51
    EP 349 learning finished: 0.035853722981737945, 0.012557584458937399
    350
    EP 350 simulation finished, start learning
    4.32
    EP 350 learning finished: 0.05359292658810233, 0.014407764546623033
    351
    EP 351 simulation finished, start learning
    3.6
    EP 351 learning finished: 0.08092171579182406, 0.01962860368796863
    352
    EP 352 simulation finished, start learning
    4.34
    EP 352 learning finished: 0.06273207242777445, 0.017028943766087323
    353
    EP 353 simulation finished, start learning
    4.22
    EP 353 learning finished: 0.044858932328413204, 0.014971407250910692
    354
    EP 354 simulation finished, start learning
    4.43
    EP 354 learning finished: 0.027199982892021524, 0.012689370008753068
    355
    EP 355 simulation finished, start learning
    4.41
    EP 355 learning finished: 0.011876389068241622, 0.00744760354527448
    356
    EP 356 simulation finished, start learning
    4.6
    EP 356 learning finished: 0.017481545267884258, 0.009249631395896074
    357
    EP 357 simulation finished, start learning
    4.05
    EP 357 learning finished: 0.03489213616913389, 0.012367958005455648
    358
    EP 358 simulation finished, start learning
    4.32
    EP 358 learning finished: 0.03479452755486889, 0.01165797256992048
    359
    EP 359 simulation finished, start learning
    4.8
    EP 359 learning finished: 0.034696630962457675, 0.011733086660013234
    360
    EP 360 simulation finished, start learning
    4.33
    EP 360 learning finished: 0.03460640678903104, 0.012506462158823931
    361
    EP 361 simulation finished, start learning
    4.32
    EP 361 learning finished: 0.017266393721470763, 0.008923887898974433
    362
    EP 362 simulation finished, start learning
    4.51
    EP 362 learning finished: 0.030593916405354804, 0.011048882679526528
    363
    EP 363 simulation finished, start learning
    4.23
    EP 363 learning finished: 0.030511098461122543, 0.011434141753386603
    364
    EP 364 simulation finished, start learning
    4.32
    EP 364 learning finished: 0.013291816246351118, 0.008897107339863672
    365
    EP 365 simulation finished, start learning
    4.23
    EP 365 learning finished: 0.01474738139348375, 0.009546429605764573
    366
    EP 366 simulation finished, start learning
    4.72
    EP 366 learning finished: 0.017122869069467583, 0.009994236156609068
    367
    EP 367 simulation finished, start learning
    4.11
    EP 367 learning finished: 0.017102279611517446, 0.010139756097132727
    368
    EP 368 simulation finished, start learning
    4.62
    EP 368 learning finished: 0.03413584847231377, 0.012269022495160561
    369
    EP 369 simulation finished, start learning
    4.32
    EP 369 learning finished: 0.03404151602469341, 0.011541703514725535
    370
    EP 370 simulation finished, start learning
    4.14
    EP 370 learning finished: 0.016989365933618217, 0.009924061192418066
    371
    EP 371 simulation finished, start learning
    4.5
    EP 371 learning finished: 0.030074810894593185, 0.01183625028814402
    372
    EP 372 simulation finished, start learning
    4.23
    EP 372 learning finished: 0.013090779922407592, 0.009395882056393154
    373
    EP 373 simulation finished, start learning
    3.93
    EP 373 learning finished: 0.01509506692389806, 0.01021110005766405
    374
    EP 374 simulation finished, start learning
    4.41
    EP 374 learning finished: 0.013067840628060878, 0.009414483136881794
    375
    EP 375 simulation finished, start learning
    4.23
    EP 375 learning finished: 0.014168289348823748, 0.0078062698447662155
    376
    EP 376 simulation finished, start learning
    4.32
    EP 376 learning finished: 0.0168739220614178, 0.010341041583673771
    377
    EP 377 simulation finished, start learning
    4.21
    EP 377 learning finished: 0.029871829618628, 0.011646417214551027
    378
    EP 378 simulation finished, start learning
    4.34
    EP 378 learning finished: 0.029793102420732964, 0.01181523264055547
    379
    EP 379 simulation finished, start learning
    4.43
    EP 379 learning finished: 0.02971597169576512, 0.011660985999191334
    380
    EP 380 simulation finished, start learning
    3.9
    EP 380 learning finished: 0.01292958977558456, 0.008741019563991278
    381
    EP 381 simulation finished, start learning
    4.64
    EP 381 learning finished: 0.009283574747094292, 0.006756832828141768
    382
    EP 382 simulation finished, start learning
    4.42
    EP 382 learning finished: 0.00826867295509004, 0.006457821818559732
    383
    EP 383 simulation finished, start learning
    4.4
    EP 383 learning finished: 0.00733965221515831, 0.0053933493693136825
    384
    EP 384 simulation finished, start learning
    3.83
    EP 384 learning finished: 0.007749041621046615, 0.005964613812287449
    385
    EP 385 simulation finished, start learning
    4.14
    EP 385 learning finished: 0.008135088973929783, 0.006256236710032208
    386
    EP 386 simulation finished, start learning
    4.82
    EP 386 learning finished: 0.00758345485020584, 0.0058848230792042005
    387
    EP 387 simulation finished, start learning
    4.35
    EP 387 learning finished: 0.011979680165374873, 0.0069464143627797625
    388
    EP 388 simulation finished, start learning
    4.9
    EP 388 learning finished: 0.01011330355522233, 0.005820896991501456
    389
    EP 389 simulation finished, start learning
    4.41
    EP 389 learning finished: 0.009098303113994177, 0.0054845404061723
    390
    EP 390 simulation finished, start learning
    3.83
    EP 390 learning finished: 0.009832962636323828, 0.006154482895369758
    391
    EP 391 simulation finished, start learning
    4.11
    EP 391 learning finished: 0.01297527826082856, 0.007714787919004706
    392
    EP 392 simulation finished, start learning
    4.41
    EP 392 learning finished: 0.012961571797294624, 0.008073434741472004
    393
    EP 393 simulation finished, start learning
    4.41
    EP 393 learning finished: 0.03877904887518824, 0.012797925722154583
    394
    EP 394 simulation finished, start learning
    3.75
    EP 394 learning finished: 0.03861779055585757, 0.01361102614534874
    395
    EP 395 simulation finished, start learning
    4.33
    EP 395 learning finished: 0.0422214668293428, 0.014277959945736404
    396
    EP 396 simulation finished, start learning
    4.62
    EP 396 learning finished: 0.05852266188961508, 0.017063641659773882
    397
    EP 397 simulation finished, start learning
    4.52
    EP 397 learning finished: 0.03287314224101934, 0.01201832984726714
    398
    EP 398 simulation finished, start learning
    4.2
    EP 398 learning finished: 0.04541560076992468, 0.013182995707235653
    399
    EP 399 simulation finished, start learning
    4.22
    EP 399 learning finished: 0.028933050684582184, 0.01077699765084302
    400
    EP 400 simulation finished, start learning
    4.32
    EP 400 learning finished: 0.012580673763743278, 0.008265143330511648
    401
    EP 401 simulation finished, start learning
    4.05
    EP 401 learning finished: 0.012566044655383024, 0.00909737838835826
    402
    EP 402 simulation finished, start learning
    4.44
    EP 402 learning finished: 0.011209699862924083, 0.0069993602099446646
    403
    EP 403 simulation finished, start learning
    4.44
    EP 403 learning finished: 0.012625409185147873, 0.007222675252786341
    404
    EP 404 simulation finished, start learning
    4.52
    EP 404 learning finished: 0.013765373331658282, 0.007515566860706239
    405
    EP 405 simulation finished, start learning
    4.71
    EP 405 learning finished: 0.010806297288397104, 0.006346493299363469
    406
    EP 406 simulation finished, start learning
    3.9
    EP 406 learning finished: 0.010718541706395968, 0.006231354009724299
    407
    EP 407 simulation finished, start learning
    4.63
    EP 407 learning finished: 0.009966714392291465, 0.005561019862717842
    408
    EP 408 simulation finished, start learning
    4.61
    EP 408 learning finished: 0.006950922395375869, 0.004844576109846344
    409
    EP 409 simulation finished, start learning
    4.32
    EP 409 learning finished: 0.00710853635720099, 0.0052810381267995865
    410
    EP 410 simulation finished, start learning
    4.41
    EP 410 learning finished: 0.006841117712625078, 0.004825820298658338
    411
    EP 411 simulation finished, start learning
    4.61
    EP 411 learning finished: 0.006884508144871404, 0.004428533287760199
    412
    EP 412 simulation finished, start learning
    4.42
    EP 412 learning finished: 0.006842405215074621, 0.004733707298058221
    413
    EP 413 simulation finished, start learning
    4.53
    EP 413 learning finished: 0.016322000961462635, 0.007448891113769562
    414
    EP 414 simulation finished, start learning
    4.23
    EP 414 learning finished: 0.01629969932784192, 0.00788759364209272
    415
    EP 415 simulation finished, start learning
    4.64
    EP 415 learning finished: 0.016278667628981225, 0.0088316158406343
    416
    EP 416 simulation finished, start learning
    4.32
    EP 416 learning finished: 0.01625930133549147, 0.008805861044896446
    417
    EP 417 simulation finished, start learning
    4.52
    EP 417 learning finished: 0.010524388140261465, 0.006387968759945295
    418
    EP 418 simulation finished, start learning
    4.52
    EP 418 learning finished: 0.00978456870580402, 0.006001586310739289
    419
    EP 419 simulation finished, start learning
    4.32
    EP 419 learning finished: 0.008356952543174145, 0.005333337962590908
    420
    EP 420 simulation finished, start learning
    3.93
    EP 420 learning finished: 0.01246733208950701, 0.00624457789718728
    421
    EP 421 simulation finished, start learning
    4.11
    EP 421 learning finished: 0.008989189719613933, 0.005765883724881367
    422
    EP 422 simulation finished, start learning
    3.95
    EP 422 learning finished: 0.012569895375058593, 0.008780350255212474
    423
    EP 423 simulation finished, start learning
    4.41
    EP 423 learning finished: 0.012554772157951533, 0.008607048416068963
    424
    EP 424 simulation finished, start learning
    4.52
    EP 424 learning finished: 0.012544446576061574, 0.0076027134007568735
    425
    EP 425 simulation finished, start learning
    4.52
    EP 425 learning finished: 0.02504889278828415, 0.00982699681328351
    426
    EP 426 simulation finished, start learning
    4.44
    EP 426 learning finished: 0.012502353816085332, 0.007574970007732773
    427
    EP 427 simulation finished, start learning
    4.32
    EP 427 learning finished: 0.012488633559780027, 0.008148248825841975
    428
    EP 428 simulation finished, start learning
    3.92
    EP 428 learning finished: 0.01282975600109637, 0.008846790171834465
    429
    EP 429 simulation finished, start learning
    4.82
    EP 429 learning finished: 0.012098711090743564, 0.006876451103511982
    430
    EP 430 simulation finished, start learning
    4.24
    EP 430 learning finished: 0.012049288855770612, 0.006586098002186939
    431
    EP 431 simulation finished, start learning
    4.43
    EP 431 learning finished: 0.010019143466166724, 0.006263957414480122
    432
    EP 432 simulation finished, start learning
    4.91
    EP 432 learning finished: 0.007302210106267637, 0.005187523466928449
    433
    EP 433 simulation finished, start learning
    4.32
    EP 433 learning finished: 0.007737815742963505, 0.00504468415946846
    434
    EP 434 simulation finished, start learning
    4.62
    EP 434 learning finished: 0.006461824857638021, 0.004540251183824988
    435
    EP 435 simulation finished, start learning
    4.24
    EP 435 learning finished: 0.012489186729652602, 0.007042354389055798
    436
    EP 436 simulation finished, start learning
    4.52
    EP 436 learning finished: 0.012480765777318936, 0.007366951579882652
    437
    EP 437 simulation finished, start learning
    4.5
    EP 437 learning finished: 0.012469481204206545, 0.006929949575972927
    438
    EP 438 simulation finished, start learning
    4.71
    EP 438 learning finished: 0.01245429484865923, 0.006638125260878998
    439
    EP 439 simulation finished, start learning
    4.14
    EP 439 learning finished: 0.0160862468157994, 0.007092327360021855
    440
    EP 440 simulation finished, start learning
    4.35
    EP 440 learning finished: 0.028469012636730184, 0.009853549285867421
    441
    EP 441 simulation finished, start learning
    4.53
    EP 441 learning finished: 0.02839291879011796, 0.010239292154291016
    442
    EP 442 simulation finished, start learning
    4.52
    EP 442 learning finished: 0.028326070791464986, 0.010476628326843004
    443
    EP 443 simulation finished, start learning
    4.33
    EP 443 learning finished: 0.012328393400832027, 0.007591261730994485
    444
    EP 444 simulation finished, start learning
    4.34
    EP 444 learning finished: 0.010368038601512808, 0.005672777406449114
    445
    EP 445 simulation finished, start learning
    4.05
    EP 445 learning finished: 0.01162274854639378, 0.006281606930410059
    446
    EP 446 simulation finished, start learning
    4.7
    EP 446 learning finished: 0.010288442377550666, 0.006262565621761714
    447
    EP 447 simulation finished, start learning
    4.14
    EP 447 learning finished: 0.011531432166094646, 0.006549807727953549
    448
    EP 448 simulation finished, start learning
    4.61
    EP 448 learning finished: 0.009273739756814336, 0.005902102122352204
    449
    EP 449 simulation finished, start learning
    4.91
    EP 449 learning finished: 0.010344279335591011, 0.005204511183946418
    450
    EP 450 simulation finished, start learning
    4.41
    EP 450 learning finished: 0.010752949597808481, 0.005462424437403968
    451
    EP 451 simulation finished, start learning
    4.5
    EP 451 learning finished: 0.010719779843434196, 0.004529725849398147
    452
    EP 452 simulation finished, start learning
    4.22
    EP 452 learning finished: 0.009153566296426163, 0.004450109885231275
    453
    EP 453 simulation finished, start learning
    4.33
    EP 453 learning finished: 0.0075956967560699695, 0.004650354281293308
    454
    EP 454 simulation finished, start learning
    4.52
    EP 454 learning finished: 0.006071617006419672, 0.004372477666845971
    455
    EP 455 simulation finished, start learning
    4.51
    EP 455 learning finished: 0.006057006396858355, 0.004576051786672271
    456
    EP 456 simulation finished, start learning
    4.31
    EP 456 learning finished: 0.006075497467932678, 0.004365711826575118
    457
    EP 457 simulation finished, start learning
    4.51
    EP 457 learning finished: 0.006126184498171483, 0.003955570583881259
    458
    EP 458 simulation finished, start learning
    4.13
    EP 458 learning finished: 0.0075224611309892, 0.004680986804016148
    459
    EP 459 simulation finished, start learning
    4.13
    EP 459 learning finished: 0.00835373713427856, 0.0051377537426589015
    460
    EP 460 simulation finished, start learning
    4.43
    EP 460 learning finished: 0.009825618522647517, 0.005791415664633111
    461
    EP 461 simulation finished, start learning
    4.44
    EP 461 learning finished: 0.009366002586062905, 0.00624328350440706
    462
    EP 462 simulation finished, start learning
    3.93
    EP 462 learning finished: 0.010616241199720378, 0.006312418804811888
    463
    EP 463 simulation finished, start learning
    4.5
    EP 463 learning finished: 0.01601265482067049, 0.008298597461958934
    464
    EP 464 simulation finished, start learning
    4.71
    EP 464 learning finished: 0.015990602111369157, 0.007501771884884434
    465
    EP 465 simulation finished, start learning
    4.8
    EP 465 learning finished: 0.01597362683196124, 0.007082157659955494
    466
    EP 466 simulation finished, start learning
    3.84
    EP 466 learning finished: 0.01595351254740507, 0.007268353627188399
    467
    EP 467 simulation finished, start learning
    4.23
    EP 467 learning finished: 0.0098632219645296, 0.005599895841325993
    468
    EP 468 simulation finished, start learning
    4.34
    EP 468 learning finished: 0.014126683852666133, 0.006577305255035526
    469
    EP 469 simulation finished, start learning
    4.52
    EP 469 learning finished: 0.015278005701780728, 0.0070092345216548795
    470
    EP 470 simulation finished, start learning
    4.82
    EP 470 learning finished: 0.011553977905286672, 0.0062562426948282734
    471
    EP 471 simulation finished, start learning
    4.62
    EP 471 learning finished: 0.01234007561632211, 0.007764797539553096
    472
    EP 472 simulation finished, start learning
    4.16
    EP 472 learning finished: 0.01233013346384837, 0.007737082105611155
    473
    EP 473 simulation finished, start learning
    4.62
    EP 473 learning finished: 0.01231662770808839, 0.007279847047385601
    474
    EP 474 simulation finished, start learning
    4.51
    EP 474 learning finished: 0.02818754876853191, 0.00946513735088613
    475
    EP 475 simulation finished, start learning
    4.41
    EP 475 learning finished: 0.015861106510175915, 0.007139441276824125
    476
    EP 476 simulation finished, start learning
    4.23
    EP 476 learning finished: 0.015843829856052594, 0.00660828965526387
    477
    EP 477 simulation finished, start learning
    4.2
    EP 477 learning finished: 0.01582554604750186, 0.006459627127887539
    478
    EP 478 simulation finished, start learning
    4.44
    EP 478 learning finished: 0.0057157881559462, 0.0043194550524224455
    479
    EP 479 simulation finished, start learning
    4.43
    EP 479 learning finished: 0.006099081227738168, 0.0046285189864102945
    480
    EP 480 simulation finished, start learning
    4.03
    EP 480 learning finished: 0.006532821002224409, 0.004847571865108573
    481
    EP 481 simulation finished, start learning
    4.53
    EP 481 learning finished: 0.00868956942236565, 0.005224721368019658
    482
    EP 482 simulation finished, start learning
    4.4
    EP 482 learning finished: 0.006741888695657394, 0.0050252158571441425
    483
    EP 483 simulation finished, start learning
    4.01
    EP 483 learning finished: 0.010591217003515396, 0.005265461807058595
    484
    EP 484 simulation finished, start learning
    3.96
    EP 484 learning finished: 0.012381791159377542, 0.005966589773178062
    485
    EP 485 simulation finished, start learning
    4.22
    EP 485 learning finished: 0.01351894156173472, 0.006271066824872662
    486
    EP 486 simulation finished, start learning
    4.61
    EP 486 learning finished: 0.012646483537061215, 0.006313890052568401
    487
    EP 487 simulation finished, start learning
    3.83
    EP 487 learning finished: 0.010367005389333883, 0.006301577049844316
    488
    EP 488 simulation finished, start learning
    4.53
    EP 488 learning finished: 0.009537615684202194, 0.005603145438791775
    489
    EP 489 simulation finished, start learning
    4.11
    EP 489 learning finished: 0.009890603371935924, 0.00506926684597057
    490
    EP 490 simulation finished, start learning
    4.61
    EP 490 learning finished: 0.007655061254681073, 0.004695140016634032
    491
    EP 491 simulation finished, start learning
    4.52
    EP 491 learning finished: 0.006077152993825052, 0.004260708783185407
    492
    EP 492 simulation finished, start learning
    4.53
    EP 492 learning finished: 0.006064434019926068, 0.004252978071066356
    493
    EP 493 simulation finished, start learning
    4.32
    EP 493 learning finished: 0.006938397310101152, 0.00470233171941814
    494
    EP 494 simulation finished, start learning
    4.5
    EP 494 learning finished: 0.008664163443675932, 0.004948972226068567
    495
    EP 495 simulation finished, start learning
    4.23
    EP 495 learning finished: 0.01586979591716542, 0.00782114583765233
    496
    EP 496 simulation finished, start learning
    4.41
    EP 496 learning finished: 0.03167484414213817, 0.009833131946294316
    497
    EP 497 simulation finished, start learning
    4.24
    EP 497 learning finished: 0.03159492759134963, 0.009799747262218736
    498
    EP 498 simulation finished, start learning
    4.71
    EP 498 learning finished: 0.04723296540068489, 0.01207943680452376
    499
    EP 499 simulation finished, start learning
    4.34
    EP 499 learning finished: 0.031383910982118124, 0.009558462332694381
    500
    EP 500 simulation finished, start learning
    4.21
    EP 500 learning finished: 0.027739030484333682, 0.009184225238108468
    501
    EP 501 simulation finished, start learning
    4.02
    EP 501 learning finished: 0.027671469553663286, 0.008976561475373401
    502
    EP 502 simulation finished, start learning
    4.82
    EP 502 learning finished: 0.012032562597806162, 0.006488608696155159
    503
    EP 503 simulation finished, start learning
    4.41
    EP 503 learning finished: 0.012022989486148106, 0.006053972510173367
    504
    EP 504 simulation finished, start learning
    4.62
    EP 504 learning finished: 0.005520693120594444, 0.0038710773134080037
    505
    EP 505 simulation finished, start learning
    3.77
    EP 505 learning finished: 0.009014236818629229, 0.00502054188721246
    506
    EP 506 simulation finished, start learning
    4.4
    EP 506 learning finished: 0.010127453769797512, 0.005254805558199867
    507
    EP 507 simulation finished, start learning
    4.41
    EP 507 learning finished: 0.010092198563022237, 0.005532923798051333
    508
    EP 508 simulation finished, start learning
    4.2
    EP 508 learning finished: 0.011199927543785293, 0.0057642658286975275
    509
    EP 509 simulation finished, start learning
    4.5
    EP 509 learning finished: 0.0077708387676835855, 0.004065672635081323
    510
    EP 510 simulation finished, start learning
    4.13
    EP 510 learning finished: 0.007741739996014019, 0.0045907535359219764
    511
    EP 511 simulation finished, start learning
    4.53
    EP 511 learning finished: 0.012029423334816158, 0.0066544392422744165
    512
    EP 512 simulation finished, start learning
    4.71
    EP 512 learning finished: 0.012014704850038527, 0.006281243664815861
    513
    EP 513 simulation finished, start learning
    4.13
    EP 513 learning finished: 0.01200092128354413, 0.007212416612221754
    514
    EP 514 simulation finished, start learning
    4.23
    EP 514 learning finished: 0.011990380532100886, 0.006833000988265188
    515
    EP 515 simulation finished, start learning
    3.45
    EP 515 learning finished: 0.009756468965221643, 0.005688364803964656
    516
    EP 516 simulation finished, start learning
    4.6
    EP 516 learning finished: 0.015515175585434804, 0.008076029816303768
    517
    EP 517 simulation finished, start learning
    4.53
    EP 517 learning finished: 0.03097410210352436, 0.010121131043660489
    518
    EP 518 simulation finished, start learning
    3.51
    EP 518 learning finished: 0.03089131252400179, 0.010365296653499575
    519
    EP 519 simulation finished, start learning
    4.32
    EP 519 learning finished: 0.03081501107980955, 0.009242944913444498
    520
    EP 520 simulation finished, start learning
    4.31
    EP 520 learning finished: 0.015375787293105958, 0.006941580557948986
    521
    EP 521 simulation finished, start learning
    4.61
    EP 521 learning finished: 0.01059387408363932, 0.0044164521188262045
    522
    EP 522 simulation finished, start learning
    4.61
    EP 522 learning finished: 0.005996281667314918, 0.003674823637484318
    523
    EP 523 simulation finished, start learning
    4.11
    EP 523 learning finished: 0.011846094367637505, 0.005701518990828565
    524
    EP 524 simulation finished, start learning
    4.41
    EP 524 learning finished: 0.01183233805041084, 0.005637421452992915
    525
    EP 525 simulation finished, start learning
    4.12
    EP 525 learning finished: 0.027133166526823516, 0.008125772391548955
    526
    EP 526 simulation finished, start learning
    4.6
    EP 526 learning finished: 0.05404663033551316, 0.012550365854413789
    527
    EP 527 simulation finished, start learning
    4.22
    EP 527 learning finished: 0.0420956618479309, 0.010890022692335558
    528
    EP 528 simulation finished, start learning
    4.23
    EP 528 learning finished: 0.041927233325207425, 0.011541369620034084
    529
    EP 529 simulation finished, start learning
    4.41
    EP 529 learning finished: 0.02669586655519751, 0.009100832345489232
    530
    EP 530 simulation finished, start learning
    4.04
    EP 530 learning finished: 0.010927447576854377, 0.005687471245131803
    531
    EP 531 simulation finished, start learning
    4.51
    EP 531 learning finished: 0.015069693187257371, 0.007570570587625743
    532
    EP 532 simulation finished, start learning
    4.23
    EP 532 learning finished: 0.015052925973311204, 0.007555619554262094
    533
    EP 533 simulation finished, start learning
    4.02
    EP 533 learning finished: 0.030048800639697504, 0.009988961263311799
    534
    EP 534 simulation finished, start learning
    4.53
    EP 534 learning finished: 0.029970751641272386, 0.009605106951519579
    535
    EP 535 simulation finished, start learning
    4.24
    EP 535 learning finished: 0.029895485481592488, 0.010095447089758736
    536
    EP 536 simulation finished, start learning
    4.33
    EP 536 learning finished: 0.044695597175483215, 0.012167942061016864
    537
    EP 537 simulation finished, start learning
    4.34
    EP 537 learning finished: 0.029698418363505397, 0.01019751976925735
    538
    EP 538 simulation finished, start learning
    4.62
    EP 538 learning finished: 0.029622564799179685, 0.009734217009438476
    539
    EP 539 simulation finished, start learning
    4.22
    EP 539 learning finished: 0.014785085689613453, 0.007021627235124228
    540
    EP 540 simulation finished, start learning
    4.41
    EP 540 learning finished: 0.0068125087892099145, 0.0043231653281120925
    541
    EP 541 simulation finished, start learning
    4.41
    EP 541 learning finished: 0.005712134354830312, 0.0038360323038292656
    542
    EP 542 simulation finished, start learning
    4.14
    EP 542 learning finished: 0.006377604011980509, 0.004562504746045211
    543
    EP 543 simulation finished, start learning
    4.11
    EP 543 learning finished: 0.007864163520601153, 0.00478580874254438
    544
    EP 544 simulation finished, start learning
    4.41
    EP 544 learning finished: 0.014780714282725427, 0.007178596484754607
    545
    EP 545 simulation finished, start learning
    4.05
    EP 545 learning finished: 0.029505178007639188, 0.009964446161036497
    546
    EP 546 simulation finished, start learning
    4.61
    EP 546 learning finished: 0.029429981060900445, 0.009361980725618224
    547
    EP 547 simulation finished, start learning
    4.53
    EP 547 learning finished: 0.029355913811745182, 0.009439630216930913
    548
    EP 548 simulation finished, start learning
    4.3
    EP 548 learning finished: 0.029281329029178238, 0.009303491643879743
    549
    EP 549 simulation finished, start learning
    4.71
    EP 549 learning finished: 0.014612251772994966, 0.0060819904536610165
    550
    EP 550 simulation finished, start learning
    3.92
    EP 550 learning finished: 0.01459708116292191, 0.0064705536670720916
    551
    EP 551 simulation finished, start learning
    4.11
    EP 551 learning finished: 0.014579169754931343, 0.006332435752919234
    552
    EP 552 simulation finished, start learning
    4.52
    EP 552 learning finished: 0.007221978744937019, 0.004471348368803992
    553
    EP 553 simulation finished, start learning
    4.11
    EP 553 learning finished: 0.02227847718254594, 0.008917229548336772
    554
    EP 554 simulation finished, start learning
    4.72
    EP 554 learning finished: 0.03672053423299815, 0.01066431762622481
    555
    EP 555 simulation finished, start learning
    4.53
    EP 555 learning finished: 0.051010283240344734, 0.012848528373173925
    556
    EP 556 simulation finished, start learning
    4.53
    EP 556 learning finished: 0.050748659184630895, 0.012623862709719566
    557
    EP 557 simulation finished, start learning
    4.6
    EP 557 learning finished: 0.028660469560673363, 0.008147220828311319
    558
    EP 558 simulation finished, start learning
    4.11
    EP 558 learning finished: 0.014303642178134623, 0.006180434743045045
    559
    EP 559 simulation finished, start learning
    4.23
    EP 559 learning finished: 0.014287733968622263, 0.006271097413695988
    560
    EP 560 simulation finished, start learning
    4.42
    EP 560 learning finished: 0.014273036855609433, 0.006149710411447629
    561
    EP 561 simulation finished, start learning
    4.22
    EP 561 learning finished: 0.014256515092608169, 0.006848689323597649
    562
    EP 562 simulation finished, start learning
    4.43
    EP 562 learning finished: 0.014240173936897804, 0.00698254523236604
    563
    EP 563 simulation finished, start learning
    4.62
    EP 563 learning finished: 0.006502905719614205, 0.004397678297306602
    564
    EP 564 simulation finished, start learning
    4.61
    EP 564 learning finished: 0.005991647143090972, 0.0041733434860707535
    565
    EP 565 simulation finished, start learning
    4.32
    EP 565 learning finished: 0.004874236706477575, 0.0037812882282940072
    566
    EP 566 simulation finished, start learning
    4.32
    EP 566 learning finished: 0.004880691284462557, 0.0033486149891821526
    567
    EP 567 simulation finished, start learning
    4.23
    EP 567 learning finished: 0.004819358741769975, 0.0035403684310999495
    568
    EP 568 simulation finished, start learning
    4.1
    EP 568 learning finished: 0.00482584845875067, 0.0035636077798351096
    569
    EP 569 simulation finished, start learning
    4.03
    EP 569 learning finished: 0.005133651299090047, 0.0038198702489253833
    570
    EP 570 simulation finished, start learning
    4.41
    EP 570 learning finished: 0.014245937324730562, 0.006030853067207668
    571
    EP 571 simulation finished, start learning
    4.12
    EP 571 learning finished: 0.025069221306544875, 0.00796273877080722
    572
    EP 572 simulation finished, start learning
    4.51
    EP 572 learning finished: 0.025003945859037202, 0.00797101738362412
    573
    EP 573 simulation finished, start learning
    4.42
    EP 573 learning finished: 0.03907096941951593, 0.009994878968636675
    574
    EP 574 simulation finished, start learning
    4.7
    EP 574 learning finished: 0.02484558950249749, 0.0077472997643634285
    575
    EP 575 simulation finished, start learning
    3.92
    EP 575 learning finished: 0.014077349010300111, 0.00611047592079809
    576
    EP 576 simulation finished, start learning
    4.7
    EP 576 learning finished: 0.014061533949039529, 0.006190545086494599
    577
    EP 577 simulation finished, start learning
    4.62
    EP 577 learning finished: 0.007126747959731006, 0.0037955690956377928
    578
    EP 578 simulation finished, start learning
    4.42
    EP 578 learning finished: 0.01404819708331928, 0.00611189639718764
    579
    EP 579 simulation finished, start learning
    4.62
    EP 579 learning finished: 0.014031956297031911, 0.005555585247331389
    580
    EP 580 simulation finished, start learning
    3.83
    EP 580 learning finished: 0.014016712605627646, 0.005907497376522149
    581
    EP 581 simulation finished, start learning
    4.62
    EP 581 learning finished: 0.014001073275391462, 0.006197783676279656
    582
    EP 582 simulation finished, start learning
    4.23
    EP 582 learning finished: 0.005249108809238123, 0.004058701557934086
    583
    EP 583 simulation finished, start learning
    4.53
    EP 583 learning finished: 0.005997306319969198, 0.004269028327241149
    584
    EP 584 simulation finished, start learning
    4.44
    EP 584 learning finished: 0.0068330760777044475, 0.004240300544790188
    585
    EP 585 simulation finished, start learning
    4.14
    EP 585 learning finished: 0.007830693082382822, 0.004639176321483991
    586
    EP 586 simulation finished, start learning
    4.32
    EP 586 learning finished: 0.0072936994108800945, 0.004472094523784165
    587
    EP 587 simulation finished, start learning
    4.53
    EP 587 learning finished: 0.006931547190482945, 0.004401863253603959
    588
    EP 588 simulation finished, start learning
    4.33
    EP 588 learning finished: 0.014004154246389788, 0.00656341172320829
    589
    EP 589 simulation finished, start learning
    4.11
    EP 589 learning finished: 0.013986333384632843, 0.005879094550827209
    590
    EP 590 simulation finished, start learning
    4.12
    EP 590 learning finished: 0.03853256340825695, 0.010118676699451731
    591
    EP 591 simulation finished, start learning
    4.62
    EP 591 learning finished: 0.038384585725445675, 0.010264018266987307
    592
    EP 592 simulation finished, start learning
    4.41
    EP 592 learning finished: 0.024402133074282517, 0.007992251927994687
    593
    EP 593 simulation finished, start learning
    4.42
    EP 593 learning finished: 0.03814769729255829, 0.010292442129891555
    594
    EP 594 simulation finished, start learning
    4.62
    EP 594 learning finished: 0.027557629393349714, 0.008277249196694925
    595
    EP 595 simulation finished, start learning
    3.71
    EP 595 learning finished: 0.027489826922067806, 0.008115908610182151
    596
    EP 596 simulation finished, start learning
    4.31
    EP 596 learning finished: 0.027424496509173224, 0.007826164762077426
    597
    EP 597 simulation finished, start learning
    4.8
    EP 597 learning finished: 0.013688708921787196, 0.005182869815255842
    598
    EP 598 simulation finished, start learning
    4.1
    EP 598 learning finished: 0.005794662063685507, 0.003148359319871711
    599
    EP 599 simulation finished, start learning
    4.61
    EP 599 learning finished: 0.004679279050016305, 0.0028630270738704646
    600
    EP 600 simulation finished, start learning
    4.03
    EP 600 learning finished: 0.004608938897334269, 0.003216343918602149
    601
    EP 601 simulation finished, start learning
    3.32
    EP 601 learning finished: 0.020737896761442443, 0.00791398166574749
    602
    EP 602 simulation finished, start learning
    3.98
    EP 602 learning finished: 0.020692311086573767, 0.008609844943409328
    603
    EP 603 simulation finished, start learning
    4.13
    EP 603 learning finished: 0.02065278348715438, 0.009268709008239876
    604
    EP 604 simulation finished, start learning
    4.23
    EP 604 learning finished: 0.03416682386494685, 0.011399845531410454
    605
    EP 605 simulation finished, start learning
    4.5
    EP 605 learning finished: 0.01354612233990693, 0.007093894057347801
    606
    EP 606 simulation finished, start learning
    4.22
    EP 606 learning finished: 0.013532642191538713, 0.00626362324582234
    607
    EP 607 simulation finished, start learning
    4.13
    EP 607 learning finished: 0.023748284450327617, 0.008058346781614508
    608
    EP 608 simulation finished, start learning
    4.25
    EP 608 learning finished: 0.010221436217257462, 0.006336945521108343
    609
    EP 609 simulation finished, start learning
    4.61
    EP 609 learning finished: 0.010210164570555236, 0.00642810330105507
    610
    EP 610 simulation finished, start learning
    4.81
    EP 610 learning finished: 0.010201845164516499, 0.006146386044141157
    611
    EP 611 simulation finished, start learning
    4.26
    EP 611 learning finished: 0.006643378117829002, 0.004359450244529957
    612
    EP 612 simulation finished, start learning
    4.6
    EP 612 learning finished: 0.01345173539893807, 0.005566649041107041
    613
    EP 613 simulation finished, start learning
    4.41
    EP 613 learning finished: 0.02685510212452158, 0.007555297405909063
    614
    EP 614 simulation finished, start learning
    3.97
    EP 614 learning finished: 0.026791338990751257, 0.008516317196444708
    615
    EP 615 simulation finished, start learning
    4.51
    EP 615 learning finished: 0.040060711300775154, 0.01003078043440316
    616
    EP 616 simulation finished, start learning
    3.95
    EP 616 learning finished: 0.02662569239030077, 0.00860688037189957
    617
    EP 617 simulation finished, start learning
    4.43
    EP 617 learning finished: 0.013288496331192334, 0.006907024954925826
    618
    EP 618 simulation finished, start learning
    4.51
    EP 618 learning finished: 0.01327531088180578, 0.006016981580465004
    619
    EP 619 simulation finished, start learning
    4.52
    EP 619 learning finished: 0.01003410448551989, 0.006024052554386354
    620
    EP 620 simulation finished, start learning
    4.4
    EP 620 learning finished: 0.010025966669751085, 0.005571777479661459
    621
    EP 621 simulation finished, start learning
    4.21
    EP 621 learning finished: 0.010015293800663017, 0.005131482738107707
    622
    EP 622 simulation finished, start learning
    4.71
    EP 622 learning finished: 0.023212267281401466, 0.007229256909885369
    623
    EP 623 simulation finished, start learning
    4.44
    EP 623 learning finished: 0.013194760419025897, 0.005398695279441294
    624
    EP 624 simulation finished, start learning
    4.31
    EP 624 learning finished: 0.0131786015818669, 0.005564695363537098
    625
    EP 625 simulation finished, start learning
    4.71
    EP 625 learning finished: 0.013165490692177428, 0.005653838788664582
    626
    EP 626 simulation finished, start learning
    4.5
    EP 626 learning finished: 0.006349379766533048, 0.003705997864326349
    627
    EP 627 simulation finished, start learning
    4.5
    EP 627 learning finished: 0.006422457170529583, 0.003513449289267593
    628
    EP 628 simulation finished, start learning
    4.41
    EP 628 learning finished: 0.006414844345428605, 0.0034647227251222525
    629
    EP 629 simulation finished, start learning
    4.2
    EP 629 learning finished: 0.009941534577443178, 0.005017464596442996
    630
    EP 630 simulation finished, start learning
    4.02
    EP 630 learning finished: 0.009929853818118062, 0.005357227786118912
    631
    EP 631 simulation finished, start learning
    4.32
    EP 631 learning finished: 0.009917516616848587, 0.005635906973629663
    632
    EP 632 simulation finished, start learning
    4.32
    EP 632 learning finished: 0.009906847360497295, 0.0057701619775277395
    633
    EP 633 simulation finished, start learning
    3.86
    EP 633 learning finished: 0.01070175859954503, 0.005083329650935535
    634
    EP 634 simulation finished, start learning
    4.24
    EP 634 learning finished: 0.011521971608203785, 0.0050014053817276265
    635
    EP 635 simulation finished, start learning
    4.62
    EP 635 learning finished: 0.011476384547449925, 0.00490100054164581
    636
    EP 636 simulation finished, start learning
    4.36
    EP 636 learning finished: 0.013116996837673867, 0.0072079192084165316
    637
    EP 637 simulation finished, start learning
    3.94
    EP 637 learning finished: 0.026187606882054126, 0.008854950412775509
    638
    EP 638 simulation finished, start learning
    4.64
    EP 638 learning finished: 0.02612760997543262, 0.00896983798123093
    639
    EP 639 simulation finished, start learning
    4.2
    EP 639 learning finished: 0.02606427683458179, 0.00859131595832352
    640
    EP 640 simulation finished, start learning
    4.02
    EP 640 learning finished: 0.013009054549012689, 0.006336093031456731
    641
    EP 641 simulation finished, start learning
    4.42
    EP 641 learning finished: 0.012995321434292606, 0.005894698204968425
    642
    EP 642 simulation finished, start learning
    4.44
    EP 642 learning finished: 0.025945701001404586, 0.00781640372512508
    643
    EP 643 simulation finished, start learning
    4.22
    EP 643 learning finished: 0.025885934829879886, 0.008257410710786672
    644
    EP 644 simulation finished, start learning
    3.93
    EP 644 learning finished: 0.025824296561740567, 0.008157437818898808
    645
    EP 645 simulation finished, start learning
    4.31
    EP 645 learning finished: 0.012888318233926555, 0.005943574665556876
    646
    EP 646 simulation finished, start learning
    4.71
    EP 646 learning finished: 0.007096924118023651, 0.0036440914505616226
    647
    EP 647 simulation finished, start learning
    4.02
    EP 647 learning finished: 0.005730297631075132, 0.0037049434509555113
    648
    EP 648 simulation finished, start learning
    4.42
    EP 648 learning finished: 0.012880142478543632, 0.005531456755807795
    649
    EP 649 simulation finished, start learning
    4.62
    EP 649 learning finished: 0.012866791133099298, 0.00582783150339002
    650
    EP 650 simulation finished, start learning
    4.51
    EP 650 learning finished: 0.03534995929837237, 0.009434682355238996
    651
    EP 651 simulation finished, start learning
    4.61
    EP 651 learning finished: 0.0352208227849502, 0.009145364498272338
    652
    EP 652 simulation finished, start learning
    4.26
    EP 652 learning finished: 0.02236155384970573, 0.00750708741657391
    653
    EP 653 simulation finished, start learning
    4.31
    EP 653 learning finished: 0.02231124704553622, 0.00747966100481771
    654
    EP 654 simulation finished, start learning
    4.62
    EP 654 learning finished: 0.007619277219731657, 0.004025411852167575
    655
    EP 655 simulation finished, start learning
    4.31
    EP 655 learning finished: 0.00585207351232242, 0.0038063181842223535
    656
    EP 656 simulation finished, start learning
    4.24
    EP 656 learning finished: 0.005021539979445322, 0.003615628391450798
    657
    EP 657 simulation finished, start learning
    4.53
    EP 657 learning finished: 0.012715510442493638, 0.005585005073885559
    658
    EP 658 simulation finished, start learning
    4.73
    EP 658 learning finished: 0.01270306222643569, 0.005408969892030768
    659
    EP 659 simulation finished, start learning
    4.42
    EP 659 learning finished: 0.012688002035428525, 0.005505794863295795
    660
    EP 660 simulation finished, start learning
    4.32
    EP 660 learning finished: 0.012674321709461367, 0.005217040082901765
    661
    EP 661 simulation finished, start learning
    4.32
    EP 661 learning finished: 0.012661430130499696, 0.005158779325132201
    662
    EP 662 simulation finished, start learning
    4.32
    EP 662 learning finished: 0.012648016225810466, 0.005187763788158624
    663
    EP 663 simulation finished, start learning
    4.02
    EP 663 learning finished: 0.022126519564565683, 0.006976801395978782
    664
    EP 664 simulation finished, start learning
    4.53
    EP 664 learning finished: 0.02207710081946246, 0.0070668204742746115
    665
    EP 665 simulation finished, start learning
    4.44
    EP 665 learning finished: 0.009460025382107773, 0.005190989976895229
    666
    EP 666 simulation finished, start learning
    4.22
    EP 666 learning finished: 0.009452424342896393, 0.005439644020588237
    667
    EP 667 simulation finished, start learning
    4.04
    EP 667 learning finished: 0.006285556820110116, 0.003948918891578519
    668
    EP 668 simulation finished, start learning
    4.43
    EP 668 learning finished: 0.006116105508773728, 0.004196817020711545
    669
    EP 669 simulation finished, start learning
    4.32
    EP 669 learning finished: 0.02831691169584971, 0.009201840308094825
    670
    EP 670 simulation finished, start learning
    3.93
    EP 670 learning finished: 0.028222135427110828, 0.009083686602001132
    671
    EP 671 simulation finished, start learning
    4.04
    EP 671 learning finished: 0.028131211309226245, 0.009338075776652133
    672
    EP 672 simulation finished, start learning
    4.13
    EP 672 learning finished: 0.02803817203913539, 0.009392359599096642
    673
    EP 673 simulation finished, start learning
    4.34
    EP 673 learning finished: 0.007261856455784255, 0.004668169439103309
    674
    EP 674 simulation finished, start learning
    4.8
    EP 674 learning finished: 0.006053075985718692, 0.004154691799027906
    675
    EP 675 simulation finished, start learning
    4.61
    EP 675 learning finished: 0.005480818863283332, 0.0033985548001567167
    676
    EP 676 simulation finished, start learning
    4.22
    EP 676 learning finished: 0.005470372445714133, 0.003263204309426726
    677
    EP 677 simulation finished, start learning
    4.53
    EP 677 learning finished: 0.005199784561099818, 0.0030886363380666637
    678
    EP 678 simulation finished, start learning
    4.2
    EP 678 learning finished: 0.006234362410793892, 0.003476942551966289
    679
    EP 679 simulation finished, start learning
    4.41
    EP 679 learning finished: 0.012420095583532742, 0.005577231505225082
    680
    EP 680 simulation finished, start learning
    3.91
    EP 680 learning finished: 0.021719107646816838, 0.006947280225838644
    681
    EP 681 simulation finished, start learning
    4.04
    EP 681 learning finished: 0.021668877292526645, 0.0069221145458134474
    682
    EP 682 simulation finished, start learning
    4.62
    EP 682 learning finished: 0.021620795722327557, 0.00673398822189597
    683
    EP 683 simulation finished, start learning
    4.7
    EP 683 learning finished: 0.009258794743558951, 0.004497337832800004
    684
    EP 684 simulation finished, start learning
    4.82
    EP 684 learning finished: 0.003963484330768097, 0.0029652574678600274
    685
    EP 685 simulation finished, start learning
    4.32
    EP 685 learning finished: 0.004606511811928726, 0.002922343738150109
    686
    EP 686 simulation finished, start learning
    4.51
    EP 686 learning finished: 0.012319907207881897, 0.004879128838774467
    687
    EP 687 simulation finished, start learning
    4.41
    EP 687 learning finished: 0.01230780295494327, 0.0053224397708954525
    688
    EP 688 simulation finished, start learning
    3.72
    EP 688 learning finished: 0.024574353537175853, 0.00738216469967649
    689
    EP 689 simulation finished, start learning
    4.55
    EP 689 learning finished: 0.024519328699518894, 0.007635789323793792
    690
    EP 690 simulation finished, start learning
    4.21
    EP 690 learning finished: 0.03361282861917658, 0.009309205786937724
    691
    EP 691 simulation finished, start learning
    4.22
    EP 691 learning finished: 0.03349754552428763, 0.009170982309097889
    692
    EP 692 simulation finished, start learning
    4.21
    EP 692 learning finished: 0.02125101494949888, 0.006925661731327655
    693
    EP 693 simulation finished, start learning
    4.41
    EP 693 learning finished: 0.030268380692820784, 0.008137654236062867
    694
    EP 694 simulation finished, start learning
    4.21
    EP 694 learning finished: 0.021131400834327962, 0.006525366001768636
    695
    EP 695 simulation finished, start learning
    3.82
    EP 695 learning finished: 0.042116441219633316, 0.010184365805924264
    696
    EP 696 simulation finished, start learning
    4.22
    EP 696 learning finished: 0.04191991984019339, 0.010224600137571946
    697
    EP 697 simulation finished, start learning
    4.31
    EP 697 learning finished: 0.032820879066274955, 0.008632102821595211
    698
    EP 698 simulation finished, start learning
    4.62
    EP 698 learning finished: 0.02081413573042923, 0.006633991501371265
    699
    EP 699 simulation finished, start learning
    4.44
    EP 699 learning finished: 0.011895546127998813, 0.005120204794680561
    700
    EP 700 simulation finished, start learning
    4.34
    EP 700 learning finished: 0.020744062631763427, 0.006859036620742387
    701
    EP 701 simulation finished, start learning
    4.52
    EP 701 learning finished: 0.020697536021444307, 0.006650870220251903
    702
    EP 702 simulation finished, start learning
    4.71
    EP 702 learning finished: 0.03245964349150299, 0.00848307120517037
    703
    EP 703 simulation finished, start learning
    4.53
    EP 703 learning finished: 0.02058133387999206, 0.006376141329984246
    704
    EP 704 simulation finished, start learning
    4.41
    EP 704 learning finished: 0.011770770339527559, 0.004474844638696259
    705
    EP 705 simulation finished, start learning
    4.3
    EP 705 learning finished: 0.023500770479316202, 0.006248860315431888
    706
    EP 706 simulation finished, start learning
    4.41
    EP 706 learning finished: 0.011732854909837442, 0.004632126836877859
    707
    EP 707 simulation finished, start learning
    4.41
    EP 707 learning finished: 0.011719952357267971, 0.004719206696429741
    708
    EP 708 simulation finished, start learning
    4.53
    EP 708 learning finished: 0.023401251042407978, 0.006840193374785031
    709
    EP 709 simulation finished, start learning
    4.23
    EP 709 learning finished: 0.011682185866561667, 0.005265948085175627
    710
    EP 710 simulation finished, start learning
    4.91
    EP 710 learning finished: 0.011669498874613993, 0.00512155043151985
    711
    EP 711 simulation finished, start learning
    3.84
    EP 711 learning finished: 0.01165750999476245, 0.005341865067219231
    712
    EP 712 simulation finished, start learning
    4.26
    EP 712 learning finished: 0.011646226170501706, 0.005707207786452173
    713
    EP 713 simulation finished, start learning
    4.22
    EP 713 learning finished: 0.011634342559020183, 0.005563721501605556
    714
    EP 714 simulation finished, start learning
    4.34
    EP 714 learning finished: 0.011622616373063721, 0.006001277023493993
    715
    EP 715 simulation finished, start learning
    4.11
    EP 715 learning finished: 0.011609971188914159, 0.005619708510807324
    716
    EP 716 simulation finished, start learning
    4.17
    EP 716 learning finished: 0.006236905696276551, 0.003845620586610939
    717
    EP 717 simulation finished, start learning
    3.64
    EP 717 learning finished: 0.008636471915210109, 0.0057269024051992785
    718
    EP 718 simulation finished, start learning
    4.62
    EP 718 learning finished: 0.008631953817726412, 0.005186742422728467
    719
    EP 719 simulation finished, start learning
    4.52
    EP 719 learning finished: 0.008624746999878141, 0.0051399870797403935
    720
    EP 720 simulation finished, start learning
    4.11
    EP 720 learning finished: 0.008617465169844962, 0.004510051930247154
    721
    EP 721 simulation finished, start learning
    3.75
    EP 721 learning finished: 0.005276837524391453, 0.003292088226089618
    722
    EP 722 simulation finished, start learning
    4.61
    EP 722 learning finished: 0.004897560074937246, 0.0032027621406527373
    723
    EP 723 simulation finished, start learning
    4.34
    EP 723 learning finished: 0.008615728479991702, 0.004962827779952643
    724
    EP 724 simulation finished, start learning
    4.62
    EP 724 learning finished: 0.008607010326073805, 0.004927173894811195
    725
    EP 725 simulation finished, start learning
    4.71
    EP 725 learning finished: 0.020133666970885145, 0.005949201923247216
    726
    EP 726 simulation finished, start learning
    3.72
    EP 726 learning finished: 0.02008993925029401, 0.006252129386242089
    727
    EP 727 simulation finished, start learning
    4.81
    EP 727 learning finished: 0.01149722817708465, 0.00434073711352075
    728
    EP 728 simulation finished, start learning
    4.32
    EP 728 learning finished: 0.011485688908983605, 0.004404238933778022
    729
    EP 729 simulation finished, start learning
    4.32
    EP 729 learning finished: 0.0036940867092312146, 0.0026788054835051867
    730
    EP 730 simulation finished, start learning
    4.6
    EP 730 learning finished: 0.011476345734589444, 0.004158844566258442
    731
    EP 731 simulation finished, start learning
    4.23
    EP 731 learning finished: 0.022917204596211604, 0.006371707468617889
    732
    EP 732 simulation finished, start learning
    4.23
    EP 732 learning finished: 0.022865930563746417, 0.006486610336389494
    733
    EP 733 simulation finished, start learning
    4.42
    EP 733 learning finished: 0.034201839143076285, 0.00833882785394439
    734
    EP 734 simulation finished, start learning
    4.34
    EP 734 learning finished: 0.022739782545126985, 0.007163171235569976
    735
    EP 735 simulation finished, start learning
    4.14
    EP 735 learning finished: 0.011351828471362069, 0.00538180766368157
    736
    EP 736 simulation finished, start learning
    4.2
    EP 736 learning finished: 0.011340571688300827, 0.005084004854632933
    737
    EP 737 simulation finished, start learning
    4.5
    EP 737 learning finished: 0.01132861496373394, 0.004876489842623381
    738
    EP 738 simulation finished, start learning
    4.31
    EP 738 learning finished: 0.011318053307537213, 0.004567429136239198
    739
    EP 739 simulation finished, start learning
    4.43
    EP 739 learning finished: 0.011306308653981745, 0.004639096072831824
    740
    EP 740 simulation finished, start learning
    4.24
    EP 740 learning finished: 0.011293618167911745, 0.005043703885801183
    741
    EP 741 simulation finished, start learning
    4.32
    EP 741 learning finished: 0.005461552333898112, 0.00365053688435187
    742
    EP 742 simulation finished, start learning
    4.71
    EP 742 learning finished: 0.011284658727280572, 0.005180372000445151
    743
    EP 743 simulation finished, start learning
    4.21
    EP 743 learning finished: 0.0112730319198171, 0.004817060193037885
    744
    EP 744 simulation finished, start learning
    4.6
    EP 744 learning finished: 0.02251268240957316, 0.00622845175945308
    745
    EP 745 simulation finished, start learning
    4.53
    EP 745 learning finished: 0.02246225070975072, 0.0060493754281102
    746
    EP 746 simulation finished, start learning
    4.02
    EP 746 learning finished: 0.011212947759599246, 0.004438933670880256
    747
    EP 747 simulation finished, start learning
    4.8
    EP 747 learning finished: 0.02239242852509804, 0.0060297360963274356
    748
    EP 748 simulation finished, start learning
    4.13
    EP 748 learning finished: 0.02770917781144533, 0.007442479240851914
    749
    EP 749 simulation finished, start learning
    4.61
    EP 749 learning finished: 0.03583949404249509, 0.008864654303465022
    750
    EP 750 simulation finished, start learning
    4.14
    EP 750 learning finished: 0.03567625923165707, 0.009034947529895495
    751
    EP 751 simulation finished, start learning
    4.62
    EP 751 learning finished: 0.024500176577390675, 0.007405139910244903
    752
    EP 752 simulation finished, start learning
    4.32
    EP 752 learning finished: 0.008150911424365859, 0.0046169275939777306
    753
    EP 753 simulation finished, start learning
    4.32
    EP 753 learning finished: 0.005056555196482293, 0.00309331867102595
    754
    EP 754 simulation finished, start learning
    4.5
    EP 754 learning finished: 0.011015108603359636, 0.004416728675590065
    755
    EP 755 simulation finished, start learning
    4.23
    EP 755 learning finished: 0.01100428909577289, 0.004639469342433718
    756
    EP 756 simulation finished, start learning
    4.31
    EP 756 learning finished: 0.010993422227961958, 0.004305032202312968
    757
    EP 757 simulation finished, start learning
    4.52
    EP 757 learning finished: 0.010982561058259677, 0.004270594050174381
    758
    EP 758 simulation finished, start learning
    4.34
    EP 758 learning finished: 0.004087190445709155, 0.003097219573852783
    759
    EP 759 simulation finished, start learning
    4.14
    EP 759 learning finished: 0.0053578534902314345, 0.003397547648908983
    760
    EP 760 simulation finished, start learning
    4.32
    EP 760 learning finished: 0.010976573062006167, 0.005298016646084018
    761
    EP 761 simulation finished, start learning
    4.42
    EP 761 learning finished: 0.01096531405765333, 0.005328872915531117
    762
    EP 762 simulation finished, start learning
    4.35
    EP 762 learning finished: 0.02189498376350585, 0.006994410075050084
    763
    EP 763 simulation finished, start learning
    4.43
    EP 763 learning finished: 0.02184965182002685, 0.006603728962423598
    764
    EP 764 simulation finished, start learning
    3.75
    EP 764 learning finished: 0.010907568678291568, 0.00523238184235808
    765
    EP 765 simulation finished, start learning
    4.15
    EP 765 learning finished: 0.010897384493670614, 0.00556705406348416
    766
    EP 766 simulation finished, start learning
    4.52
    EP 766 learning finished: 0.006697983537257812, 0.003683788620500413
    767
    EP 767 simulation finished, start learning
    4.04
    EP 767 learning finished: 0.007025569079999627, 0.004086057846976132
    768
    EP 768 simulation finished, start learning
    4.43
    EP 768 learning finished: 0.006318856454178956, 0.0037058023705102507
    769
    EP 769 simulation finished, start learning
    4.33
    EP 769 learning finished: 0.005614602188146023, 0.0034723185550606197
    770
    EP 770 simulation finished, start learning
    4.2
    EP 770 learning finished: 0.004372172262840479, 0.003097774393459929
    771
    EP 771 simulation finished, start learning
    4.32
    EP 771 learning finished: 0.004224013704032359, 0.0028505778101409382
    772
    EP 772 simulation finished, start learning
    4.22
    EP 772 learning finished: 0.0038747817143587326, 0.0027985941975975
    773
    EP 773 simulation finished, start learning
    4.31
    EP 773 learning finished: 0.005040360494064089, 0.0028265081232762648
    774
    EP 774 simulation finished, start learning
    4.33
    EP 774 learning finished: 0.004340334683590186, 0.0030974991952798328
    775
    EP 775 simulation finished, start learning
    4.12
    EP 775 learning finished: 0.010902177504469535, 0.004700346708541285
    776
    EP 776 simulation finished, start learning
    4.04
    EP 776 learning finished: 0.010891497447754972, 0.0051660386945595604
    777
    EP 777 simulation finished, start learning
    4.13
    EP 777 learning finished: 0.010880924813275072, 0.0053821080795947536
    778
    EP 778 simulation finished, start learning
    4.32
    EP 778 learning finished: 0.010870184521735382, 0.005481124551803163
    779
    EP 779 simulation finished, start learning
    4.62
    EP 779 learning finished: 0.005409650077098149, 0.0035557619141203625
    780
    EP 780 simulation finished, start learning
    4.05
    EP 780 learning finished: 0.006513993415914854, 0.003607180739843822
    781
    EP 781 simulation finished, start learning
    4.53
    EP 781 learning finished: 0.004484567645960169, 0.0032714552526596683
    782
    EP 782 simulation finished, start learning
    4.22
    EP 782 learning finished: 0.0047023766159384195, 0.003300832052519335
    783
    EP 783 simulation finished, start learning
    4.52
    EP 783 learning finished: 0.00625525107063396, 0.003555441337057393
    784
    EP 784 simulation finished, start learning
    4.52
    EP 784 learning finished: 0.004684171459132358, 0.0029644097438913755
    785
    EP 785 simulation finished, start learning
    4.41
    EP 785 learning finished: 0.008040216916462839, 0.004263535580108169
    786
    EP 786 simulation finished, start learning
    4.43
    EP 786 learning finished: 0.008033401793344552, 0.004300749820515056
    787
    EP 787 simulation finished, start learning
    4.21
    EP 787 learning finished: 0.00802593858019307, 0.004071204207126182
    788
    EP 788 simulation finished, start learning
    4.51
    EP 788 learning finished: 0.01885207688504309, 0.005758993848711332
    789
    EP 789 simulation finished, start learning
    4.61
    EP 789 learning finished: 0.010822023265259162, 0.004268348398153303
    790
    EP 790 simulation finished, start learning
    4.24
    EP 790 learning finished: 0.010811305436405405, 0.004245160216549711
    791
    EP 791 simulation finished, start learning
    4.42
    EP 791 learning finished: 0.01080088228436793, 0.004282674794641987
    792
    EP 792 simulation finished, start learning
    4.62
    EP 792 learning finished: 0.003717000486048505, 0.0025524805949916165
    793
    EP 793 simulation finished, start learning
    4.5
    EP 793 learning finished: 0.015938608410458244, 0.005517252747773445
    794
    EP 794 simulation finished, start learning
    4.41
    EP 794 learning finished: 0.015909195747354943, 0.0051198475208700245
    795
    EP 795 simulation finished, start learning
    4.61
    EP 795 learning finished: 0.015874089777426548, 0.005201417543400311
    796
    EP 796 simulation finished, start learning
    4.51
    EP 796 learning finished: 0.023746583441202063, 0.006549455686824264
    797
    EP 797 simulation finished, start learning
    4.05
    EP 797 learning finished: 0.007898773923603386, 0.004305657345028043
    798
    EP 798 simulation finished, start learning
    4.51
    EP 798 learning finished: 0.018577018346981333, 0.006050093146529416
    799
    EP 799 simulation finished, start learning
    4.51
    EP 799 learning finished: 0.029189255836275407, 0.007577531830817108
    800
    EP 800 simulation finished, start learning
    4.32
    EP 800 learning finished: 0.021269098908099693, 0.006522639396755163
    801
    EP 801 simulation finished, start learning
    4.32
    EP 801 learning finished: 0.03181967667584118, 0.007716201002955003
    802
    EP 802 simulation finished, start learning
    3.75
    EP 802 learning finished: 0.02115735223207693, 0.0066888422283764015
    803
    EP 803 simulation finished, start learning
    4.43
    EP 803 learning finished: 0.01056267875317829, 0.005369358283013881
    804
    EP 804 simulation finished, start learning
    4.13
    EP 804 learning finished: 0.01831088912339851, 0.0066334377469606425
    805
    EP 805 simulation finished, start learning
    4.23
    EP 805 learning finished: 0.0077541557998728194, 0.004977545189268777
    806
    EP 806 simulation finished, start learning
    3.74
    EP 806 learning finished: 0.00774862051271441, 0.004675396254904503
    807
    EP 807 simulation finished, start learning
    3.93
    EP 807 learning finished: 0.018244664494883267, 0.006285138418271312
    808
    EP 808 simulation finished, start learning
    4.05
    EP 808 learning finished: 0.01820933015652315, 0.006488412189339006
    809
    EP 809 simulation finished, start learning
    4.02
    EP 809 learning finished: 0.025859569040662223, 0.007786302578418021
    810
    EP 810 simulation finished, start learning
    4.32
    EP 810 learning finished: 0.02578465073399272, 0.007690362366354601
    811
    EP 811 simulation finished, start learning
    4.71
    EP 811 learning finished: 0.0333385203301404, 0.008668181638860517
    812
    EP 812 simulation finished, start learning
    4.32
    EP 812 learning finished: 0.025595244641703374, 0.006951763433590348
    813
    EP 813 simulation finished, start learning
    4.03
    EP 813 learning finished: 0.017933816094089394, 0.005709419113962018
    814
    EP 814 simulation finished, start learning
    4.71
    EP 814 learning finished: 0.017897241498232574, 0.005528181002880217
    815
    EP 815 simulation finished, start learning
    4.53
    EP 815 learning finished: 0.00425815605657931, 0.002680736072707172
    816
    EP 816 simulation finished, start learning
    4.41
    EP 816 learning finished: 0.010303544288245092, 0.004286807075809307
    817
    EP 817 simulation finished, start learning
    4.32
    EP 817 learning finished: 0.010294538468915704, 0.004144256846765042
    818
    EP 818 simulation finished, start learning
    4.51
    EP 818 learning finished: 0.010284462896649416, 0.003980841400026709
    819
    EP 819 simulation finished, start learning
    3.93
    EP 819 learning finished: 0.010275397363592731, 0.00434182980533393
    820
    EP 820 simulation finished, start learning
    4.62
    EP 820 learning finished: 0.003915955613322808, 0.002661961716802589
    821
    EP 821 simulation finished, start learning
    4.71
    EP 821 learning finished: 0.004968829995291912, 0.002679862815746503
    822
    EP 822 simulation finished, start learning
    4.62
    EP 822 learning finished: 0.010267602951763877, 0.004421647385614828
    823
    EP 823 simulation finished, start learning
    4.61
    EP 823 learning finished: 0.010259285145831853, 0.004094076909562834
    824
    EP 824 simulation finished, start learning
    3.92
    EP 824 learning finished: 0.010248457511920606, 0.004260744141513821
    825
    EP 825 simulation finished, start learning
    4.81
    EP 825 learning finished: 0.010238762488445019, 0.004032599047573508
    826
    EP 826 simulation finished, start learning
    4.45
    EP 826 learning finished: 0.010230064699182861, 0.004307417420357377
    827
    EP 827 simulation finished, start learning
    4.62
    EP 827 learning finished: 0.020428515060451957, 0.005862651641776277
    828
    EP 828 simulation finished, start learning
    4.23
    EP 828 learning finished: 0.0203875031324543, 0.0058519664185100835
    829
    EP 829 simulation finished, start learning
    4.71
    EP 829 learning finished: 0.02034506090962296, 0.006064615494565236
    830
    EP 830 simulation finished, start learning
    4.3
    EP 830 learning finished: 0.02030437673627361, 0.005669729802932603
    831
    EP 831 simulation finished, start learning
    4.43
    EP 831 learning finished: 0.010136814965537866, 0.00410644175340084
    832
    EP 832 simulation finished, start learning
    4.3
    EP 832 learning finished: 0.010127938357371136, 0.0038655901629501463
    833
    EP 833 simulation finished, start learning
    4.53
    EP 833 learning finished: 0.020222991607614205, 0.0055260137050255525
    834
    EP 834 simulation finished, start learning
    4.12
    EP 834 learning finished: 0.010097600214908908, 0.00407943670081264
    835
    EP 835 simulation finished, start learning
    4.22
    EP 835 learning finished: 0.010086911907201568, 0.004266309118840184
    836
    EP 836 simulation finished, start learning
    4.44
    EP 836 learning finished: 0.010077195493732205, 0.004747729309560948
    837
    EP 837 simulation finished, start learning
    4.41
    EP 837 learning finished: 0.004456419123584965, 0.0029701731236831583
    838
    EP 838 simulation finished, start learning
    4.52
    EP 838 learning finished: 0.004657252756949504, 0.003073705919837503
    839
    EP 839 simulation finished, start learning
    4.46
    EP 839 learning finished: 0.00505780171928194, 0.0033330668759153546
    840
    EP 840 simulation finished, start learning
    3.74
    EP 840 learning finished: 0.007372284377847649, 0.004785733383608104
    841
    EP 841 simulation finished, start learning
    4.11
    EP 841 learning finished: 0.00736657557300191, 0.0048338850865867755
    842
    EP 842 simulation finished, start learning
    3.71
    EP 842 learning finished: 0.0073595242510862136, 0.004712078362446211
    843
    EP 843 simulation finished, start learning
    4.02
    EP 843 learning finished: 0.007354820074837676, 0.0044590676556512205
    844
    EP 844 simulation finished, start learning
    4.22
    EP 844 learning finished: 0.005852774656418092, 0.0029081646467462954
    845
    EP 845 simulation finished, start learning
    4.17
    EP 845 learning finished: 0.006241134834532147, 0.0035531600503754626
    846
    EP 846 simulation finished, start learning
    4.43
    EP 846 learning finished: 0.005615773797494228, 0.003551721361330825
    847
    EP 847 simulation finished, start learning
    4.42
    EP 847 learning finished: 0.005603714323854092, 0.0032796600394261097
    848
    EP 848 simulation finished, start learning
    4.26
    EP 848 learning finished: 0.006208049267712568, 0.0033566778254012775
    849
    EP 849 simulation finished, start learning
    4.43
    EP 849 learning finished: 0.0048617825092591715, 0.002772090878218355
    850
    EP 850 simulation finished, start learning
    4.7
    EP 850 learning finished: 0.003857378419290569, 0.002473277953383738
    851
    EP 851 simulation finished, start learning
    4.02
    EP 851 learning finished: 0.003851283010440909, 0.0025653310753194583
    852
    EP 852 simulation finished, start learning
    4.14
    EP 852 learning finished: 0.003185844353801315, 0.0024313154488607345
    853
    EP 853 simulation finished, start learning
    4.03
    EP 853 learning finished: 0.010055919485349563, 0.0041704609274363
    854
    EP 854 simulation finished, start learning
    4.22
    EP 854 learning finished: 0.01004668498649508, 0.004422892073257263
    855
    EP 855 simulation finished, start learning
    4.41
    EP 855 learning finished: 0.017378029948829976, 0.005712419388262428
    856
    EP 856 simulation finished, start learning
    4.42
    EP 856 learning finished: 0.027342378791253452, 0.0071364428314784805
    857
    EP 857 simulation finished, start learning
    4.14
    EP 857 learning finished: 0.017289212043444024, 0.0057438618398582735
    858
    EP 858 simulation finished, start learning
    3.92
    EP 858 learning finished: 0.01725351950588916, 0.005796704866878481
    859
    EP 859 simulation finished, start learning
    4.62
    EP 859 learning finished: 0.009949816678258117, 0.004340682195280721
    860
    EP 860 simulation finished, start learning
    4.34
    EP 860 learning finished: 0.006208236287498892, 0.0030938978572798703
    861
    EP 861 simulation finished, start learning
    4.71
    EP 861 learning finished: 0.003994996277263851, 0.0027165634831109267
    862
    EP 862 simulation finished, start learning
    3.72
    EP 862 learning finished: 0.007275550599036507, 0.0042409972208539886
    863
    EP 863 simulation finished, start learning
    4.6
    EP 863 learning finished: 0.017195794450255875, 0.0057273536996256995
    864
    EP 864 simulation finished, start learning
    3.84
    EP 864 learning finished: 0.01716034788172749, 0.005810740442028599
    865
    EP 865 simulation finished, start learning
    4.62
    EP 865 learning finished: 0.027004651177563233, 0.0073432425541154105
    866
    EP 866 simulation finished, start learning
    4.14
    EP 866 learning finished: 0.0197296919403378, 0.005953419934067838
    867
    EP 867 simulation finished, start learning
    4.05
    EP 867 learning finished: 0.009850017291650914, 0.005023564912075192
    868
    EP 868 simulation finished, start learning
    4.04
    EP 868 learning finished: 0.009841418791051382, 0.0049713979865781965
    869
    EP 869 simulation finished, start learning
    4.64
    EP 869 learning finished: 0.005682046735437041, 0.0034910859646627066
    870
    EP 870 simulation finished, start learning
    4.53
    EP 870 learning finished: 0.00522406185359614, 0.003491975356801902
    871
    EP 871 simulation finished, start learning
    4.13
    EP 871 learning finished: 0.005386904448144492, 0.0033365404234958853
    872
    EP 872 simulation finished, start learning
    4.82
    EP 872 learning finished: 0.004791809081595488, 0.0028181233425949395
    873
    EP 873 simulation finished, start learning
    4.52
    EP 873 learning finished: 0.004199543591170141, 0.002880255730996595
    874
    EP 874 simulation finished, start learning
    4.44
    EP 874 learning finished: 0.004483741566986055, 0.002807024410752016
    875
    EP 875 simulation finished, start learning
    3.71
    EP 875 learning finished: 0.0033998969883271, 0.0025240384075845648
    876
    EP 876 simulation finished, start learning
    4.72
    EP 876 learning finished: 0.003395231587411729, 0.002552458993517822
    877
    EP 877 simulation finished, start learning
    4.31
    EP 877 learning finished: 0.003597835523670767, 0.0025273176389800276
    878
    EP 878 simulation finished, start learning
    4.8
    EP 878 learning finished: 0.003784374241114108, 0.002351768684550374
    879
    EP 879 simulation finished, start learning
    4.41
    EP 879 learning finished: 0.00310618468020607, 0.0022285034434759154
    880
    EP 880 simulation finished, start learning
    4.5
    EP 880 learning finished: 0.003378339880093506, 0.00209476509501035
    881
    EP 881 simulation finished, start learning
    3.96
    EP 881 learning finished: 0.009849847886634677, 0.003970578976831026
    882
    EP 882 simulation finished, start learning
    4.5
    EP 882 learning finished: 0.009840798043459387, 0.003981673865674533
    883
    EP 883 simulation finished, start learning
    4.22
    EP 883 learning finished: 0.009832529306013527, 0.004280353057323438
    884
    EP 884 simulation finished, start learning
    4.81
    EP 884 learning finished: 0.009823394555247944, 0.004311670570635566
    885
    EP 885 simulation finished, start learning
    4.11
    EP 885 learning finished: 0.003551436808379549, 0.002252626836747614
    886
    EP 886 simulation finished, start learning
    4.43
    EP 886 learning finished: 0.004540257400137193, 0.0025191163011477613
    887
    EP 887 simulation finished, start learning
    4.41
    EP 887 learning finished: 0.003149895490025756, 0.0023776787276545974
    888
    EP 888 simulation finished, start learning
    4.22
    EP 888 learning finished: 0.003931232744122326, 0.002553622733187192
    889
    EP 889 simulation finished, start learning
    4.11
    EP 889 learning finished: 0.003923590410742861, 0.002586886037128977
    890
    EP 890 simulation finished, start learning
    4.23
    EP 890 learning finished: 0.00528665091024294, 0.0025995560703068557
    891
    EP 891 simulation finished, start learning
    4.62
    EP 891 learning finished: 0.009822881140049589, 0.004072616487320009
    892
    EP 892 simulation finished, start learning
    4.4
    EP 892 learning finished: 0.016975114283187764, 0.005346570376810078
    893
    EP 893 simulation finished, start learning
    4.8
    EP 893 learning finished: 0.016940132532557328, 0.005093155602937216
    894
    EP 894 simulation finished, start learning
    4.33
    EP 894 learning finished: 0.026663944542512752, 0.006486864062021406
    895
    EP 895 simulation finished, start learning
    3.84
    EP 895 learning finished: 0.01685237078591939, 0.005154867635112008
    896
    EP 896 simulation finished, start learning
    4.31
    EP 896 learning finished: 0.009731217015602311, 0.003997903075877811
    897
    EP 897 simulation finished, start learning
    4.61
    EP 897 learning finished: 0.009721574510521691, 0.004117494251820138
    898
    EP 898 simulation finished, start learning
    4.22
    EP 898 learning finished: 0.004071998536460342, 0.002653449640076152
    899
    EP 899 simulation finished, start learning
    4.52
    EP 899 learning finished: 0.003484262173905961, 0.0023784553914509843
    900
    EP 900 simulation finished, start learning
    4.44
    EP 900 learning finished: 0.004061968335321353, 0.0025548949647612083
    901
    EP 901 simulation finished, start learning
    4.32
    EP 901 learning finished: 0.00971797898026594, 0.004292048363608932
    902
    EP 902 simulation finished, start learning
    4.41
    EP 902 learning finished: 0.016778334144259333, 0.0054040610924600925
    903
    EP 903 simulation finished, start learning
    4.45
    EP 903 learning finished: 0.026416766837456862, 0.00722679175969058
    904
    EP 904 simulation finished, start learning
    4.61
    EP 904 learning finished: 0.026335133160959145, 0.006972490830760236
    905
    EP 905 simulation finished, start learning
    4.01
    EP 905 learning finished: 0.016641909570260793, 0.0053511830760015755
    906
    EP 906 simulation finished, start learning
    4.02
    EP 906 learning finished: 0.016609113850115875, 0.005304571932525685
    907
    EP 907 simulation finished, start learning
    4.5
    EP 907 learning finished: 0.006986012350441939, 0.003317599866270745
    908
    EP 908 simulation finished, start learning
    4.71
    EP 908 learning finished: 0.006979497369542287, 0.0030815551078463934
    909
    EP 909 simulation finished, start learning
    4.32
    EP 909 learning finished: 0.006972520688499628, 0.003079039332752368
    910
    EP 910 simulation finished, start learning
    4.35
    EP 910 learning finished: 0.01653542580970324, 0.004830792776547928
    911
    EP 911 simulation finished, start learning
    4.44
    EP 911 learning finished: 0.016501465072582988, 0.005141226555528178
    912
    EP 912 simulation finished, start learning
    4.41
    EP 912 learning finished: 0.0164723303640244, 0.005186674936390763
    913
    EP 913 simulation finished, start learning
    4.21
    EP 913 learning finished: 0.025945595627433704, 0.006701453700169206
    914
    EP 914 simulation finished, start learning
    4.41
    EP 914 learning finished: 0.009496395880091768, 0.0035877482045378883
    915
    EP 915 simulation finished, start learning
    4.71
    EP 915 learning finished: 0.018966185785313172, 0.004884334804948172
    916
    EP 916 simulation finished, start learning
    4.41
    EP 916 learning finished: 0.01892937181257537, 0.004912572087561055
    917
    EP 917 simulation finished, start learning
    4.03
    EP 917 learning finished: 0.0094504112969922, 0.0035493684811000272
    918
    EP 918 simulation finished, start learning
    4.23
    EP 918 learning finished: 0.00944157758620534, 0.003932287893571029
    919
    EP 919 simulation finished, start learning
    4.31
    EP 919 learning finished: 0.0068451976275447635, 0.0038486983704775213
    920
    EP 920 simulation finished, start learning
    4.44
    EP 920 learning finished: 0.006840574208343497, 0.004031539163468878
    921
    EP 921 simulation finished, start learning
    4.23
    EP 921 learning finished: 0.006968585418285045, 0.004166923647739888
    922
    EP 922 simulation finished, start learning
    4.2
    EP 922 learning finished: 0.006828896000069307, 0.003944275888273434
    923
    EP 923 simulation finished, start learning
    4.02
    EP 923 learning finished: 0.006116408588745514, 0.002862947444635703
    924
    EP 924 simulation finished, start learning
    4.61
    EP 924 learning finished: 0.006497775991022081, 0.0028134635503024676
    925
    EP 925 simulation finished, start learning
    4.41
    EP 925 learning finished: 0.004729235035910051, 0.002458452207093842
    926
    EP 926 simulation finished, start learning
    4.34
    EP 926 learning finished: 0.004600936069870443, 0.002534161933764286
    927
    EP 927 simulation finished, start learning
    4.62
    EP 927 learning finished: 0.003384366674829664, 0.0022167707423506045
    928
    EP 928 simulation finished, start learning
    4.12
    EP 928 learning finished: 0.01623186931945142, 0.004926460132527548
    929
    EP 929 simulation finished, start learning
    3.54
    EP 929 learning finished: 0.01620148712314773, 0.0054746198912596356
    930
    EP 930 simulation finished, start learning
    4.7
    EP 930 learning finished: 0.01616991737569151, 0.005098165304630697
    931
    EP 931 simulation finished, start learning
    3.93
    EP 931 learning finished: 0.016138536018210914, 0.005325686524045636
    932
    EP 932 simulation finished, start learning
    4.53
    EP 932 learning finished: 0.0039371127630332126, 0.0027644646831189354
    933
    EP 933 simulation finished, start learning
    4.91
    EP 933 learning finished: 0.0030879852884098113, 0.002289834265064646
    934
    EP 934 simulation finished, start learning
    4.44
    EP 934 learning finished: 0.0041597932156420825, 0.0026541772811488804
    935
    EP 935 simulation finished, start learning
    3.44
    EP 935 learning finished: 0.004423012732139853, 0.002791498113820031
    936
    EP 936 simulation finished, start learning
    4.34
    EP 936 learning finished: 0.004682927075684518, 0.002721679359066087
    937
    EP 937 simulation finished, start learning
    4.42
    EP 937 learning finished: 0.004595146431555008, 0.002698109987074796
    938
    EP 938 simulation finished, start learning
    4.13
    EP 938 learning finished: 0.00466816665248615, 0.0028559784949043396
    939
    EP 939 simulation finished, start learning
    4.52
    EP 939 learning finished: 0.004126515271586584, 0.0025074563057323473
    940
    EP 940 simulation finished, start learning
    4.91
    EP 940 learning finished: 0.003321888381222493, 0.002363218546559409
    941
    EP 941 simulation finished, start learning
    4.45
    EP 941 learning finished: 0.009355889128194406, 0.004098052010760966
    942
    EP 942 simulation finished, start learning
    4.22
    EP 942 learning finished: 0.009348005653301072, 0.004004824812241736
    943
    EP 943 simulation finished, start learning
    4.51
    EP 943 learning finished: 0.00933979959205189, 0.0039414194070944335
    944
    EP 944 simulation finished, start learning
    4.31
    EP 944 learning finished: 0.009330843443329684, 0.0038633798170704856
    945
    EP 945 simulation finished, start learning
    4.53
    EP 945 learning finished: 0.0031504979074483686, 0.0023420855288615408
    946
    EP 946 simulation finished, start learning
    4.11
    EP 946 learning finished: 0.003331478580399505, 0.0020801585977636705
    947
    EP 947 simulation finished, start learning
    4.02
    EP 947 learning finished: 0.0038804670681287234, 0.0024158498180580543
    948
    EP 948 simulation finished, start learning
    4.31
    EP 948 learning finished: 0.004204684938123648, 0.0026116324414324765
    949
    EP 949 simulation finished, start learning
    4.71
    EP 949 learning finished: 0.0036729902751488908, 0.002279867129348171
    950
    EP 950 simulation finished, start learning
    4.35
    EP 950 learning finished: 0.004719268874119109, 0.0025224703137851634
    951
    EP 951 simulation finished, start learning
    4.0
    EP 951 learning finished: 0.009331564597097497, 0.003730922944221504
    952
    EP 952 simulation finished, start learning
    4.62
    EP 952 learning finished: 0.016073055361697897, 0.004901076609453862
    953
    EP 953 simulation finished, start learning
    4.31
    EP 953 learning finished: 0.01604228790523976, 0.0051415748747552685
    954
    EP 954 simulation finished, start learning
    4.34
    EP 954 learning finished: 0.016013060580274185, 0.005254336181219917
    955
    EP 955 simulation finished, start learning
    3.93
    EP 955 learning finished: 0.006717863775659083, 0.0039688017955364545
    956
    EP 956 simulation finished, start learning
    4.23
    EP 956 learning finished: 0.004037349511996524, 0.002751174862883232
    957
    EP 957 simulation finished, start learning
    4.1
    EP 957 learning finished: 0.004005469889456492, 0.002715496531340733
    958
    EP 958 simulation finished, start learning
    4.32
    EP 958 learning finished: 0.0032462825529151473, 0.0024147445810645727
    959
    EP 959 simulation finished, start learning
    4.14
    EP 959 learning finished: 0.003502038465562407, 0.0024143387355657353
    960
    EP 960 simulation finished, start learning
    4.53
    EP 960 learning finished: 0.00927115998020156, 0.0038988376137064997
    961
    EP 961 simulation finished, start learning
    3.92
    EP 961 learning finished: 0.009262725592996901, 0.004055959629219785
    962
    EP 962 simulation finished, start learning
    4.22
    EP 962 learning finished: 0.01595028664703335, 0.0054010697617859664
    963
    EP 963 simulation finished, start learning
    4.53
    EP 963 learning finished: 0.0159187082405019, 0.005428252161124784
    964
    EP 964 simulation finished, start learning
    4.53
    EP 964 learning finished: 0.006676772513582563, 0.004076994347584535
    965
    EP 965 simulation finished, start learning
    4.52
    EP 965 learning finished: 0.013336884179913966, 0.005142277450146319
    966
    EP 966 simulation finished, start learning
    4.02
    EP 966 learning finished: 0.006659551692067112, 0.003818710206206314
    967
    EP 967 simulation finished, start learning
    4.31
    EP 967 learning finished: 0.006653635988147499, 0.003541601345782297
    968
    EP 968 simulation finished, start learning
    4.62
    EP 968 learning finished: 0.006648729825048605, 0.0032626357542368347
    969
    EP 969 simulation finished, start learning
    3.93
    EP 969 learning finished: 0.003196574563666843, 0.002263988829116196
    970
    EP 970 simulation finished, start learning
    4.53
    EP 970 learning finished: 0.009179996232344223, 0.0037658638596867653
    971
    EP 971 simulation finished, start learning
    4.12
    EP 971 learning finished: 0.009171449608260218, 0.0038812422538505587
    972
    EP 972 simulation finished, start learning
    4.52
    EP 972 learning finished: 0.009163009346433881, 0.003904523355048539
    973
    EP 973 simulation finished, start learning
    4.06
    EP 973 learning finished: 0.01577077670298266, 0.005073832017332691
    974
    EP 974 simulation finished, start learning
    4.41
    EP 974 learning finished: 0.0066115909820751995, 0.003425351661260123
    975
    EP 975 simulation finished, start learning
    4.13
    EP 975 learning finished: 0.00660681765149107, 0.00347379116341943
    976
    EP 976 simulation finished, start learning
    4.63
    EP 976 learning finished: 0.006600548956086527, 0.003528684287298726
    977
    EP 977 simulation finished, start learning
    4.41
    EP 977 learning finished: 0.002696617311393368, 0.0019724918352797736
    978
    EP 978 simulation finished, start learning
    4.34
    EP 978 learning finished: 0.003510614403069212, 0.002202940125905902
    979
    EP 979 simulation finished, start learning
    4.44
    EP 979 learning finished: 0.0037894944337182324, 0.0023839790101819838
    980
    EP 980 simulation finished, start learning
    3.92
    EP 980 learning finished: 0.0033141508257941155, 0.00242648939318831
    981
    EP 981 simulation finished, start learning
    4.72
    EP 981 learning finished: 0.0035934365382530586, 0.0024268139350371123
    982
    EP 982 simulation finished, start learning
    4.62
    EP 982 learning finished: 0.009124886181989922, 0.003746372178734821
    983
    EP 983 simulation finished, start learning
    4.71
    EP 983 learning finished: 0.01570625942347803, 0.004467948705813167
    984
    EP 984 simulation finished, start learning
    4.02
    EP 984 learning finished: 0.015676351740790295, 0.004625023295547592
    985
    EP 985 simulation finished, start learning
    4.32
    EP 985 learning finished: 0.01564741545369941, 0.004666754743477479
    986
    EP 986 simulation finished, start learning
    4.53
    EP 986 learning finished: 0.006556967670647396, 0.003231643499124933
    987
    EP 987 simulation finished, start learning
    4.43
    EP 987 learning finished: 0.004759605268162659, 0.002462436026380743
    988
    EP 988 simulation finished, start learning
    4.31
    EP 988 learning finished: 0.003377219423192912, 0.00233763674505146
    989
    EP 989 simulation finished, start learning
    4.23
    EP 989 learning finished: 0.006553807563722813, 0.003570249041461082
    990
    EP 990 simulation finished, start learning
    4.04
    EP 990 learning finished: 0.006548579653181008, 0.003920032234628816
    991
    EP 991 simulation finished, start learning
    4.71
    EP 991 learning finished: 0.006543769405713107, 0.0035648549083896106
    992
    EP 992 simulation finished, start learning
    4.05
    EP 992 learning finished: 0.006538503484364355, 0.003693751122289276
    993
    EP 993 simulation finished, start learning
    4.12
    EP 993 learning finished: 0.009036463468184763, 0.0039937538651043285
    994
    EP 994 simulation finished, start learning
    4.21
    EP 994 learning finished: 0.01804875237157333, 0.005057561846590748
    995
    EP 995 simulation finished, start learning
    3.42
    EP 995 learning finished: 0.027006970366375782, 0.00681604967710155
    996
    EP 996 simulation finished, start learning
    4.82
    EP 996 learning finished: 0.026928001295331683, 0.006471009836245638
    997
    EP 997 simulation finished, start learning
    4.71
    EP 997 learning finished: 0.017908426648929776, 0.004843681214050359
    998
    EP 998 simulation finished, start learning
    4.71
    EP 998 learning finished: 0.00894164264721553, 0.0034420370943581227
    999
    EP 999 simulation finished, start learning
    4.14
    EP 999 learning finished: 0.0026765943140034665, 0.001953295311559818



```python
v_opt, pi_opt= q_opt.q.yield_v_pi()
```


```python
v_opt
```




    0    41.316134
    1    45.395789
    2    50.515048
    dtype: float64




```python
pi_opt
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
for s in state_space:
    for a in control_space:
        print(f'state: {s}, action_buffer: {[a]}, action to take: {get_opt_act(q_opt.q, model, s, [a])}')
        
pi_opt = lambda x: get_opt_act(q_opt.q, model, x[0], [x[1]])
```

    state: 0, action_buffer: [-1], action to take: 1
    state: 0, action_buffer: [1], action to take: 1
    state: 1, action_buffer: [-1], action to take: 1
    state: 1, action_buffer: [1], action to take: -1
    state: 2, action_buffer: [-1], action to take: 1
    state: 2, action_buffer: [1], action to take: -1



```python
q_opt.q
```




                0          1          2
    -1  25.622767  36.756796  50.515048
     1  41.316134  45.395789  35.990414




```python
def eval_pi_model(env, q, model, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = get_opt_act(q, model, obs, action_buffer)
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
#     return np.mean(ret) ,ret
    return np.mean(r_ls), r_ls.mean(axis = 1)
```


```python
val, v_samples_lin = eval_pi_model(env, q_opt.q, model, 0.95)
val
```




    1.5354082575596955




```python
exp2_opt = v_samples_lin
```


```python
from matplotlib.pyplot import figure

figure(figsize=(8, 6), dpi=100)
sns.distplot(exp2_opt, label = 'Opt. Action Selection')
sns.distplot(exp2_model, label = 'model')
sns.distplot(exp2_random, label = 'random')
sns.distplot(exp1_opt, label = 'Augmented MDP')
plt.xlabel("average reward")
plt.ylabel("freq.")
plt.legend()
# plt.set_size_inches(18.5, 10.5)
plt.savefig('exp2.png', dpi=100)
```


```python
plt.savefig('exp1.png')
```


    <Figure size 432x288 with 0 Axes>



```python
# For a given policy function, we solve the system A^{\pi}x=b^{\pi} to get the optimal value function

def linear_system(pi, trans_prob, reward, state_space, gamma):
    n = len(state_space)
    A = np.zeros(shape = (n,n))
    b = np.zeros(n)
    for i, s in enumerate(state_space):
        for j, next_s in enumerate(state_space):
            A[i,j] = trans_prob(next_s, s, pi(s))
            
        b[i] = -reward(s, pi(s))
        print(f'policy({s}): {pi(s)}')
    A = gamma*A-np.eye(n)
    return A, b
        
```


```python
from numpy.linalg import solve
```


```python
# policy of opt action selection
A, b = linear_system(pi_opt, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): 1
    policy((1, 1)): -1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.  ,  0.76, -1.  ,  0.  ,  0.  ,  0.19],
            [ 0.57,  0.  ,  0.  , -1.  ,  0.38,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([27.27312864, 29.76118804, 27.71489735, 30.27493478, 38.76118804,
           26.82312864])




```python
pi_AMDP_func = lambda x:pi_AMDP[x]
```


```python
# policy of AMDP

A, b = linear_system(pi_AMDP_func, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): -1
    policy((1, 1)): -1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.76,  0.  , -1.  ,  0.  ,  0.19,  0.  ],
            [ 0.57,  0.  ,  0.  , -1.  ,  0.38,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([27.27312864, 29.76118804, 28.09220349, 30.27493478, 38.76118804,
           26.82312864])




```python
# policy of planning based mathod
```


```python
A, b = linear_system(pi_model, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): 1
    policy((1, 1)): 1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.  ,  0.76, -1.  ,  0.  ,  0.  ,  0.19],
            [ 0.  ,  0.57,  0.  , -1.  ,  0.  ,  0.38],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([ 2.91797923,  4.12418866,  3.60329944,  3.28861965, 13.12418866,
            2.46797923])




```python

```


```python

```


```python

```


```python

```


```python
pi_d
```




    (0, -1)    1
    (0, 1)     1
    (1, -1)   -1
    (1, 1)    -1
    (2, -1)    1
    (2, 1)    -1
    dtype: int64




```python
def eval_pi_model(env, pi, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = pi[(obs, action_buffer[0])]
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
#     return np.mean(ret) ,ret
    return np.mean(r_ls), r_ls.mean(axis = 1)
```


```python
val, v_samples_lin = eval_pi_model(env, pi_d, 0.95)
val
```




    1.5329450309428467




```python

```


```python

```


```python

```


```python

```


```python
# Q-learning using LEARNING-BASED model, async reward collection and optimal action selection



class Q_learning:
    def __init__(self, q, env, delay_step, model, model_wrapper, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(400)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        self.model_wrapper = model_wrapper
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            model = self.model_wrapper(self.model)
            for _ in range(100):
                
                
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (get_opt_act(q, model, obs, self.action_memory), False)
                last_action = self.action_memory.pop(0)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.update_model(self.pool)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_model(self, pool):
        tmp = []
        for ele in pool:
            tmp.append(ele)
    
        x_train = np.array(tmp)[:, [0,2]]
        y_train = np.array(tmp)[:, 1]
        self.model.fit(x_train, y_train, batch_size=64, epochs=3, validation_split=0.2)
        
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q

def get_opt_act(q,model, x, acts):
    act_idx = np.argmax([np.sum([q[s, u]*model(s, x, acts) for s in q.state_space]) for u in q.control_space])
    return q.control_space[act_idx]
```


```python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
```


```python
inputs =  keras.Input(shape=(2,))

layer1 = layers.Dense(64, activation="relu", name="layer1")(inputs)
layer1 = layers.Dropout(0.5)(layer1)
layer2 = layers.Dense(64, activation="relu", name="layer2")(layer1)
layer2 = layers.Dropout(0.5)(layer2)
outputs = layers.Dense(3, activation="softmax", name="output_layer")(layer2)

model = keras.Model(inputs = inputs, outputs = outputs)
model.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.RMSprop(),
    metrics=["accuracy"],
)


model.summary()
```

    Model: "model"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    input_1 (InputLayer)         [(None, 2)]               0         
    _________________________________________________________________
    layer1 (Dense)               (None, 64)                192       
    _________________________________________________________________
    dropout (Dropout)            (None, 64)                0         
    _________________________________________________________________
    layer2 (Dense)               (None, 64)                4160      
    _________________________________________________________________
    dropout_1 (Dropout)          (None, 64)                0         
    _________________________________________________________________
    output_layer (Dense)         (None, 3)                 195       
    =================================================================
    Total params: 4,547
    Trainable params: 4,547
    Non-trainable params: 0
    _________________________________________________________________



```python
tmp = []
for ele in q_learn.pool:
    tmp.append(ele)
    
x_train = np.array(tmp)[:, [0,2]]
y_train = np.array(tmp)[:, 1]
def one_hot_encode(y):
    y = y.astype(int)
    elements = set(y)
    n = len(elements)
    ret = np.zeros((len(y), n))
    ret[np.arange(y.size),y] = 1
    return ret
# y_train = one_hot_encode(y_train)
print(x_train.shape, y_train.shape)
```

    (400, 2) (400,)



```python
model.predict([[1, 1]])[0,0]
```




    0.28935617




```python
history = model.fit(x_train, y_train, batch_size=64, epochs=10, validation_split=0.2)
```

    Train on 320 samples, validate on 80 samples
    Epoch 1/10
    320/320 [==============================] - 0s 288us/sample - loss: 0.7200 - accuracy: 0.8156 - val_loss: 0.7028 - val_accuracy: 0.8000
    Epoch 2/10
    320/320 [==============================] - 0s 196us/sample - loss: 0.7178 - accuracy: 0.8250 - val_loss: 0.7029 - val_accuracy: 0.8000
    Epoch 3/10
    320/320 [==============================] - 0s 193us/sample - loss: 0.7206 - accuracy: 0.8156 - val_loss: 0.7034 - val_accuracy: 0.8000
    Epoch 4/10
    320/320 [==============================] - 0s 189us/sample - loss: 0.7223 - accuracy: 0.7937 - val_loss: 0.7030 - val_accuracy: 0.8000
    Epoch 5/10
    320/320 [==============================] - 0s 194us/sample - loss: 0.7185 - accuracy: 0.8156 - val_loss: 0.7026 - val_accuracy: 0.8000
    Epoch 6/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7228 - accuracy: 0.8094 - val_loss: 0.7036 - val_accuracy: 0.8000
    Epoch 7/10
    320/320 [==============================] - 0s 195us/sample - loss: 0.7166 - accuracy: 0.8094 - val_loss: 0.7030 - val_accuracy: 0.8000
    Epoch 8/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7212 - accuracy: 0.8156 - val_loss: 0.7024 - val_accuracy: 0.8000
    Epoch 9/10
    320/320 [==============================] - 0s 193us/sample - loss: 0.7305 - accuracy: 0.7719 - val_loss: 0.7032 - val_accuracy: 0.8000
    Epoch 10/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7183 - accuracy: 0.8188 - val_loss: 0.7020 - val_accuracy: 0.8000



```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
# lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
def model_wrapper(model):
    def m(s, x, acts):
        inputs = np.array([[x, acts[0]]])
        return model.predict(inputs)[0, int(s)]
    return m
lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_learn = Q_learning(q, env, 1, model, model_wrapper, MAX_ITER = 1000)
learned_q = q_learn.train(lr_fn)
```


```python
m = model_wrapper(model)
m(0, 1, [1])
```




    0.35112822




```python
v_l, pi_l= q_learn.q.yield_v_pi()
```


```python
v_l
```




    0    41.123725
    1    42.282631
    2    50.209974
    dtype: float64




```python
pi_l
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
def eval_pi_model(env, q, model, model_wrapper, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    model = model_wrapper(model)
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = get_opt_act(q, model, obs, action_buffer)
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
    return np.mean(ret) ,ret
```


```python
val, v_samples_lin = eval_pi_model(env, q_learn.q, model, model_wrapper, 0.95)
val
```


```python
sns.distplot(v_samples_lin, label = 'model')
plt.legend()
```


```python

```


```python

```


```python

```


```python
df = np.arange(0, 100).reshape(25, -1)
df
```




    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11],
           [12, 13, 14, 15],
           [16, 17, 18, 19],
           [20, 21, 22, 23],
           [24, 25, 26, 27],
           [28, 29, 30, 31],
           [32, 33, 34, 35],
           [36, 37, 38, 39],
           [40, 41, 42, 43],
           [44, 45, 46, 47],
           [48, 49, 50, 51],
           [52, 53, 54, 55],
           [56, 57, 58, 59],
           [60, 61, 62, 63],
           [64, 65, 66, 67],
           [68, 69, 70, 71],
           [72, 73, 74, 75],
           [76, 77, 78, 79],
           [80, 81, 82, 83],
           [84, 85, 86, 87],
           [88, 89, 90, 91],
           [92, 93, 94, 95],
           [96, 97, 98, 99]])




```python
# T = 3
T = 3
[N, D] = df.shape
dataX = np.zeros((N - T + 1, T, D))
for i in range(T, N + 1):
    dataX[i - T] = df[i - T:i, :]
    
dataX

# unfolded LSTM
```




    array([[[ 0.,  1.,  2.,  3.],
            [ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.]],
    
           [[ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.],
            [12., 13., 14., 15.]],
    
           [[ 8.,  9., 10., 11.],
            [12., 13., 14., 15.],
            [16., 17., 18., 19.]],
    
           [[12., 13., 14., 15.],
            [16., 17., 18., 19.],
            [20., 21., 22., 23.]],
    
           [[16., 17., 18., 19.],
            [20., 21., 22., 23.],
            [24., 25., 26., 27.]],
    
           [[20., 21., 22., 23.],
            [24., 25., 26., 27.],
            [28., 29., 30., 31.]],
    
           [[24., 25., 26., 27.],
            [28., 29., 30., 31.],
            [32., 33., 34., 35.]],
    
           [[28., 29., 30., 31.],
            [32., 33., 34., 35.],
            [36., 37., 38., 39.]],
    
           [[32., 33., 34., 35.],
            [36., 37., 38., 39.],
            [40., 41., 42., 43.]],
    
           [[36., 37., 38., 39.],
            [40., 41., 42., 43.],
            [44., 45., 46., 47.]],
    
           [[40., 41., 42., 43.],
            [44., 45., 46., 47.],
            [48., 49., 50., 51.]],
    
           [[44., 45., 46., 47.],
            [48., 49., 50., 51.],
            [52., 53., 54., 55.]],
    
           [[48., 49., 50., 51.],
            [52., 53., 54., 55.],
            [56., 57., 58., 59.]],
    
           [[52., 53., 54., 55.],
            [56., 57., 58., 59.],
            [60., 61., 62., 63.]],
    
           [[56., 57., 58., 59.],
            [60., 61., 62., 63.],
            [64., 65., 66., 67.]],
    
           [[60., 61., 62., 63.],
            [64., 65., 66., 67.],
            [68., 69., 70., 71.]],
    
           [[64., 65., 66., 67.],
            [68., 69., 70., 71.],
            [72., 73., 74., 75.]],
    
           [[68., 69., 70., 71.],
            [72., 73., 74., 75.],
            [76., 77., 78., 79.]],
    
           [[72., 73., 74., 75.],
            [76., 77., 78., 79.],
            [80., 81., 82., 83.]],
    
           [[76., 77., 78., 79.],
            [80., 81., 82., 83.],
            [84., 85., 86., 87.]],
    
           [[80., 81., 82., 83.],
            [84., 85., 86., 87.],
            [88., 89., 90., 91.]],
    
           [[84., 85., 86., 87.],
            [88., 89., 90., 91.],
            [92., 93., 94., 95.]],
    
           [[88., 89., 90., 91.],
            [92., 93., 94., 95.],
            [96., 97., 98., 99.]]])




```python
dataX.shape
```




    (23, 3, 4)




```python
[np.random.choice(range(24), replace = True) for _ in range(10)]
```




    [2, 6, 23, 17, 22, 11, 1, 13, 22, 14]




```python
dataX.reshape(dataX.shape + (1,)).shape
```




    (23, 3, 4, 1)




```python
time = range(1,6)
disc = np.array([np.exp(-0.05*t) for t in time])
disc
```




    array([0.95122942, 0.90483742, 0.86070798, 0.81873075, 0.77880078])




```python
def A(lamb):
    return 0.016*(np.array([np.exp(-n*lamb) for n in time])*disc).sum()

def B(lamb):
    return 0.5*0.016*C(lamb)

def C(lamb):
    return (disc*np.array([np.exp(-(n-1)*lamb)-np.exp(-n*lamb) for n in time])).sum()
```


```python
lamb = 0.0154
A(lamb)+B(lamb), C(lamb)
```




    (0.06654174415562401, 0.06404516300888626)




```python

```

0ff5c693c3c4916b0d7e83e55d560400f062a1a9
ipynb
Jupyter Notebook
simplyabcTask-Copy2.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
simplyabcTask-Copy2.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
simplyabcTask-Copy2.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
Qwen/Qwen-72B
1. YES
2. YES