###### Formulation:
* Stages: $i = 1,2,3,\cdots$.
* State Space: $\mathcal{X} = \{a,b,c\}$.
* Control Space: $\mathcal{U} = \{-1,1\}$
* Feasible control mapping $\mathcal{X}\rightarrow\mathcal{U}: \mathcal{U}(x)=\{-1,1\}$
* Transition mapping, given $x_i, u_i$, $\mathcal{X}\rightarrow [0,1]$, for $i\in \{1,2,\cdots\}$. 
$P(j\mid i, u=-1)$

| i\\j | a   | b   | c   
| ---  | --- | --- | --- 
| a    | 1.0 | 0.0 | 0.0 
| b    | 0.8| 0.0 | 0.2
| c    | 0.0 | 1.0 | 0.0 

$P(j\mid i, u=1)$

| i\\j | a   | b   | c   
| ---  | --- | --- | --- 
| a    | 0.0 | 1.0 | 0.0 
| b    | 0.6| 0.0 | 0.4
| c    | 0.0 | 0.0 | 1.0 


* Reward function $\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R}$: 
$$r(a, -1)=-1; \quad r(a, 1)=1;\quad r(c, -1)=10;\quad r(c,1)=-10 $$
For other state and action combinations $(x, u)\in (\mathcal{S}, \mathcal{A})$, $r(x, u)=0$.
* Let $v^*(x)$ be the optimal value function for state $x$, so it satisfies the following equation:

\begin{align}
    v^*(x_i) = &\max_{u\in \mathcal{U}(x)}\{r(x_i, u)+\alpha\mathbb{E}[v^*(x_{i+1})]\}\\
\end{align}

And it can be solved by value iteration:

\begin{align}
    v^{k+1}(x_i) = \mathcal{D}(v)(x_i) = &\max_{u\in \mathcal{U}(x)}\{r(x_i, u)+\alpha\mathbb{E}[v^k(x_{i+1})]\}\\
\end{align}
with arbitrary $v^0$, since $\mathcal{D}$ is a contraction mapping.


```python
from itertools import product
import numpy as np
import pandas as pd
from copy import deepcopy
import random
import matplotlib.pyplot as plt
import seaborn as sns
```


```python
pd.DataFrame({0: [1., 0.8, 0.], 1:[0.,0.,1.], 2:[0.,0.2, 0.]})
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.8</td>
      <td>0.0</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
state_space = [0,1,2]
control_space = [-1,1]
alpha = 0.95
tun1 = pd.DataFrame({0: [1., 0.8, 0.], 1:[0.,0.,1.], 2:[0.,0.2, 0.]})
tu1 = pd.DataFrame({0: [0., 0.6, 0.], 1:[1.,0.,0.], 2:[0.,0.4, 1.]})
def trans_prob(next_x, x, u):
    if u == 1:
        t = tu1
    elif u == -1:
        t = tun1
    else:
        raise ValueError
    return t.loc[x, next_x]

def reward(x, u):
    if x == 0 and u == 1:
        return 1.
    elif x == 2 and u == -1:
        return 10.
    elif x == 2 and u == 1:
        return -10.
    elif x == 0 and u == -1:
        return -1
    else:
        return 0.
```


```python
tu1
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.6</td>
      <td>0.0</td>
      <td>0.4</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
tun1
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.8</td>
      <td>0.0</td>
      <td>0.2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
v = pd.Series([0.0 for _ in state_space], index = state_space)
pi = pd.Series([0 for _ in state_space], index = state_space)
```


```python
def value_iteration(state_space, control_space, trans_prob, reward, alpha, v, pi, MAX_ITER = 1000000, threshold = 1e-6):
    for _ in range(MAX_ITER):
        next_v = deepcopy(v)
        for x in state_space:
            tmp = [reward(x, u)+alpha*
                   np.sum([v[next_x]*trans_prob(next_x, x, u) for next_x in state_space]) 
                   for u in control_space]
            next_v[x]= np.max(tmp)
            pi[x]= control_space[np.argmax(tmp)]
        if abs(next_v-v).values.max()<1e-6:
            break
        else:
            v = next_v
            
    return v, pi
```


```python
v, pi = value_iteration(state_space, control_space, trans_prob, reward, 0.95, v, pi)
```


```python
v
```




    0    43.579477
    1    44.820504
    2    52.579477
    dtype: float64




```python
pi
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
list(product(state_space, control_space))
```




    [(0, -1), (0, 1), (1, -1), (1, 1), (2, -1), (2, 1)]




```python
# Augmented MDP
state_space_d = list(product(state_space, control_space))
control_space_d = [-1,1]
alpha = 0.95
tun1 = pd.DataFrame({0: [1., 0.8, 0.], 1:[0.,0.,1.], 2:[0.,0.2, 0.]})
tu1 = pd.DataFrame({0: [0., 0.6, 0.], 1:[1.,0.,0.], 2:[0.,0.4, 1.]})
def trans_prob_d(next_i, i, u):
    if next_i[1]==u:
        return trans_prob(next_i[0], i[0], i[1])
    else:
        return 0.0

def reward_d(x, u):
    return reward(x[0], x[1])
```


```python
v_d = pd.Series([0.0 for _ in state_space_d], index = state_space_d)
pi_d = pd.Series([0 for _ in state_space_d], index = state_space_d)
v_d
```




    (0, -1)    0.0
    (0, 1)     0.0
    (1, -1)    0.0
    (1, 1)     0.0
    (2, -1)    0.0
    (2, 1)     0.0
    dtype: float64




```python
v_AMDP, pi_AMDP = value_iteration(state_space_d, control_space_d, trans_prob_d, reward_d, 0.95, v_d, pi_d)
```


```python
v_AMDP
```




    (0, -1)    27.273109
    (0, 1)     29.761168
    (1, -1)    28.092184
    (1, 1)     30.274915
    (2, -1)    38.761168
    (2, 1)     26.823109
    dtype: float64




```python
pi_AMDP
```




    (0, -1)    1
    (0, 1)     1
    (1, -1)   -1
    (1, 1)    -1
    (2, -1)    1
    (2, 1)    -1
    dtype: int64




```python
# Q_learning env
class Env:
    def __init__(self):
#         self.gamma = gamma
        self.obs = 0
        self.control_space = (-1, 1)
        self.state_space = tuple([0,1,2])
        self.state_dim = 1
        self.action_dim = 1
        
    def step(self, action, current_obs = None):
        
        obs = self.obs if current_obs is None else current_obs
        self.obs = self._next_state(obs, action)
        r = reward(obs, action)
#         r = reward((q,p), action)
        return self.obs, r, False, {}
    
    def reset(self, random=False):
        self.obs = 0 if not random else np.random.choice([0,1,2])
        return self.obs
    
    @staticmethod
    def _next_state(obs, action):
        p = tun1.loc[obs, :] if action == -1 else tu1.loc[obs, :]
        return np.random.choice([0,1,2], p = p)
      
```


```python
class Q_table:
    def __init__(self, q_data=None, random = False):
        self.control_space = (-1,1)
        self.state_space = tuple([0,1,2])
        if q_data is None:
            if not random:
                self.data = pd.DataFrame({x:[0.0]*len(self.control_space) for x in self.state_space},
                                         index = self.control_space) 
            else:
                self.data = pd.DataFrame({x:np.random.randn(len(self.control_space)) for x in self.state_space}, 
                                         index = self.control_space) 
        else:
#             assert isinstance(q, Q_table)
            self.data = q_data
        
    def __getitem__(self, key):
        x, u = key
        return self.data.loc[u,x]
    
    def __setitem__(self, key, item):
        x, u = key
        self.data.loc[u,x] = item
    
    def max_value_act(self, x):
        tmp = [self.data.loc[k,x] for k in self.control_space]
        return np.max(tmp), self.control_space[np.argmax(tmp)]
    
    def difference(self, q):
        assert isinstance(q, Q_table)
        return Q_table(self.data-q.data)
        
    def L_infty(self):
        return np.max(np.abs(self.data).values)
    
    def L_1(self):
        return np.mean(np.abs(self.data).values)
    
    def yield_v_pi(self):
        v = pd.Series([0.0 for _ in self.state_space], index = self.state_space)
        pi = pd.Series([0 for _ in self.state_space], index = self.state_space)
        for i in self.state_space:
            v[i], pi[i] = self.max_value_act(i)     
        return v, pi
        
    def __repr__(self):

        return self.data.__repr__()
    
class Pool:
    def __init__(self, length):
        self.ls = [None]*length
        self.len = length
        self.idx = 0
        self.isFull = False
        
    def append(self, item):
        self.ls[self.idx] = item
        self.idx = (self.idx + 1) % self.len
        if not self.isFull:
            if self.idx + 1 == self.len:
                self.isFull = True
                
    def shuffle(self):
        cp = deepcopy(self.ls)
        random.shuffle(cp)
        return cp
        
    def __getitem__(self, key):
        idx = (self.idx+key) % self.len
        return self.ls[idx]
    
    def __iter__(self):
        return iter(self.ls[self.idx:]+self.ls[:self.idx])
    
    def __repr__(self):
        return str(self.ls[self.idx:]+self.ls[:self.idx])
```


```python
# Q-learning with pool


class Q_learning:
    def __init__(self, q, env, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(40)
        self.qs = []
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random=True)
            r_ls = []
            for _ in range(10):
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (q.max_value_act(obs)[1], False)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q
```


```python
# Q-learning with epsilon-greedy policy,  learning rate = 1/t
q = Q_table(random = False)
env = Env()
q_one_over_t = Q_learning(q, env, MAX_ITER = 200)
lr_fn = lambda i, s, a: 1/(i+1)
lr_fn = lambda i, s, a: 0.1
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
learned_q = q_one_over_t.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    -0.7
    EP 3 learning finished: 10.075208333333334, 3.657494399641577
    4
    EP 4 simulation finished, start learning
    1.4
    EP 4 learning finished: 0.5888457397017046, 0.2628739122947419
    5
    EP 5 simulation finished, start learning
    2.2
    EP 5 learning finished: 4.93179798089833, 1.2091979361404614
    6
    EP 6 simulation finished, start learning
    3.2
    EP 6 learning finished: 1.7711711697075305, 0.7126685438534327
    7
    EP 7 simulation finished, start learning
    3.2
    EP 7 learning finished: 1.2173679361711764, 0.6154542751447991
    8
    EP 8 simulation finished, start learning
    2.3
    EP 8 learning finished: 0.9322396052206621, 0.46885205143375125
    9
    EP 9 simulation finished, start learning
    3.2
    EP 9 learning finished: 12.562845115828447, 2.4978079963795428
    10
    EP 10 simulation finished, start learning
    3.2
    EP 10 learning finished: 1.7004185265273861, 0.8078899625765278
    11
    EP 11 simulation finished, start learning
    1.4
    EP 11 learning finished: 0.7928816699393364, 0.3759259958096119
    12
    EP 12 simulation finished, start learning
    4.1
    EP 12 learning finished: 0.5419754366806142, 0.2588643980500742
    13
    EP 13 simulation finished, start learning
    2.3
    EP 13 learning finished: 0.4627613816039897, 0.1723141769256348
    14
    EP 14 simulation finished, start learning
    1.4
    EP 14 learning finished: 0.43859220889082984, 0.1314851822193268
    15
    EP 15 simulation finished, start learning
    1.4
    EP 15 learning finished: 0.3583010940492457, 0.12456153467633262
    16
    EP 16 simulation finished, start learning
    3.2
    EP 16 learning finished: 0.30821110783339556, 0.08506287772911492
    17
    EP 17 simulation finished, start learning
    2.3
    EP 17 learning finished: 0.2651296476545735, 0.09223488161562828
    18
    EP 18 simulation finished, start learning
    2.3
    EP 18 learning finished: 0.20979044739698516, 0.0839626470164679
    19
    EP 19 simulation finished, start learning
    4.1
    EP 19 learning finished: 0.22019658787844065, 0.09443227693120182
    20
    EP 20 simulation finished, start learning
    3.2
    EP 20 learning finished: 0.22451037418059094, 0.09397861640166827
    21
    EP 21 simulation finished, start learning
    1.4
    EP 21 learning finished: 0.17296410561261055, 0.07279772519811474
    22
    EP 22 simulation finished, start learning
    3.2
    EP 22 learning finished: 0.17593187178270497, 0.07448878791442255
    23
    EP 23 simulation finished, start learning
    3.2
    EP 23 learning finished: 0.14765030515339816, 0.06836453005307137
    24
    EP 24 simulation finished, start learning
    0.5
    EP 24 learning finished: 0.1584087934911107, 0.047755851097497214
    25
    EP 25 simulation finished, start learning
    2.3
    EP 25 learning finished: 0.130640134905466, 0.050038187202076014
    26
    EP 26 simulation finished, start learning
    4.1
    EP 26 learning finished: 0.11075488618860163, 0.0526175062244952
    27
    EP 27 simulation finished, start learning
    3.2
    EP 27 learning finished: 0.10379367981628995, 0.04999609259273793
    28
    EP 28 simulation finished, start learning
    2.3
    EP 28 learning finished: 0.117590764260747, 0.05649880683974814
    29
    EP 29 simulation finished, start learning
    1.3
    EP 29 learning finished: 0.11118981542798329, 0.0741744130549735
    30
    EP 30 simulation finished, start learning
    2.3
    EP 30 learning finished: 0.10884584448621984, 0.07145418954134107
    31
    EP 31 simulation finished, start learning
    2.3
    EP 31 learning finished: 0.10705733284636099, 0.06412281519960149
    32
    EP 32 simulation finished, start learning
    1.4
    EP 32 learning finished: 0.10626851746269317, 0.07179950713262957
    33
    EP 33 simulation finished, start learning
    1.4
    EP 33 learning finished: 0.10065696012940784, 0.05000597090639788
    34
    EP 34 simulation finished, start learning
    4.1
    EP 34 learning finished: 0.10980156272155384, 0.05221264729407249
    35
    EP 35 simulation finished, start learning
    2.3
    EP 35 learning finished: 0.08641620247835746, 0.0473869681422916
    36
    EP 36 simulation finished, start learning
    1.4
    EP 36 learning finished: 0.10189320009740044, 0.04950326879996799
    37
    EP 37 simulation finished, start learning
    1.4
    EP 37 learning finished: 0.0999747495504879, 0.042333275742256316
    38
    EP 38 simulation finished, start learning
    2.3
    EP 38 learning finished: 0.09827451511300467, 0.04291268688580926
    39
    EP 39 simulation finished, start learning
    1.3
    EP 39 learning finished: 1.7137919381555409, 0.32774594869206397
    40
    EP 40 simulation finished, start learning
    3.2
    EP 40 learning finished: 1.231132888217729, 0.23810039389513046
    41
    EP 41 simulation finished, start learning
    2.3
    EP 41 learning finished: 0.9331909765264135, 0.1916027299456492
    42
    EP 42 simulation finished, start learning
    1.2
    EP 42 learning finished: 0.7299796882331075, 0.18658007814672803
    43
    EP 43 simulation finished, start learning
    1.2
    EP 43 learning finished: 0.21710661280475385, 0.07930667287804176
    44
    EP 44 simulation finished, start learning
    0.5
    EP 44 learning finished: 0.20893224852736714, 0.06742100338859093
    45
    EP 45 simulation finished, start learning
    3.2
    EP 45 learning finished: 0.20055885393834572, 0.06196933696951029
    46
    EP 46 simulation finished, start learning
    1.2
    EP 46 learning finished: 0.1931595518686775, 0.04887076330870821
    47
    EP 47 simulation finished, start learning
    1.4
    EP 47 learning finished: 0.09380625669732723, 0.03233350984283471
    48
    EP 48 simulation finished, start learning
    1.4
    EP 48 learning finished: 0.09200778400805043, 0.03072424387069217
    49
    EP 49 simulation finished, start learning
    3.2
    EP 49 learning finished: 0.0905109525846215, 0.02812021154321598
    50
    EP 50 simulation finished, start learning
    2.3
    EP 50 learning finished: 0.03758128742957112, 0.017268769938505162
    51
    EP 51 simulation finished, start learning
    3.2
    EP 51 learning finished: 0.051072606177807955, 0.021223447156839843
    52
    EP 52 simulation finished, start learning
    1.4
    EP 52 learning finished: 0.04917011036045338, 0.02082884896591608
    53
    EP 53 simulation finished, start learning
    2.2
    EP 53 learning finished: 0.09024466265497799, 0.03542455892898799
    54
    EP 54 simulation finished, start learning
    0.5
    EP 54 learning finished: 0.08849274883678016, 0.030594378907675763
    55
    EP 55 simulation finished, start learning
    2.3
    EP 55 learning finished: 0.08735796741503976, 0.034854268452226576
    56
    EP 56 simulation finished, start learning
    1.3
    EP 56 learning finished: 0.17015747435772877, 0.052070099869597986
    57
    EP 57 simulation finished, start learning
    4.1
    EP 57 learning finished: 0.08259890478074894, 0.03880411017313149
    58
    EP 58 simulation finished, start learning
    2.3
    EP 58 learning finished: 0.08139247233719482, 0.04356711382268741
    59
    EP 59 simulation finished, start learning
    2.3
    EP 59 learning finished: 0.07997005973891813, 0.036482446365049714
    60
    EP 60 simulation finished, start learning
    0.5
    EP 60 learning finished: 0.03431466962861762, 0.020489839035996955
    61
    EP 61 simulation finished, start learning
    3.2
    EP 61 learning finished: 0.09187053557079494, 0.027030440658597204
    62
    EP 62 simulation finished, start learning
    2.3
    EP 62 learning finished: 0.13225868672628316, 0.03208710774881821
    63
    EP 63 simulation finished, start learning
    1.4
    EP 63 learning finished: 0.12855174914137635, 0.03134565360601016
    64
    EP 64 simulation finished, start learning
    1.4
    EP 64 learning finished: 0.11146903680317699, 0.027788446617214763
    65
    EP 65 simulation finished, start learning
    1.4
    EP 65 learning finished: 0.0550539542145394, 0.018748132690470893
    66
    EP 66 simulation finished, start learning
    1.4
    EP 66 learning finished: 0.030643690442174076, 0.01127505584879085
    67
    EP 67 simulation finished, start learning
    1.4
    EP 67 learning finished: 0.029464265866884887, 0.01105015543015971
    68
    EP 68 simulation finished, start learning
    3.2
    EP 68 learning finished: 0.025080807141264927, 0.014092509067588743
    69
    EP 69 simulation finished, start learning
    0.2
    EP 69 learning finished: 0.6616843820392897, 0.13637047170218453
    70
    EP 70 simulation finished, start learning
    1.4
    EP 70 learning finished: 0.5426071006308613, 0.11367587711086775
    71
    EP 71 simulation finished, start learning
    2.3
    EP 71 learning finished: 0.45550585162283674, 0.10032623579844786
    72
    EP 72 simulation finished, start learning
    2.3
    EP 72 learning finished: 0.38668346899118067, 0.08815177881937537
    73
    EP 73 simulation finished, start learning
    3.2
    EP 73 learning finished: 0.035016037794257215, 0.013644379088026959
    74
    EP 74 simulation finished, start learning
    4.1
    EP 74 learning finished: 0.05508064947301605, 0.01775492923272015
    75
    EP 75 simulation finished, start learning
    1.4
    EP 75 learning finished: 0.04673213362036854, 0.01615158016175015
    76
    EP 76 simulation finished, start learning
    2.3
    EP 76 learning finished: 0.03269866924731346, 0.013610516502376177
    77
    EP 77 simulation finished, start learning
    0.5
    EP 77 learning finished: 0.02237496616725565, 0.009248136979989793
    78
    EP 78 simulation finished, start learning
    0.5
    EP 78 learning finished: 0.02734358525499303, 0.008572676458389475
    79
    EP 79 simulation finished, start learning
    1.4
    EP 79 learning finished: 0.025954558096765723, 0.008178826130419795
    80
    EP 80 simulation finished, start learning
    1.4
    EP 80 learning finished: 0.026269981782254703, 0.00976672837359338
    81
    EP 81 simulation finished, start learning
    1.4
    EP 81 learning finished: 0.023791016073776206, 0.0086869325343543
    82
    EP 82 simulation finished, start learning
    2.3
    EP 82 learning finished: 0.02024331617165842, 0.00952691646385695
    83
    EP 83 simulation finished, start learning
    2.3
    EP 83 learning finished: 0.018585140019510504, 0.010710317445715193
    84
    EP 84 simulation finished, start learning
    0.4
    EP 84 learning finished: 0.3476561144959023, 0.06610588366853969
    85
    EP 85 simulation finished, start learning
    0.4
    EP 85 learning finished: 0.3014833044727343, 0.07027505448644633
    86
    EP 86 simulation finished, start learning
    2.3
    EP 86 learning finished: 0.26480847344223246, 0.0638495977460259
    87
    EP 87 simulation finished, start learning
    0.5
    EP 87 learning finished: 0.23459460964612777, 0.05697524111063871
    88
    EP 88 simulation finished, start learning
    2.3
    EP 88 learning finished: 0.07447482401993666, 0.017743439674327677
    89
    EP 89 simulation finished, start learning
    2.3
    EP 89 learning finished: 0.016332011630744603, 0.007438992867517887
    90
    EP 90 simulation finished, start learning
    2.3
    EP 90 learning finished: 0.016136541706281093, 0.007349306666501576
    91
    EP 91 simulation finished, start learning
    3.2
    EP 91 learning finished: 0.023955973872631375, 0.01250063531599821
    92
    EP 92 simulation finished, start learning
    1.4
    EP 92 learning finished: 0.020823256552525038, 0.011168066926038259
    93
    EP 93 simulation finished, start learning
    2.3
    EP 93 learning finished: 0.02320480124699742, 0.011957575938081591
    94
    EP 94 simulation finished, start learning
    1.4
    EP 94 learning finished: 0.029011341451482764, 0.013305727921535985
    95
    EP 95 simulation finished, start learning
    1.4
    EP 95 learning finished: 0.01620103725602995, 0.009238551531520164
    96
    EP 96 simulation finished, start learning
    1.4
    EP 96 learning finished: 0.015900687707080863, 0.008286780160392008
    97
    EP 97 simulation finished, start learning
    0.5
    EP 97 learning finished: 0.04237871651677061, 0.012263196775528229
    98
    EP 98 simulation finished, start learning
    2.3
    EP 98 learning finished: 0.02777670558104184, 0.009237580639213464
    99
    EP 99 simulation finished, start learning
    1.1
    EP 99 learning finished: 0.14775709451102292, 0.03417411669181066
    100
    EP 100 simulation finished, start learning
    2.3
    EP 100 learning finished: 0.1428878131594291, 0.03323407117727902
    101
    EP 101 simulation finished, start learning
    3.1
    EP 101 learning finished: 0.20553833376864716, 0.04210657148506236
    102
    EP 102 simulation finished, start learning
    2.3
    EP 102 learning finished: 0.19562505857830192, 0.03954700995750319
    103
    EP 103 simulation finished, start learning
    1.3
    EP 103 learning finished: 0.2208068512202903, 0.055361070204847485
    104
    EP 104 simulation finished, start learning
    2.3
    EP 104 learning finished: 0.19858648793415323, 0.051405511255765646
    105
    EP 105 simulation finished, start learning
    0.5
    EP 105 learning finished: 0.17947256386258026, 0.03541385052372448
    106
    EP 106 simulation finished, start learning
    0.5
    EP 106 learning finished: 0.1633766163961745, 0.03386414437090727
    107
    EP 107 simulation finished, start learning
    1.4
    EP 107 learning finished: 0.026925149606933374, 0.00860580454174631
    108
    EP 108 simulation finished, start learning
    2.2
    EP 108 learning finished: 0.06163159468371138, 0.018890962819835416
    109
    EP 109 simulation finished, start learning
    3.2
    EP 109 learning finished: 0.0606583809559047, 0.02077450077532217
    110
    EP 110 simulation finished, start learning
    1.4
    EP 110 learning finished: 0.05981657681092978, 0.018436982240394622
    111
    EP 111 simulation finished, start learning
    1.4
    EP 111 learning finished: 0.058930694195072686, 0.01820365479045738
    112
    EP 112 simulation finished, start learning
    3.2
    EP 112 learning finished: 0.02562625212783587, 0.01137778744435547
    113
    EP 113 simulation finished, start learning
    2.3
    EP 113 learning finished: 0.037747223142806874, 0.012527279640735559
    114
    EP 114 simulation finished, start learning
    1.4
    EP 114 learning finished: 0.036916412756500705, 0.01377119508663475
    115
    EP 115 simulation finished, start learning
    3.2
    EP 115 learning finished: 0.03615179636281596, 0.015397317370413019
    116
    EP 116 simulation finished, start learning
    4.1
    EP 116 learning finished: 0.02784262299327267, 0.013322217463689926
    117
    EP 117 simulation finished, start learning
    2.1
    EP 117 learning finished: 0.058479241494260314, 0.020705439876001375
    118
    EP 118 simulation finished, start learning
    4.1
    EP 118 learning finished: 0.05760333188895839, 0.021724807509022614
    119
    EP 119 simulation finished, start learning
    3.2
    EP 119 learning finished: 0.05674180481868252, 0.019390121624920525
    120
    EP 120 simulation finished, start learning
    1.4
    EP 120 learning finished: 0.055931530260572515, 0.01723877553202661
    121
    EP 121 simulation finished, start learning
    2.3
    EP 121 learning finished: 0.04494562099712596, 0.01561286774375444
    122
    EP 122 simulation finished, start learning
    2.1
    EP 122 learning finished: 0.15921795688752383, 0.042378320563798866
    123
    EP 123 simulation finished, start learning
    4.1
    EP 123 learning finished: 0.14645492690990647, 0.042947090729517225
    124
    EP 124 simulation finished, start learning
    4.1
    EP 124 learning finished: 0.13535097112705508, 0.04268632761301886
    125
    EP 125 simulation finished, start learning
    3.2
    EP 125 learning finished: 0.12487415175475647, 0.03421581435589675
    126
    EP 126 simulation finished, start learning
    1.4
    EP 126 learning finished: 0.02721867109600673, 0.011172292746329878
    127
    EP 127 simulation finished, start learning
    3.2
    EP 127 learning finished: 0.022222925997819942, 0.008386515170359488
    128
    EP 128 simulation finished, start learning
    4.0
    EP 128 learning finished: 0.05549897039579932, 0.01768861249417893
    129
    EP 129 simulation finished, start learning
    0.5
    EP 129 learning finished: 0.05472372724297525, 0.017074579630094327
    130
    EP 130 simulation finished, start learning
    1.1
    EP 130 learning finished: 0.22966089813367851, 0.056693152372415824
    131
    EP 131 simulation finished, start learning
    1.2
    EP 131 learning finished: 0.19866715552799796, 0.05902501838880703
    132
    EP 132 simulation finished, start learning
    2.3
    EP 132 learning finished: 0.17392975442301584, 0.044828803469033396
    133
    EP 133 simulation finished, start learning
    2.3
    EP 133 learning finished: 0.15358501114148737, 0.04090166890759376
    134
    EP 134 simulation finished, start learning
    3.2
    EP 134 learning finished: 0.050447926571144386, 0.015164001867445117
    135
    EP 135 simulation finished, start learning
    2.3
    EP 135 learning finished: 0.015126541889209477, 0.006377143621980312
    136
    EP 136 simulation finished, start learning
    1.4
    EP 136 learning finished: 0.013308355996301202, 0.007475403439352195
    137
    EP 137 simulation finished, start learning
    1.4
    EP 137 learning finished: 0.011619600763987492, 0.006710154490842927
    138
    EP 138 simulation finished, start learning
    4.1
    EP 138 learning finished: 0.013035814219318098, 0.006773891836387176
    139
    EP 139 simulation finished, start learning
    3.2
    EP 139 learning finished: 0.01706838024303181, 0.008602449477380128
    140
    EP 140 simulation finished, start learning
    0.5
    EP 140 learning finished: 0.016107283136912898, 0.006571348595672433
    141
    EP 141 simulation finished, start learning
    2.3
    EP 141 learning finished: 0.019399354868040675, 0.007213527871651119
    142
    EP 142 simulation finished, start learning
    1.4
    EP 142 learning finished: 0.012320087683781011, 0.005666783316104744
    143
    EP 143 simulation finished, start learning
    3.2
    EP 143 learning finished: 0.012033091235746696, 0.005047198460381115
    144
    EP 144 simulation finished, start learning
    3.1
    EP 144 learning finished: 0.07404376671644286, 0.019083030160547725
    145
    EP 145 simulation finished, start learning
    4.0
    EP 145 learning finished: 0.1364045766983839, 0.031008796298095536
    146
    EP 146 simulation finished, start learning
    0.3
    EP 146 learning finished: 0.12339075294324964, 0.03637716028060748
    147
    EP 147 simulation finished, start learning
    0.5
    EP 147 learning finished: 0.11155782432271977, 0.034699097219379325
    148
    EP 148 simulation finished, start learning
    1.2
    EP 148 learning finished: 0.0977558529252276, 0.031004000525302573
    149
    EP 149 simulation finished, start learning
    3.2
    EP 149 learning finished: 0.09520409137095598, 0.020995470366759268
    150
    EP 150 simulation finished, start learning
    0.5
    EP 150 learning finished: 0.04667327962857115, 0.01301073604893895
    151
    EP 151 simulation finished, start learning
    2.3
    EP 151 learning finished: 0.04612956576437366, 0.012319170073268845
    152
    EP 152 simulation finished, start learning
    0.5
    EP 152 learning finished: 0.012245493598362245, 0.004622740102506147
    153
    EP 153 simulation finished, start learning
    1.4
    EP 153 learning finished: 0.013666004611190274, 0.004880474343407215
    154
    EP 154 simulation finished, start learning
    3.2
    EP 154 learning finished: 0.011086937385771378, 0.006519001629275027
    155
    EP 155 simulation finished, start learning
    0.2
    EP 155 learning finished: 0.09875451702672144, 0.024240268990464386
    156
    EP 156 simulation finished, start learning
    1.2
    EP 156 learning finished: 0.0904920476066966, 0.03223754456164215
    157
    EP 157 simulation finished, start learning
    1.3
    EP 157 learning finished: 0.12240207316536278, 0.03629363655018514
    158
    EP 158 simulation finished, start learning
    1.4
    EP 158 learning finished: 0.10821298943396762, 0.032515271666979796
    159
    EP 159 simulation finished, start learning
    1.4
    EP 159 learning finished: 0.04420506703255267, 0.017552766318794284
    160
    EP 160 simulation finished, start learning
    3.2
    EP 160 learning finished: 0.03229679276662445, 0.00905078991227389
    161
    EP 161 simulation finished, start learning
    1.4
    EP 161 learning finished: 0.010352805957790423, 0.0034180413913370935
    162
    EP 162 simulation finished, start learning
    2.3
    EP 162 learning finished: 0.009509741523295645, 0.003490372494630615
    163
    EP 163 simulation finished, start learning
    1.4
    EP 163 learning finished: 0.009988756716289515, 0.005728205445590125
    164
    EP 164 simulation finished, start learning
    1.4
    EP 164 learning finished: 0.01071769589881022, 0.004993402829985172
    165
    EP 165 simulation finished, start learning
    2.3
    EP 165 learning finished: 0.009921324030404577, 0.006040583236189612
    166
    EP 166 simulation finished, start learning
    3.2
    EP 166 learning finished: 0.011333464097983637, 0.006593454876094936
    167
    EP 167 simulation finished, start learning
    1.3
    EP 167 learning finished: 0.0322854007700446, 0.009915622675757815
    168
    EP 168 simulation finished, start learning
    2.3
    EP 168 learning finished: 0.03938647943965279, 0.01649439659422421
    169
    EP 169 simulation finished, start learning
    2.2
    EP 169 learning finished: 0.07804780434674896, 0.02699574520016886
    170
    EP 170 simulation finished, start learning
    2.3
    EP 170 learning finished: 0.07704078644637846, 0.026074434763008785
    171
    EP 171 simulation finished, start learning
    1.4
    EP 171 learning finished: 0.07607426798518269, 0.02081290000244292
    172
    EP 172 simulation finished, start learning
    3.1
    EP 172 learning finished: 0.04437600081617443, 0.02222978239124555
    173
    EP 173 simulation finished, start learning
    3.2
    EP 173 learning finished: 0.04382096293909932, 0.012203127654393834
    174
    EP 174 simulation finished, start learning
    3.1
    EP 174 learning finished: 0.08608508346345012, 0.020960686506498405
    175
    EP 175 simulation finished, start learning
    2.3
    EP 175 learning finished: 0.08397675894402834, 0.022182693909521916
    176
    EP 176 simulation finished, start learning
    1.2
    EP 176 learning finished: 0.08189601347781927, 0.02288251726986627
    177
    EP 177 simulation finished, start learning
    3.2
    EP 177 learning finished: 0.07997541430202926, 0.022461977244133607
    178
    EP 178 simulation finished, start learning
    3.2
    EP 178 learning finished: 0.039301958520018054, 0.014036665629013623
    179
    EP 179 simulation finished, start learning
    3.2
    EP 179 learning finished: 0.038831352007879794, 0.012982535666801484
    180
    EP 180 simulation finished, start learning
    2.3
    EP 180 learning finished: 0.012877292887772285, 0.0052220300217236115
    181
    EP 181 simulation finished, start learning
    2.3
    EP 181 learning finished: 0.011585303662602797, 0.0046523182208281595
    182
    EP 182 simulation finished, start learning
    1.3
    EP 182 learning finished: 0.03849996174031123, 0.010525907238166857
    183
    EP 183 simulation finished, start learning
    2.3
    EP 183 learning finished: 0.03807864291826668, 0.011810877067087647
    184
    EP 184 simulation finished, start learning
    1.4
    EP 184 learning finished: 0.03763110305179662, 0.011627958283467077
    185
    EP 185 simulation finished, start learning
    1.4
    EP 185 learning finished: 0.037239621648162746, 0.012904208228521785
    186
    EP 186 simulation finished, start learning
    2.3
    EP 186 learning finished: 0.01604417514428569, 0.007058823175569569
    187
    EP 187 simulation finished, start learning
    3.2
    EP 187 learning finished: 0.012479305606332147, 0.006115660821066034
    188
    EP 188 simulation finished, start learning
    3.2
    EP 188 learning finished: 0.014952625262173669, 0.006672254447853317
    189
    EP 189 simulation finished, start learning
    0.3
    EP 189 learning finished: 0.036995843362711334, 0.010354967686098634
    190
    EP 190 simulation finished, start learning
    2.2
    EP 190 learning finished: 0.07275715444745501, 0.017373814434932616
    191
    EP 191 simulation finished, start learning
    3.2
    EP 191 learning finished: 0.07114564728027517, 0.01708190029190752
    192
    EP 192 simulation finished, start learning
    2.3
    EP 192 learning finished: 0.06958249794752813, 0.01629704732544986
    193
    EP 193 simulation finished, start learning
    5.0
    EP 193 learning finished: 0.034242578059290985, 0.012962241119597243
    194
    EP 194 simulation finished, start learning
    1.4
    EP 194 learning finished: 0.016308173276701865, 0.005647961751311807
    195
    EP 195 simulation finished, start learning
    1.4
    EP 195 learning finished: 0.011220412547929826, 0.00463969723170828
    196
    EP 196 simulation finished, start learning
    2.3
    EP 196 learning finished: 0.009650721464232959, 0.005607275407777242
    197
    EP 197 simulation finished, start learning
    0.3
    EP 197 learning finished: 0.034002576969117904, 0.009342627273327242
    198
    EP 198 simulation finished, start learning
    3.2
    EP 198 learning finished: 0.033671376227578875, 0.010022748850623783
    199
    EP 199 simulation finished, start learning
    1.3
    EP 199 learning finished: 0.0333093983451187, 0.014962810968699447



```python
q_one_over_t.q
```




                0          1          2
    -1   6.265211  10.694199  22.739000
     1  13.857943  15.046288   9.818335




```python
v_l, pi_l= q_one_over_t.q.yield_v_pi()
```


```python
v_l
```




    0    13.857943
    1    15.046288
    2    22.739000
    dtype: float64




```python
pi_l
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
# 1 action delay, 0 obs. delay

class delayed_wrapper:
    def __init__(self, env, delay_steps):
        self.env = env
        self.num = delay_steps
        self.state_dim = env.state_dim
        self.action_dim = env.action_dim
        self.action_buffer = [-1]*delay_steps
        
    def step(self, action):
        app_action = self.action_buffer.pop(0)
        self.action_buffer.append(action)
        return  self.env.step(app_action)
    
    def reset(self, random = False):
        self.action_buffer = [0]*self.num
        return self.env.reset(random)
    

```


```python
# Q-learning using model, async reward collection


class Q_learning:
    def __init__(self, q, env, delay_step, model, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(40)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        model = self.model
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            for _ in range(10):
                
                est_obs = model(obs, self.action_memory)
                last_action = self.action_memory.pop(0)
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (q.max_value_act(est_obs)[1], False)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q
```


```python
mu1 = np.argmax(tu1.values,axis=1)
mun1 = np.argmax(tun1.values,axis=1)
```


```python

def model_model(x, u):
    assert type(u)==list
    u = u[0]
    m = mu1 
    if u == 1:
        m = mu1
    elif u == -1:
        m = mun1
    else:
        raise ValueError
    return m[x]
```


```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
# lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_model = Q_learning(q, env, 1, model_model, MAX_ITER = 1000)
learned_q = q_model.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    -0.69
    EP 3 learning finished: 10.397786613470558, 3.4985800677907246
    4
    EP 4 simulation finished, start learning
    0.99
    EP 4 learning finished: 2.6648440133445526, 1.5227860220168707
    5
    EP 5 simulation finished, start learning
    0.78
    EP 5 learning finished: 3.23151065279135, 1.422093463086135
    6
    EP 6 simulation finished, start learning
    1.41
    EP 6 learning finished: 2.982257516434988, 1.2561509518678873
    7
    EP 7 simulation finished, start learning
    1.44
    EP 7 learning finished: 1.2263434835022515, 0.8593462298800508
    8
    EP 8 simulation finished, start learning
    2.16
    EP 8 learning finished: 1.0257984746420092, 0.6991139141183916
    9
    EP 9 simulation finished, start learning
    1.3
    EP 9 learning finished: 0.7328088778892088, 0.42746340885059886
    10
    EP 10 simulation finished, start learning
    1.24
    EP 10 learning finished: 0.5467917000935572, 0.34390778412100004
    11
    EP 11 simulation finished, start learning
    1.8
    EP 11 learning finished: 0.4296087658119987, 0.2750986406160531
    12
    EP 12 simulation finished, start learning
    1.62
    EP 12 learning finished: 0.8478241919987601, 0.34039242224929184
    13
    EP 13 simulation finished, start learning
    1.7
    EP 13 learning finished: 0.6604279485971665, 0.29322997826255337
    14
    EP 14 simulation finished, start learning
    1.51
    EP 14 learning finished: 0.5096305123038469, 0.2542956745885287
    15
    EP 15 simulation finished, start learning
    0.99
    EP 15 learning finished: 0.6797586675578913, 0.25988901161782785
    16
    EP 16 simulation finished, start learning
    1.11
    EP 16 learning finished: 0.27624337097627993, 0.1693103274007015
    17
    EP 17 simulation finished, start learning
    0.8
    EP 17 learning finished: 0.22868722429978128, 0.1454478134166773
    18
    EP 18 simulation finished, start learning
    1.34
    EP 18 learning finished: 0.2032083915075713, 0.1282671853045588
    19
    EP 19 simulation finished, start learning
    1.01
    EP 19 learning finished: 0.18401588054592377, 0.09756764321198785
    20
    EP 20 simulation finished, start learning
    1.6
    EP 20 learning finished: 0.1707242899310959, 0.0958531212727139
    21
    EP 21 simulation finished, start learning
    1.5
    EP 21 learning finished: 0.15643569882281216, 0.0937261710647328
    22
    EP 22 simulation finished, start learning
    1.61
    EP 22 learning finished: 0.14849172987806902, 0.10425748517020524
    23
    EP 23 simulation finished, start learning
    1.31
    EP 23 learning finished: 0.1413920736690608, 0.09470595516940088
    24
    EP 24 simulation finished, start learning
    1.1
    EP 24 learning finished: 0.20143116209542278, 0.10089218943832723
    25
    EP 25 simulation finished, start learning
    0.81
    EP 25 learning finished: 0.2694564129736641, 0.1052372572880557
    26
    EP 26 simulation finished, start learning
    1.5
    EP 26 learning finished: 0.16403707734905204, 0.08588887980180537
    27
    EP 27 simulation finished, start learning
    1.18
    EP 27 learning finished: 0.1554711667838813, 0.08021887415670144
    28
    EP 28 simulation finished, start learning
    1.81
    EP 28 learning finished: 0.12275537112782686, 0.07556199015749154
    29
    EP 29 simulation finished, start learning
    1.43
    EP 29 learning finished: 0.14064132652080374, 0.07690412127083694
    30
    EP 30 simulation finished, start learning
    1.2
    EP 30 learning finished: 0.14305721314851816, 0.0829164253942527
    31
    EP 31 simulation finished, start learning
    1.52
    EP 31 learning finished: 0.16705056006833097, 0.08473830305602122
    32
    EP 32 simulation finished, start learning
    2.1
    EP 32 learning finished: 0.11599122915036197, 0.07602257675343334
    33
    EP 33 simulation finished, start learning
    1.52
    EP 33 learning finished: 0.08924871082220598, 0.06500526934790567
    34
    EP 34 simulation finished, start learning
    1.62
    EP 34 learning finished: 0.08121200081381375, 0.06662949902579711
    35
    EP 35 simulation finished, start learning
    1.2
    EP 35 learning finished: 0.08162461639348173, 0.06368418312507806
    36
    EP 36 simulation finished, start learning
    1.02
    EP 36 learning finished: 0.08275063065218546, 0.05898331225215051
    37
    EP 37 simulation finished, start learning
    1.43
    EP 37 learning finished: 0.08179131965545672, 0.05574067424677853
    38
    EP 38 simulation finished, start learning
    1.61
    EP 38 learning finished: 0.08091905458407389, 0.050917987221475336
    39
    EP 39 simulation finished, start learning
    1.53
    EP 39 learning finished: 0.0748554613469743, 0.04699768344170282
    40
    EP 40 simulation finished, start learning
    1.5
    EP 40 learning finished: 0.10180326123714067, 0.058454699815916854
    41
    EP 41 simulation finished, start learning
    1.31
    EP 41 learning finished: 0.09113022853209429, 0.05577754155583925
    42
    EP 42 simulation finished, start learning
    1.32
    EP 42 learning finished: 0.0669573547079736, 0.04619087031653013
    43
    EP 43 simulation finished, start learning
    1.36
    EP 43 learning finished: 0.06425780211312926, 0.04678320578253429
    44
    EP 44 simulation finished, start learning
    0.9
    EP 44 learning finished: 0.0647735301156871, 0.037267749002472726
    45
    EP 45 simulation finished, start learning
    1.4
    EP 45 learning finished: 0.06443939850825053, 0.039687933317558155
    46
    EP 46 simulation finished, start learning
    1.83
    EP 46 learning finished: 0.07658928578113411, 0.04661259564978071
    47
    EP 47 simulation finished, start learning
    1.73
    EP 47 learning finished: 0.07101726166065703, 0.048019584911250135
    48
    EP 48 simulation finished, start learning
    1.59
    EP 48 learning finished: 0.11209031815442216, 0.05511466064123619
    49
    EP 49 simulation finished, start learning
    0.89
    EP 49 learning finished: 0.1065820535505182, 0.05246511630843317
    50
    EP 50 simulation finished, start learning
    1.44
    EP 50 learning finished: 0.07213096795962848, 0.04834490186620736
    51
    EP 51 simulation finished, start learning
    1.26
    EP 51 learning finished: 0.060564891170301394, 0.04405471503832933
    52
    EP 52 simulation finished, start learning
    1.23
    EP 52 learning finished: 0.07791179675622573, 0.03938313908305554
    53
    EP 53 simulation finished, start learning
    1.2
    EP 53 learning finished: 0.08031357538467532, 0.03932129213479326
    54
    EP 54 simulation finished, start learning
    0.89
    EP 54 learning finished: 0.0704330843394736, 0.0362842910787012
    55
    EP 55 simulation finished, start learning
    0.7
    EP 55 learning finished: 0.056717344969982264, 0.0315170115084662
    56
    EP 56 simulation finished, start learning
    1.11
    EP 56 learning finished: 0.05424797704508677, 0.03239292717984318
    57
    EP 57 simulation finished, start learning
    0.99
    EP 57 learning finished: 0.06325993035222055, 0.03270445034796617
    58
    EP 58 simulation finished, start learning
    1.82
    EP 58 learning finished: 0.05490259377478601, 0.03203198939188331
    59
    EP 59 simulation finished, start learning
    1.5
    EP 59 learning finished: 0.06986042206685639, 0.03999404736231771
    60
    EP 60 simulation finished, start learning
    1.7
    EP 60 learning finished: 0.06536640890344358, 0.03471689322296564
    61
    EP 61 simulation finished, start learning
    1.38
    EP 61 learning finished: 0.04945941141760812, 0.03286673756935935
    62
    EP 62 simulation finished, start learning
    1.37
    EP 62 learning finished: 0.05086554239291452, 0.03170251343080318
    63
    EP 63 simulation finished, start learning
    1.39
    EP 63 learning finished: 0.050093424607224435, 0.02941922296730966
    64
    EP 64 simulation finished, start learning
    1.42
    EP 64 learning finished: 0.05891029181804619, 0.029762235634120355
    65
    EP 65 simulation finished, start learning
    1.13
    EP 65 learning finished: 0.06059470262830935, 0.02876164446252627
    66
    EP 66 simulation finished, start learning
    1.52
    EP 66 learning finished: 0.05309693297065721, 0.0271696994871163
    67
    EP 67 simulation finished, start learning
    1.49
    EP 67 learning finished: 0.03811101168465214, 0.02751786137481697
    68
    EP 68 simulation finished, start learning
    1.78
    EP 68 learning finished: 0.03770009968239485, 0.026798096658288213
    69
    EP 69 simulation finished, start learning
    1.3
    EP 69 learning finished: 0.03729314291884123, 0.02624772969594839
    70
    EP 70 simulation finished, start learning
    2.42
    EP 70 learning finished: 0.04372221260595843, 0.030485971442510085
    71
    EP 71 simulation finished, start learning
    1.23
    EP 71 learning finished: 0.033867889657706485, 0.027609011735927485
    72
    EP 72 simulation finished, start learning
    1.92
    EP 72 learning finished: 0.04392814077558782, 0.029357998363291078
    73
    EP 73 simulation finished, start learning
    1.44
    EP 73 learning finished: 0.04292556873772213, 0.031227801870482057
    74
    EP 74 simulation finished, start learning
    1.92
    EP 74 learning finished: 0.04318505214205892, 0.029750760569826618
    75
    EP 75 simulation finished, start learning
    1.23
    EP 75 learning finished: 0.0423665576233514, 0.02857543199401776
    76
    EP 76 simulation finished, start learning
    1.32
    EP 76 learning finished: 0.03179117340897086, 0.024396212636260906
    77
    EP 77 simulation finished, start learning
    1.32
    EP 77 learning finished: 0.0413326695800631, 0.028150280750496155
    78
    EP 78 simulation finished, start learning
    1.71
    EP 78 learning finished: 0.052552787560745884, 0.026558621777045193
    79
    EP 79 simulation finished, start learning
    1.21
    EP 79 learning finished: 0.058435418222641644, 0.027328130576730558
    80
    EP 80 simulation finished, start learning
    1.44
    EP 80 learning finished: 0.054824136780087684, 0.02640205783898549
    81
    EP 81 simulation finished, start learning
    1.2
    EP 81 learning finished: 0.05373899979449348, 0.025842623504028772
    82
    EP 82 simulation finished, start learning
    0.83
    EP 82 learning finished: 0.03895359491584571, 0.022253544310578317
    83
    EP 83 simulation finished, start learning
    1.22
    EP 83 learning finished: 0.040597796508409445, 0.02236104026720061
    84
    EP 84 simulation finished, start learning
    1.28
    EP 84 learning finished: 0.03783583995641315, 0.024731237190356897
    85
    EP 85 simulation finished, start learning
    1.51
    EP 85 learning finished: 0.030802482260741115, 0.01954614862241962
    86
    EP 86 simulation finished, start learning
    1.31
    EP 86 learning finished: 0.03966452044675428, 0.022370123338054288
    87
    EP 87 simulation finished, start learning
    1.11
    EP 87 learning finished: 0.03926431588019774, 0.02178486752456828
    88
    EP 88 simulation finished, start learning
    1.32
    EP 88 learning finished: 0.04294122315072357, 0.021060918096533026
    89
    EP 89 simulation finished, start learning
    1.11
    EP 89 learning finished: 0.04038984704520665, 0.020224555139947736
    90
    EP 90 simulation finished, start learning
    2.12
    EP 90 learning finished: 0.029297031466672507, 0.018559121370253766
    91
    EP 91 simulation finished, start learning
    1.5
    EP 91 learning finished: 0.025457389696718735, 0.01614052638374819
    92
    EP 92 simulation finished, start learning
    1.43
    EP 92 learning finished: 0.026994877154150387, 0.017830384296442976
    93
    EP 93 simulation finished, start learning
    1.35
    EP 93 learning finished: 0.03250334136061639, 0.018506255131366462
    94
    EP 94 simulation finished, start learning
    1.44
    EP 94 learning finished: 0.02547842610734108, 0.01654876184031906
    95
    EP 95 simulation finished, start learning
    1.62
    EP 95 learning finished: 0.037233250461534695, 0.020743840497749428
    96
    EP 96 simulation finished, start learning
    1.62
    EP 96 learning finished: 0.03839182441016753, 0.020968803629518067
    97
    EP 97 simulation finished, start learning
    2.19
    EP 97 learning finished: 0.0324305570893717, 0.02077451151045177
    98
    EP 98 simulation finished, start learning
    0.88
    EP 98 learning finished: 0.03885575541615083, 0.021176079533230308
    99
    EP 99 simulation finished, start learning
    1.02
    EP 99 learning finished: 0.028377571394745615, 0.016527395482454743
    100
    EP 100 simulation finished, start learning
    1.64
    EP 100 learning finished: 0.02957493571012293, 0.01658949132324909
    101
    EP 101 simulation finished, start learning
    1.22
    EP 101 learning finished: 0.041683131752483504, 0.017543063505880124
    102
    EP 102 simulation finished, start learning
    1.61
    EP 102 learning finished: 0.03448478116924747, 0.017163313502458994
    103
    EP 103 simulation finished, start learning
    1.0
    EP 103 learning finished: 0.050656475232326414, 0.019575067038004452
    104
    EP 104 simulation finished, start learning
    1.19
    EP 104 learning finished: 0.036228298187806374, 0.016955492824538492
    105
    EP 105 simulation finished, start learning
    1.2
    EP 105 learning finished: 0.023925514006664983, 0.016676976539640027
    106
    EP 106 simulation finished, start learning
    1.02
    EP 106 learning finished: 0.02515854759952063, 0.01616406848777352
    107
    EP 107 simulation finished, start learning
    1.08
    EP 107 learning finished: 0.02636508490182443, 0.016473446339395476
    108
    EP 108 simulation finished, start learning
    1.79
    EP 108 learning finished: 0.03104114790297352, 0.01951247058036169
    109
    EP 109 simulation finished, start learning
    1.76
    EP 109 learning finished: 0.02857523272956719, 0.017409294684457066
    110
    EP 110 simulation finished, start learning
    1.14
    EP 110 learning finished: 0.026820352628496025, 0.017019547827458698
    111
    EP 111 simulation finished, start learning
    0.89
    EP 111 learning finished: 0.03089150522432149, 0.01628502086138095
    112
    EP 112 simulation finished, start learning
    1.59
    EP 112 learning finished: 0.029930027311610985, 0.017240239845085565
    113
    EP 113 simulation finished, start learning
    1.2
    EP 113 learning finished: 0.02885061029008895, 0.01814085532289944
    114
    EP 114 simulation finished, start learning
    0.71
    EP 114 learning finished: 0.028920919536721712, 0.017682770902494777
    115
    EP 115 simulation finished, start learning
    1.29
    EP 115 learning finished: 0.032746502869002825, 0.019038854506350827
    116
    EP 116 simulation finished, start learning
    1.77
    EP 116 learning finished: 0.02862626153209291, 0.018004043071171576
    117
    EP 117 simulation finished, start learning
    1.32
    EP 117 learning finished: 0.02829540782036588, 0.016223626711765842
    118
    EP 118 simulation finished, start learning
    1.41
    EP 118 learning finished: 0.026536045552688847, 0.01743961412650563
    119
    EP 119 simulation finished, start learning
    1.14
    EP 119 learning finished: 0.024691414786582655, 0.014936371121614833
    120
    EP 120 simulation finished, start learning
    1.34
    EP 120 learning finished: 0.026795949100744565, 0.014676947919354255
    121
    EP 121 simulation finished, start learning
    1.92
    EP 121 learning finished: 0.027578673529008668, 0.017565627686785195
    122
    EP 122 simulation finished, start learning
    1.25
    EP 122 learning finished: 0.029424175990850898, 0.017660700148429243
    123
    EP 123 simulation finished, start learning
    1.5
    EP 123 learning finished: 0.026703200053290743, 0.01751101135396471
    124
    EP 124 simulation finished, start learning
    1.82
    EP 124 learning finished: 0.024152380365002557, 0.015125378203391548
    125
    EP 125 simulation finished, start learning
    1.98
    EP 125 learning finished: 0.02927778382326096, 0.0158354249543106
    126
    EP 126 simulation finished, start learning
    1.41
    EP 126 learning finished: 0.03435106206410765, 0.019020223759562842
    127
    EP 127 simulation finished, start learning
    1.71
    EP 127 learning finished: 0.02760702939309212, 0.018042515546063658
    128
    EP 128 simulation finished, start learning
    1.61
    EP 128 learning finished: 0.037088495474442595, 0.019498178110890645
    129
    EP 129 simulation finished, start learning
    1.5
    EP 129 learning finished: 0.037154311551583774, 0.018043194387780126
    130
    EP 130 simulation finished, start learning
    1.22
    EP 130 learning finished: 0.02339402713223926, 0.013730847481523275
    131
    EP 131 simulation finished, start learning
    0.84
    EP 131 learning finished: 0.02229323794794169, 0.012962749674493956
    132
    EP 132 simulation finished, start learning
    1.41
    EP 132 learning finished: 0.0219586542483583, 0.013025532591308083
    133
    EP 133 simulation finished, start learning
    1.32
    EP 133 learning finished: 0.015938231353461063, 0.010740809127855874
    134
    EP 134 simulation finished, start learning
    1.02
    EP 134 learning finished: 0.023008714999134483, 0.012306766158139007
    135
    EP 135 simulation finished, start learning
    1.41
    EP 135 learning finished: 0.01990532557645963, 0.012255242185443826
    136
    EP 136 simulation finished, start learning
    1.02
    EP 136 learning finished: 0.020665439629215854, 0.011364827323300316
    137
    EP 137 simulation finished, start learning
    1.42
    EP 137 learning finished: 0.020457615292789555, 0.011360292882288606
    138
    EP 138 simulation finished, start learning
    1.09
    EP 138 learning finished: 0.01686496737950982, 0.009627196629488447
    139
    EP 139 simulation finished, start learning
    1.11
    EP 139 learning finished: 0.016849927048621538, 0.009261465545861528
    140
    EP 140 simulation finished, start learning
    1.44
    EP 140 learning finished: 0.016019410662861944, 0.009267405374951446
    141
    EP 141 simulation finished, start learning
    1.33
    EP 141 learning finished: 0.03258928550858187, 0.01301665657481473
    142
    EP 142 simulation finished, start learning
    1.5
    EP 142 learning finished: 0.026343369381185866, 0.012585693417615987
    143
    EP 143 simulation finished, start learning
    1.92
    EP 143 learning finished: 0.02773418657977622, 0.013448705134464944
    144
    EP 144 simulation finished, start learning
    1.32
    EP 144 learning finished: 0.02653523464097063, 0.013193057757298732
    145
    EP 145 simulation finished, start learning
    1.41
    EP 145 learning finished: 0.021361364870065458, 0.012101257501759536
    146
    EP 146 simulation finished, start learning
    1.63
    EP 146 learning finished: 0.022369677917478015, 0.012148046252092351
    147
    EP 147 simulation finished, start learning
    1.28
    EP 147 learning finished: 0.019600735099851718, 0.012110547746874367
    148
    EP 148 simulation finished, start learning
    1.8
    EP 148 learning finished: 0.01465440424714437, 0.011315586537954836
    149
    EP 149 simulation finished, start learning
    1.35
    EP 149 learning finished: 0.016212635287477184, 0.011097372156017174
    150
    EP 150 simulation finished, start learning
    1.41
    EP 150 learning finished: 0.02897235919200014, 0.012986988669879315
    151
    EP 151 simulation finished, start learning
    1.34
    EP 151 learning finished: 0.029481795684409562, 0.012313196073428992
    152
    EP 152 simulation finished, start learning
    1.32
    EP 152 learning finished: 0.022873104497644903, 0.010735697336285396
    153
    EP 153 simulation finished, start learning
    1.41
    EP 153 learning finished: 0.03376649342088989, 0.013440595889589643
    154
    EP 154 simulation finished, start learning
    1.71
    EP 154 learning finished: 0.027949096152234176, 0.012696299275039161
    155
    EP 155 simulation finished, start learning
    1.74
    EP 155 learning finished: 0.022922536648078307, 0.011991051569898955
    156
    EP 156 simulation finished, start learning
    1.71
    EP 156 learning finished: 0.02340894556100359, 0.012153934110929576
    157
    EP 157 simulation finished, start learning
    1.38
    EP 157 learning finished: 0.018423424626886842, 0.010662480704035637
    158
    EP 158 simulation finished, start learning
    1.38
    EP 158 learning finished: 0.016890208691261677, 0.010308672736099922
    159
    EP 159 simulation finished, start learning
    1.62
    EP 159 learning finished: 0.022096465107726004, 0.01098147986774419
    160
    EP 160 simulation finished, start learning
    1.45
    EP 160 learning finished: 0.021898936719587425, 0.010656717736616214
    161
    EP 161 simulation finished, start learning
    1.51
    EP 161 learning finished: 0.021069148713090513, 0.010567185341754382
    162
    EP 162 simulation finished, start learning
    1.13
    EP 162 learning finished: 0.022770802736758355, 0.010495510822224515
    163
    EP 163 simulation finished, start learning
    0.92
    EP 163 learning finished: 0.015459328535655459, 0.01002165239337908
    164
    EP 164 simulation finished, start learning
    0.9
    EP 164 learning finished: 0.014736359837133506, 0.0095381224000158
    165
    EP 165 simulation finished, start learning
    1.91
    EP 165 learning finished: 0.018635038434966944, 0.010206061491208432
    166
    EP 166 simulation finished, start learning
    1.29
    EP 166 learning finished: 0.018396564833540907, 0.009000817177635826
    167
    EP 167 simulation finished, start learning
    1.38
    EP 167 learning finished: 0.012221502526250205, 0.007821625076150399
    168
    EP 168 simulation finished, start learning
    1.5
    EP 168 learning finished: 0.018958012753353515, 0.009888232909196345
    169
    EP 169 simulation finished, start learning
    1.6
    EP 169 learning finished: 0.012025869609061601, 0.009055238833939624
    170
    EP 170 simulation finished, start learning
    1.71
    EP 170 learning finished: 0.015216187548135807, 0.01033338034930568
    171
    EP 171 simulation finished, start learning
    1.51
    EP 171 learning finished: 0.015107802906470624, 0.01024810473867325
    172
    EP 172 simulation finished, start learning
    1.73
    EP 172 learning finished: 0.020690578637852397, 0.010305670388818614
    173
    EP 173 simulation finished, start learning
    1.11
    EP 173 learning finished: 0.0210922032260612, 0.010056929243082102
    174
    EP 174 simulation finished, start learning
    1.32
    EP 174 learning finished: 0.0152820599339627, 0.008903950327725655
    175
    EP 175 simulation finished, start learning
    1.8
    EP 175 learning finished: 0.014600956847820257, 0.00900686870709233
    176
    EP 176 simulation finished, start learning
    1.44
    EP 176 learning finished: 0.011184316303427977, 0.008286117688580083
    177
    EP 177 simulation finished, start learning
    1.44
    EP 177 learning finished: 0.012739332125756064, 0.00858774323966216
    178
    EP 178 simulation finished, start learning
    1.39
    EP 178 learning finished: 0.019110201037541685, 0.009586041220299416
    179
    EP 179 simulation finished, start learning
    1.11
    EP 179 learning finished: 0.0295448490611534, 0.01081102202835608
    180
    EP 180 simulation finished, start learning
    1.83
    EP 180 learning finished: 0.027498682974307798, 0.010467320141501979
    181
    EP 181 simulation finished, start learning
    0.98
    EP 181 learning finished: 0.019858556985122178, 0.01000784947647008
    182
    EP 182 simulation finished, start learning
    1.44
    EP 182 learning finished: 0.013456160006537488, 0.00901859988031628
    183
    EP 183 simulation finished, start learning
    1.01
    EP 183 learning finished: 0.010956521648742168, 0.007296090453825066
    184
    EP 184 simulation finished, start learning
    1.82
    EP 184 learning finished: 0.01791990948979283, 0.008700542546775658
    185
    EP 185 simulation finished, start learning
    1.62
    EP 185 learning finished: 0.017468801054752348, 0.008861518037199664
    186
    EP 186 simulation finished, start learning
    1.46
    EP 186 learning finished: 0.01699197338952274, 0.008707542450403949
    187
    EP 187 simulation finished, start learning
    1.6
    EP 187 learning finished: 0.016853957838330658, 0.008842624190243656
    188
    EP 188 simulation finished, start learning
    2.82
    EP 188 learning finished: 0.01635893408729494, 0.009294489814749637
    189
    EP 189 simulation finished, start learning
    2.02
    EP 189 learning finished: 0.01398323622691322, 0.008194017043896848
    190
    EP 190 simulation finished, start learning
    1.53
    EP 190 learning finished: 0.014983165940975596, 0.009330579284703452
    191
    EP 191 simulation finished, start learning
    1.41
    EP 191 learning finished: 0.01469259340561635, 0.010118519541905199
    192
    EP 192 simulation finished, start learning
    1.24
    EP 192 learning finished: 0.015032206917977575, 0.008468277359829912
    193
    EP 193 simulation finished, start learning
    1.68
    EP 193 learning finished: 0.01897944896025372, 0.008895739788977508
    194
    EP 194 simulation finished, start learning
    1.81
    EP 194 learning finished: 0.012395409648069844, 0.007813929046132095
    195
    EP 195 simulation finished, start learning
    1.2
    EP 195 learning finished: 0.017314158614404107, 0.008610199447013578
    196
    EP 196 simulation finished, start learning
    1.31
    EP 196 learning finished: 0.021265033999068095, 0.009353407782463444
    197
    EP 197 simulation finished, start learning
    1.5
    EP 197 learning finished: 0.016637117035598337, 0.008624837255707618
    198
    EP 198 simulation finished, start learning
    1.62
    EP 198 learning finished: 0.01697910103628253, 0.008443685469353035
    199
    EP 199 simulation finished, start learning
    1.08
    EP 199 learning finished: 0.009608684655272981, 0.007495412265480894
    200
    EP 200 simulation finished, start learning
    1.6
    EP 200 learning finished: 0.009670093129317436, 0.006902651117793053
    201
    EP 201 simulation finished, start learning
    1.01
    EP 201 learning finished: 0.010035786827209847, 0.006568127808866582
    202
    EP 202 simulation finished, start learning
    1.72
    EP 202 learning finished: 0.00975542958347475, 0.007325684383340618
    203
    EP 203 simulation finished, start learning
    2.22
    EP 203 learning finished: 0.012283938051755428, 0.00858343592764991
    204
    EP 204 simulation finished, start learning
    1.5
    EP 204 learning finished: 0.012209106222146104, 0.008453640557430179
    205
    EP 205 simulation finished, start learning
    1.33
    EP 205 learning finished: 0.012976233413349192, 0.008757291013499943
    206
    EP 206 simulation finished, start learning
    1.53
    EP 206 learning finished: 0.01329200578422629, 0.00868870886137696
    207
    EP 207 simulation finished, start learning
    0.81
    EP 207 learning finished: 0.009177563358809593, 0.006323259011934941
    208
    EP 208 simulation finished, start learning
    1.83
    EP 208 learning finished: 0.008714694721721372, 0.006464768938515775
    209
    EP 209 simulation finished, start learning
    1.36
    EP 209 learning finished: 0.00906795583135711, 0.0064628356467807775
    210
    EP 210 simulation finished, start learning
    1.61
    EP 210 learning finished: 0.00911406972524631, 0.007195440677552807
    211
    EP 211 simulation finished, start learning
    1.8
    EP 211 learning finished: 0.014722434061866352, 0.008960543165793014
    212
    EP 212 simulation finished, start learning
    1.12
    EP 212 learning finished: 0.014570195128301933, 0.00945762399415262
    213
    EP 213 simulation finished, start learning
    1.17
    EP 213 learning finished: 0.014429728505113815, 0.00913822608538902
    214
    EP 214 simulation finished, start learning
    0.81
    EP 214 learning finished: 0.014018536542920401, 0.008760522638295262
    215
    EP 215 simulation finished, start learning
    1.32
    EP 215 learning finished: 0.01603100452737749, 0.007913730052986937
    216
    EP 216 simulation finished, start learning
    0.91
    EP 216 learning finished: 0.010628135512074977, 0.006973306221994176
    217
    EP 217 simulation finished, start learning
    1.71
    EP 217 learning finished: 0.010962570133845162, 0.007234612392954236
    218
    EP 218 simulation finished, start learning
    1.0
    EP 218 learning finished: 0.011283863133469296, 0.006489330500795809
    219
    EP 219 simulation finished, start learning
    1.09
    EP 219 learning finished: 0.009684920480411918, 0.005460865057222837
    220
    EP 220 simulation finished, start learning
    2.33
    EP 220 learning finished: 0.008371123003874104, 0.006010449981414645
    221
    EP 221 simulation finished, start learning
    1.02
    EP 221 learning finished: 0.008702242809267347, 0.005693226996816551
    222
    EP 222 simulation finished, start learning
    0.9
    EP 222 learning finished: 0.008631008272358187, 0.005827675308403594
    223
    EP 223 simulation finished, start learning
    1.14
    EP 223 learning finished: 0.008302958793288795, 0.005703303727601454
    224
    EP 224 simulation finished, start learning
    1.92
    EP 224 learning finished: 0.008537670263903507, 0.005582581413859404
    225
    EP 225 simulation finished, start learning
    2.01
    EP 225 learning finished: 0.007661137158645914, 0.005823803478277452
    226
    EP 226 simulation finished, start learning
    1.9
    EP 226 learning finished: 0.00922731605810867, 0.005421343238262821
    227
    EP 227 simulation finished, start learning
    1.81
    EP 227 learning finished: 0.010972604194975588, 0.006452615198215976
    228
    EP 228 simulation finished, start learning
    2.01
    EP 228 learning finished: 0.011365084313904816, 0.006439876919596799
    229
    EP 229 simulation finished, start learning
    1.44
    EP 229 learning finished: 0.009362312180801524, 0.006202404006747209
    230
    EP 230 simulation finished, start learning
    0.81
    EP 230 learning finished: 0.010974747637023441, 0.006590886650523468
    231
    EP 231 simulation finished, start learning
    0.96
    EP 231 learning finished: 0.008357157518878111, 0.005502924639301909
    232
    EP 232 simulation finished, start learning
    1.83
    EP 232 learning finished: 0.008397666729550224, 0.005407618592913721
    233
    EP 233 simulation finished, start learning
    1.52
    EP 233 learning finished: 0.008350169167458787, 0.005328892903739242
    234
    EP 234 simulation finished, start learning
    1.73
    EP 234 learning finished: 0.007861919642914827, 0.004831852548948727
    235
    EP 235 simulation finished, start learning
    0.72
    EP 235 learning finished: 0.007191306711002099, 0.006385582637464656
    236
    EP 236 simulation finished, start learning
    1.02
    EP 236 learning finished: 0.010768066679267108, 0.00678641007244174
    237
    EP 237 simulation finished, start learning
    1.81
    EP 237 learning finished: 0.014976334545895753, 0.007630640340943924
    238
    EP 238 simulation finished, start learning
    1.4
    EP 238 learning finished: 0.015220765818270365, 0.00872208975839269
    239
    EP 239 simulation finished, start learning
    1.7
    EP 239 learning finished: 0.01088922043567031, 0.007242192093402859
    240
    EP 240 simulation finished, start learning
    1.79
    EP 240 learning finished: 0.008649556772425626, 0.006702151207993741
    241
    EP 241 simulation finished, start learning
    1.11
    EP 241 learning finished: 0.007740661201319554, 0.00578026234102295
    242
    EP 242 simulation finished, start learning
    1.43
    EP 242 learning finished: 0.007363443072659237, 0.004889455044325845
    243
    EP 243 simulation finished, start learning
    0.73
    EP 243 learning finished: 0.011428632809087347, 0.006029905302104825
    244
    EP 244 simulation finished, start learning
    1.1
    EP 244 learning finished: 0.015829708244293528, 0.006507494534967684
    245
    EP 245 simulation finished, start learning
    1.42
    EP 245 learning finished: 0.019785918785855117, 0.00730130111184657
    246
    EP 246 simulation finished, start learning
    1.11
    EP 246 learning finished: 0.01989325263772379, 0.007092723106619421
    247
    EP 247 simulation finished, start learning
    1.35
    EP 247 learning finished: 0.015931631887964315, 0.006582553262025392
    248
    EP 248 simulation finished, start learning
    1.59
    EP 248 learning finished: 0.013571758814467927, 0.006362038413118516
    249
    EP 249 simulation finished, start learning
    0.9
    EP 249 learning finished: 0.008823282374498476, 0.005287507197334247
    250
    EP 250 simulation finished, start learning
    1.82
    EP 250 learning finished: 0.0120881511694364, 0.0060665986586855665
    251
    EP 251 simulation finished, start learning
    1.01
    EP 251 learning finished: 0.010802981323136507, 0.005695042463658003
    252
    EP 252 simulation finished, start learning
    1.6
    EP 252 learning finished: 0.012323205277398941, 0.005976984299891501
    253
    EP 253 simulation finished, start learning
    1.21
    EP 253 learning finished: 0.016458400944333818, 0.006762234960647244
    254
    EP 254 simulation finished, start learning
    1.31
    EP 254 learning finished: 0.012779251574032813, 0.006901280291523075
    255
    EP 255 simulation finished, start learning
    1.31
    EP 255 learning finished: 0.009134200618504451, 0.00640828794057402
    256
    EP 256 simulation finished, start learning
    1.1
    EP 256 learning finished: 0.0127251636379615, 0.006730566712648785
    257
    EP 257 simulation finished, start learning
    1.62
    EP 257 learning finished: 0.012748468810963587, 0.006220397271330984
    258
    EP 258 simulation finished, start learning
    1.11
    EP 258 learning finished: 0.007119931069093255, 0.005525510160257892
    259
    EP 259 simulation finished, start learning
    1.54
    EP 259 learning finished: 0.009308918309670844, 0.0062870280113882515
    260
    EP 260 simulation finished, start learning
    1.17
    EP 260 learning finished: 0.008992770129882643, 0.005265475883846922
    261
    EP 261 simulation finished, start learning
    1.2
    EP 261 learning finished: 0.00922432706987486, 0.005275029139773402
    262
    EP 262 simulation finished, start learning
    1.29
    EP 262 learning finished: 0.0073964801103922895, 0.004795367559954873
    263
    EP 263 simulation finished, start learning
    1.19
    EP 263 learning finished: 0.007720015558351889, 0.004795186886568399
    264
    EP 264 simulation finished, start learning
    1.22
    EP 264 learning finished: 0.012583197428758908, 0.005822593384524843
    265
    EP 265 simulation finished, start learning
    1.41
    EP 265 learning finished: 0.012368688372506043, 0.005771057141314984
    266
    EP 266 simulation finished, start learning
    1.7
    EP 266 learning finished: 0.012275463193599734, 0.006469042800065179
    267
    EP 267 simulation finished, start learning
    1.62
    EP 267 learning finished: 0.00627702557803822, 0.005649563146039149
    268
    EP 268 simulation finished, start learning
    1.43
    EP 268 learning finished: 0.006819463192265118, 0.004922542692797016
    269
    EP 269 simulation finished, start learning
    1.42
    EP 269 learning finished: 0.0062790391480902485, 0.004897545450507875
    270
    EP 270 simulation finished, start learning
    1.93
    EP 270 learning finished: 0.006252603522732159, 0.004159245665065055
    271
    EP 271 simulation finished, start learning
    1.35
    EP 271 learning finished: 0.005881998659090115, 0.004616561304392104
    272
    EP 272 simulation finished, start learning
    1.2
    EP 272 learning finished: 0.008996131421444176, 0.005212134726658195
    273
    EP 273 simulation finished, start learning
    1.89
    EP 273 learning finished: 0.007905351335258715, 0.005206126202940216
    274
    EP 274 simulation finished, start learning
    1.11
    EP 274 learning finished: 0.008667294236337497, 0.005939959477656724
    275
    EP 275 simulation finished, start learning
    1.21
    EP 275 learning finished: 0.006740515689227067, 0.005217696253671041
    276
    EP 276 simulation finished, start learning
    1.17
    EP 276 learning finished: 0.008364304502862296, 0.005890745877586312
    277
    EP 277 simulation finished, start learning
    1.81
    EP 277 learning finished: 0.008337195901209071, 0.005802794590834992
    278
    EP 278 simulation finished, start learning
    1.23
    EP 278 learning finished: 0.008051408888668732, 0.004951644054779436
    279
    EP 279 simulation finished, start learning
    1.3
    EP 279 learning finished: 0.008282491621237398, 0.004999951598435655
    280
    EP 280 simulation finished, start learning
    1.71
    EP 280 learning finished: 0.006158399463556563, 0.004734853222342868
    281
    EP 281 simulation finished, start learning
    0.91
    EP 281 learning finished: 0.006622422982230702, 0.004414012752569363
    282
    EP 282 simulation finished, start learning
    1.31
    EP 282 learning finished: 0.006522322728958585, 0.004531969130644538
    283
    EP 283 simulation finished, start learning
    1.71
    EP 283 learning finished: 0.011748453333540354, 0.006296462814124067
    284
    EP 284 simulation finished, start learning
    1.53
    EP 284 learning finished: 0.006384197356874566, 0.005399528986752777
    285
    EP 285 simulation finished, start learning
    1.26
    EP 285 learning finished: 0.0073442412046116345, 0.005638560009586217
    286
    EP 286 simulation finished, start learning
    1.8
    EP 286 learning finished: 0.01008117762016525, 0.006190960731832599
    287
    EP 287 simulation finished, start learning
    1.29
    EP 287 learning finished: 0.010484027412303476, 0.005284153043720726
    288
    EP 288 simulation finished, start learning
    1.35
    EP 288 learning finished: 0.009669987483830411, 0.005079013294562469
    289
    EP 289 simulation finished, start learning
    1.33
    EP 289 learning finished: 0.012839537336789775, 0.00578851133109944
    290
    EP 290 simulation finished, start learning
    2.04
    EP 290 learning finished: 0.010203920188359916, 0.005335952397465131
    291
    EP 291 simulation finished, start learning
    1.21
    EP 291 learning finished: 0.006895165056334918, 0.004864168437246204
    292
    EP 292 simulation finished, start learning
    0.78
    EP 292 learning finished: 0.009370443960618147, 0.005187635699962574
    293
    EP 293 simulation finished, start learning
    1.3
    EP 293 learning finished: 0.0059274494551306844, 0.004360498536785033
    294
    EP 294 simulation finished, start learning
    1.2
    EP 294 learning finished: 0.01110382784000663, 0.005100271338239833
    295
    EP 295 simulation finished, start learning
    1.4
    EP 295 learning finished: 0.017168950339941702, 0.005962678894465334
    296
    EP 296 simulation finished, start learning
    1.23
    EP 296 learning finished: 0.014155277062418037, 0.006322179727332511
    297
    EP 297 simulation finished, start learning
    1.1
    EP 297 learning finished: 0.01451171089264669, 0.006296310416401181
    298
    EP 298 simulation finished, start learning
    1.52
    EP 298 learning finished: 0.011241620025064236, 0.006264017470321607
    299
    EP 299 simulation finished, start learning
    1.71
    EP 299 learning finished: 0.011276669336343303, 0.0052792717479475115
    300
    EP 300 simulation finished, start learning
    0.84
    EP 300 learning finished: 0.011142020443122291, 0.00562976520789545
    301
    EP 301 simulation finished, start learning
    1.11
    EP 301 learning finished: 0.010997731760513219, 0.00615376484842578
    302
    EP 302 simulation finished, start learning
    1.13
    EP 302 learning finished: 0.011143709023436088, 0.005629336395483501
    303
    EP 303 simulation finished, start learning
    1.4
    EP 303 learning finished: 0.011283137845561342, 0.006471771372228652
    304
    EP 304 simulation finished, start learning
    0.82
    EP 304 learning finished: 0.011410445179819817, 0.005615338830741538
    305
    EP 305 simulation finished, start learning
    1.52
    EP 305 learning finished: 0.008277233040541176, 0.005181743401685424
    306
    EP 306 simulation finished, start learning
    0.69
    EP 306 learning finished: 0.010405418467431815, 0.005955671833669444
    307
    EP 307 simulation finished, start learning
    1.92
    EP 307 learning finished: 0.007996453018639826, 0.0052070465948474975
    308
    EP 308 simulation finished, start learning
    1.47
    EP 308 learning finished: 0.011356130017931676, 0.00678898569268771
    309
    EP 309 simulation finished, start learning
    1.49
    EP 309 learning finished: 0.014472271336053666, 0.007251715179455331
    310
    EP 310 simulation finished, start learning
    1.35
    EP 310 learning finished: 0.011181643865967317, 0.0059042144001502095
    311
    EP 311 simulation finished, start learning
    1.39
    EP 311 learning finished: 0.010932304144608551, 0.005685651057315037
    312
    EP 312 simulation finished, start learning
    2.34
    EP 312 learning finished: 0.005262519464025672, 0.004821692196988063
    313
    EP 313 simulation finished, start learning
    1.82
    EP 313 learning finished: 0.005916617422087711, 0.004947105723766221
    314
    EP 314 simulation finished, start learning
    0.7
    EP 314 learning finished: 0.007672241565510163, 0.005356740117915315
    315
    EP 315 simulation finished, start learning
    1.2
    EP 315 learning finished: 0.010923488383067337, 0.0066633300429614906
    316
    EP 316 simulation finished, start learning
    1.53
    EP 316 learning finished: 0.01378667644258691, 0.00600064277277627
    317
    EP 317 simulation finished, start learning
    1.23
    EP 317 learning finished: 0.013210627674720143, 0.0056847952358246205
    318
    EP 318 simulation finished, start learning
    1.79
    EP 318 learning finished: 0.009744385607556083, 0.00540746449130225
    319
    EP 319 simulation finished, start learning
    1.2
    EP 319 learning finished: 0.0093994152568726, 0.0045684737372644335
    320
    EP 320 simulation finished, start learning
    1.34
    EP 320 learning finished: 0.009844148332026847, 0.004650880831251314
    321
    EP 321 simulation finished, start learning
    1.81
    EP 321 learning finished: 0.009957648746073744, 0.0048281806791212745
    322
    EP 322 simulation finished, start learning
    1.32
    EP 322 learning finished: 0.013260415810508519, 0.005225735631594712
    323
    EP 323 simulation finished, start learning
    1.39
    EP 323 learning finished: 0.010146009537612244, 0.004760507488865843
    324
    EP 324 simulation finished, start learning
    0.89
    EP 324 learning finished: 0.006492546516927433, 0.00396477007154387
    325
    EP 325 simulation finished, start learning
    1.0
    EP 325 learning finished: 0.0059921812912602235, 0.0032936796912294377
    326
    EP 326 simulation finished, start learning
    1.1
    EP 326 learning finished: 0.006024382577392018, 0.003509735862426903
    327
    EP 327 simulation finished, start learning
    1.53
    EP 327 learning finished: 0.009898793443872123, 0.004455895919727837
    328
    EP 328 simulation finished, start learning
    1.11
    EP 328 learning finished: 0.009741564522149915, 0.004573528875421691
    329
    EP 329 simulation finished, start learning
    0.81
    EP 329 learning finished: 0.009774439622715292, 0.004420765646012607
    330
    EP 330 simulation finished, start learning
    1.47
    EP 330 learning finished: 0.005041550108256132, 0.00420445716609219
    331
    EP 331 simulation finished, start learning
    1.31
    EP 331 learning finished: 0.006849978244893862, 0.003842111851216856
    332
    EP 332 simulation finished, start learning
    1.61
    EP 332 learning finished: 0.006678966934398289, 0.0039036821633076832
    333
    EP 333 simulation finished, start learning
    1.53
    EP 333 learning finished: 0.0069461339001595235, 0.004847962927035414
    334
    EP 334 simulation finished, start learning
    1.51
    EP 334 learning finished: 0.004858035323001886, 0.00434656677398267
    335
    EP 335 simulation finished, start learning
    1.61
    EP 335 learning finished: 0.005218531057611386, 0.0039854377733066215
    336
    EP 336 simulation finished, start learning
    1.53
    EP 336 learning finished: 0.009573770062528908, 0.005204696745665001
    337
    EP 337 simulation finished, start learning
    1.71
    EP 337 learning finished: 0.004911839044673627, 0.004465392954022818
    338
    EP 338 simulation finished, start learning
    1.38
    EP 338 learning finished: 0.006845988099883016, 0.00486391961714953
    339
    EP 339 simulation finished, start learning
    1.5
    EP 339 learning finished: 0.0096712247939017, 0.006049830534928802
    340
    EP 340 simulation finished, start learning
    1.5
    EP 340 learning finished: 0.009407809403763068, 0.005222698372885286
    341
    EP 341 simulation finished, start learning
    1.32
    EP 341 learning finished: 0.015013907020648531, 0.006785857874206454
    342
    EP 342 simulation finished, start learning
    1.09
    EP 342 learning finished: 0.015065925644414335, 0.0067263384932999175
    343
    EP 343 simulation finished, start learning
    1.19
    EP 343 learning finished: 0.012369310571241243, 0.005431658305199131
    344
    EP 344 simulation finished, start learning
    1.11
    EP 344 learning finished: 0.009443582259285321, 0.004869718169492702
    345
    EP 345 simulation finished, start learning
    0.9
    EP 345 learning finished: 0.005726566352340257, 0.003069908264822748
    346
    EP 346 simulation finished, start learning
    1.13
    EP 346 learning finished: 0.005466580294505974, 0.002542449244458863
    347
    EP 347 simulation finished, start learning
    1.71
    EP 347 learning finished: 0.005174036656082848, 0.0025879787051673966
    348
    EP 348 simulation finished, start learning
    1.0
    EP 348 learning finished: 0.005099230064191218, 0.003024656788701563
    349
    EP 349 simulation finished, start learning
    2.1
    EP 349 learning finished: 0.00591746378517044, 0.004481606647678345
    350
    EP 350 simulation finished, start learning
    1.7
    EP 350 learning finished: 0.005782782540654452, 0.004630943948604305
    351
    EP 351 simulation finished, start learning
    1.19
    EP 351 learning finished: 0.008643333452960533, 0.005749478874333154
    352
    EP 352 simulation finished, start learning
    1.56
    EP 352 learning finished: 0.008670781428348207, 0.0054571395442515325
    353
    EP 353 simulation finished, start learning
    1.26
    EP 353 learning finished: 0.008598931370833895, 0.005324466894401543
    354
    EP 354 simulation finished, start learning
    2.29
    EP 354 learning finished: 0.008846928329203863, 0.005456840337257536
    355
    EP 355 simulation finished, start learning
    1.32
    EP 355 learning finished: 0.006981514896876817, 0.004743319739530054
    356
    EP 356 simulation finished, start learning
    1.35
    EP 356 learning finished: 0.006822406077521492, 0.004677166785507225
    357
    EP 357 simulation finished, start learning
    1.72
    EP 357 learning finished: 0.005209843757633337, 0.003567856597470017
    358
    EP 358 simulation finished, start learning
    0.78
    EP 358 learning finished: 0.004867947886770807, 0.0032657760108640113
    359
    EP 359 simulation finished, start learning
    1.5
    EP 359 learning finished: 0.004851048029228977, 0.002771417898679888
    360
    EP 360 simulation finished, start learning
    1.41
    EP 360 learning finished: 0.004765302802978155, 0.003203275987998152
    361
    EP 361 simulation finished, start learning
    1.62
    EP 361 learning finished: 0.004799839489907143, 0.0031957414642042417
    362
    EP 362 simulation finished, start learning
    1.5
    EP 362 learning finished: 0.005833541096412986, 0.003730199826183167
    363
    EP 363 simulation finished, start learning
    0.96
    EP 363 learning finished: 0.005932857793078838, 0.0037074116401667467
    364
    EP 364 simulation finished, start learning
    2.2
    EP 364 learning finished: 0.004934614747572397, 0.0033880021874696333
    365
    EP 365 simulation finished, start learning
    0.91
    EP 365 learning finished: 0.004785864614973434, 0.003182137391153065
    366
    EP 366 simulation finished, start learning
    1.91
    EP 366 learning finished: 0.004666186120724447, 0.003521701140893304
    367
    EP 367 simulation finished, start learning
    1.44
    EP 367 learning finished: 0.009011376879669797, 0.004346497133437192
    368
    EP 368 simulation finished, start learning
    1.74
    EP 368 learning finished: 0.008985561286408483, 0.004238571106193412
    369
    EP 369 simulation finished, start learning
    1.2
    EP 369 learning finished: 0.009050560623988346, 0.004361348398982834
    370
    EP 370 simulation finished, start learning
    2.01
    EP 370 learning finished: 0.004472574635414617, 0.0036023907777371633
    371
    EP 371 simulation finished, start learning
    2.01
    EP 371 learning finished: 0.0055904040041774294, 0.0030975861193169862
    372
    EP 372 simulation finished, start learning
    1.71
    EP 372 learning finished: 0.005407374580546076, 0.003703028692347251
    373
    EP 373 simulation finished, start learning
    1.61
    EP 373 learning finished: 0.008625801634268626, 0.004922565105686634
    374
    EP 374 simulation finished, start learning
    1.71
    EP 374 learning finished: 0.008587954097212247, 0.005294260976130912
    375
    EP 375 simulation finished, start learning
    1.38
    EP 375 learning finished: 0.012941042898560795, 0.006256110913691752
    376
    EP 376 simulation finished, start learning
    1.69
    EP 376 learning finished: 0.008478228879766903, 0.005531214279585288
    377
    EP 377 simulation finished, start learning
    1.59
    EP 377 learning finished: 0.005784518891440271, 0.004355663844420358
    378
    EP 378 simulation finished, start learning
    1.53
    EP 378 learning finished: 0.004397066468598609, 0.0038222490212612334
    379
    EP 379 simulation finished, start learning
    0.84
    EP 379 learning finished: 0.004431707980415922, 0.0030757652993826192
    380
    EP 380 simulation finished, start learning
    2.23
    EP 380 learning finished: 0.004124932529823155, 0.003213594083543233
    381
    EP 381 simulation finished, start learning
    1.2
    EP 381 learning finished: 0.0042098801014986975, 0.0031527885583303807
    382
    EP 382 simulation finished, start learning
    1.11
    EP 382 learning finished: 0.004439799058840066, 0.003049687239957244
    383
    EP 383 simulation finished, start learning
    2.1
    EP 383 learning finished: 0.004828400948031941, 0.0028990336892957083
    384
    EP 384 simulation finished, start learning
    0.93
    EP 384 learning finished: 0.0044439505320426065, 0.0030010988212465506
    385
    EP 385 simulation finished, start learning
    1.9
    EP 385 learning finished: 0.004188968552217176, 0.003132605389372619
    386
    EP 386 simulation finished, start learning
    1.62
    EP 386 learning finished: 0.004591929836184505, 0.003222611956846736
    387
    EP 387 simulation finished, start learning
    1.21
    EP 387 learning finished: 0.004164173623957268, 0.002969364289472587
    388
    EP 388 simulation finished, start learning
    1.73
    EP 388 learning finished: 0.004757350683917139, 0.0032504742082331006
    389
    EP 389 simulation finished, start learning
    0.99
    EP 389 learning finished: 0.004092597848289969, 0.003010529178008111
    390
    EP 390 simulation finished, start learning
    0.7
    EP 390 learning finished: 0.004459689645443632, 0.002790729074361842
    391
    EP 391 simulation finished, start learning
    1.29
    EP 391 learning finished: 0.005698913471226419, 0.003227894734891675
    392
    EP 392 simulation finished, start learning
    1.03
    EP 392 learning finished: 0.004853613809755686, 0.0025527265342031313
    393
    EP 393 simulation finished, start learning
    1.6
    EP 393 learning finished: 0.004507621767286807, 0.0027685309792033763
    394
    EP 394 simulation finished, start learning
    1.62
    EP 394 learning finished: 0.006076743323399469, 0.0034308155626045314
    395
    EP 395 simulation finished, start learning
    1.41
    EP 395 learning finished: 0.0040129932647481326, 0.0030261663347503167
    396
    EP 396 simulation finished, start learning
    0.87
    EP 396 learning finished: 0.003955120856998562, 0.002997038026001597
    397
    EP 397 simulation finished, start learning
    1.2
    EP 397 learning finished: 0.006114531439573412, 0.0033073430044326577
    398
    EP 398 simulation finished, start learning
    1.53
    EP 398 learning finished: 0.006287562524899926, 0.003311888772179176
    399
    EP 399 simulation finished, start learning
    1.5
    EP 399 learning finished: 0.006251961664466066, 0.003948604472756874
    400
    EP 400 simulation finished, start learning
    0.82
    EP 400 learning finished: 0.005717439483547082, 0.0038615553934363924
    401
    EP 401 simulation finished, start learning
    0.9
    EP 401 learning finished: 0.008289579143148984, 0.004190638106435311
    402
    EP 402 simulation finished, start learning
    0.99
    EP 402 learning finished: 0.005373700128766146, 0.003561075298240013
    403
    EP 403 simulation finished, start learning
    1.72
    EP 403 learning finished: 0.005360829902702591, 0.002951701245033078
    404
    EP 404 simulation finished, start learning
    1.51
    EP 404 learning finished: 0.0054439141382616185, 0.0031441175436797883
    405
    EP 405 simulation finished, start learning
    1.71
    EP 405 learning finished: 0.003891406679921161, 0.0025147850523247564
    406
    EP 406 simulation finished, start learning
    1.13
    EP 406 learning finished: 0.004089138602541453, 0.0032148127196161744
    407
    EP 407 simulation finished, start learning
    1.73
    EP 407 learning finished: 0.008345256115564581, 0.0039268906160193255
    408
    EP 408 simulation finished, start learning
    2.0
    EP 408 learning finished: 0.012565074112155017, 0.004688059704988312
    409
    EP 409 simulation finished, start learning
    1.24
    EP 409 learning finished: 0.012509662362640839, 0.004545338489012766
    410
    EP 410 simulation finished, start learning
    1.69
    EP 410 learning finished: 0.00823486360615, 0.003990484423018081
    411
    EP 411 simulation finished, start learning
    1.89
    EP 411 learning finished: 0.008285524874144912, 0.004008783143977605
    412
    EP 412 simulation finished, start learning
    1.23
    EP 412 learning finished: 0.004058577677151121, 0.003594880555463161
    413
    EP 413 simulation finished, start learning
    1.63
    EP 413 learning finished: 0.0041286375846691925, 0.0036440678532615087
    414
    EP 414 simulation finished, start learning
    1.4
    EP 414 learning finished: 0.007779508327592666, 0.0043297957032339225
    415
    EP 415 simulation finished, start learning
    1.53
    EP 415 learning finished: 0.007942524793648431, 0.003707991032988467
    416
    EP 416 simulation finished, start learning
    1.08
    EP 416 learning finished: 0.007721567533744533, 0.0037175092745214875
    417
    EP 417 simulation finished, start learning
    1.23
    EP 417 learning finished: 0.01268517354589882, 0.004535826381205865
    418
    EP 418 simulation finished, start learning
    0.82
    EP 418 learning finished: 0.01269813047824897, 0.0043207659315533204
    419
    EP 419 simulation finished, start learning
    1.44
    EP 419 learning finished: 0.01491684802478943, 0.00465569031410477
    420
    EP 420 simulation finished, start learning
    1.69
    EP 420 learning finished: 0.012372843651206722, 0.004245484756275175
    421
    EP 421 simulation finished, start learning
    1.62
    EP 421 learning finished: 0.009869710594196945, 0.0038337724712036447
    422
    EP 422 simulation finished, start learning
    1.1
    EP 422 learning finished: 0.005038219771396513, 0.003109487032692352
    423
    EP 423 simulation finished, start learning
    0.87
    EP 423 learning finished: 0.005106295877803646, 0.003169097756212954
    424
    EP 424 simulation finished, start learning
    1.43
    EP 424 learning finished: 0.005169398582284046, 0.0031900449715799675
    425
    EP 425 simulation finished, start learning
    1.31
    EP 425 learning finished: 0.003810811085527277, 0.0027722819122365885
    426
    EP 426 simulation finished, start learning
    1.8
    EP 426 learning finished: 0.005219627176623831, 0.0032743060595379347
    427
    EP 427 simulation finished, start learning
    0.9
    EP 427 learning finished: 0.003746689646776602, 0.00269892312002303
    428
    EP 428 simulation finished, start learning
    1.14
    EP 428 learning finished: 0.00991914988293452, 0.003802773160534715
    429
    EP 429 simulation finished, start learning
    1.38
    EP 429 learning finished: 0.009792728524296024, 0.004331851959438637
    430
    EP 430 simulation finished, start learning
    1.5
    EP 430 learning finished: 0.0096739150319749, 0.004234824401853743
    431
    EP 431 simulation finished, start learning
    1.61
    EP 431 learning finished: 0.009700962322714446, 0.0043749478730286855
    432
    EP 432 simulation finished, start learning
    1.22
    EP 432 learning finished: 0.004867752497769828, 0.003584984946748785
    433
    EP 433 simulation finished, start learning
    2.09
    EP 433 learning finished: 0.004789239283926605, 0.003183622295788169
    434
    EP 434 simulation finished, start learning
    1.49
    EP 434 learning finished: 0.00428315463863882, 0.003372448897913216
    435
    EP 435 simulation finished, start learning
    0.6
    EP 435 learning finished: 0.0038295623485389285, 0.003172207226966428
    436
    EP 436 simulation finished, start learning
    1.15
    EP 436 learning finished: 0.0049043015418099856, 0.0035334307004631547
    437
    EP 437 simulation finished, start learning
    1.41
    EP 437 learning finished: 0.007166713900577548, 0.003731502227854122
    438
    EP 438 simulation finished, start learning
    0.71
    EP 438 learning finished: 0.009477841830328515, 0.0034075978621785907
    439
    EP 439 simulation finished, start learning
    1.91
    EP 439 learning finished: 0.011570440542033822, 0.00399750795320486
    440
    EP 440 simulation finished, start learning
    1.12
    EP 440 learning finished: 0.007018471201391208, 0.0031995093826241807
    441
    EP 441 simulation finished, start learning
    1.2
    EP 441 learning finished: 0.004806956939937379, 0.0027578685978412545
    442
    EP 442 simulation finished, start learning
    1.02
    EP 442 learning finished: 0.003755555155969148, 0.0024169122495788877
    443
    EP 443 simulation finished, start learning
    1.41
    EP 443 learning finished: 0.003922288235239435, 0.0020051489349537377
    444
    EP 444 simulation finished, start learning
    1.52
    EP 444 learning finished: 0.0038297799225901485, 0.002669081429275799
    445
    EP 445 simulation finished, start learning
    1.79
    EP 445 learning finished: 0.0037344978337330303, 0.0031493913418338217
    446
    EP 446 simulation finished, start learning
    1.22
    EP 446 learning finished: 0.006945811657562473, 0.0038969511338088716
    447
    EP 447 simulation finished, start learning
    1.23
    EP 447 learning finished: 0.006984915007141268, 0.003826954550330145
    448
    EP 448 simulation finished, start learning
    1.59
    EP 448 learning finished: 0.009013332481146108, 0.003650949569605544
    449
    EP 449 simulation finished, start learning
    1.18
    EP 449 learning finished: 0.006772079884186155, 0.0032232016068774647
    450
    EP 450 simulation finished, start learning
    1.71
    EP 450 learning finished: 0.006868373319498744, 0.003364592695993096
    451
    EP 451 simulation finished, start learning
    1.7
    EP 451 learning finished: 0.009003348266908517, 0.003858806577961893
    452
    EP 452 simulation finished, start learning
    1.92
    EP 452 learning finished: 0.006877806480790838, 0.003519153883487706
    453
    EP 453 simulation finished, start learning
    1.32
    EP 453 learning finished: 0.006906416177994146, 0.0034648398521139137
    454
    EP 454 simulation finished, start learning
    1.32
    EP 454 learning finished: 0.0037277105070998573, 0.0026071647073718416
    455
    EP 455 simulation finished, start learning
    1.33
    EP 455 learning finished: 0.003422501961267699, 0.0021999386513632166
    456
    EP 456 simulation finished, start learning
    0.77
    EP 456 learning finished: 0.003671720690183733, 0.0020145523110800903
    457
    EP 457 simulation finished, start learning
    2.31
    EP 457 learning finished: 0.0036008926487376414, 0.00273302375542972
    458
    EP 458 simulation finished, start learning
    1.42
    EP 458 learning finished: 0.0035555072744912763, 0.0027733598599451645
    459
    EP 459 simulation finished, start learning
    1.15
    EP 459 learning finished: 0.006853013184326073, 0.0038103708066078803
    460
    EP 460 simulation finished, start learning
    1.72
    EP 460 learning finished: 0.011217657874894371, 0.004701356069759892
    461
    EP 461 simulation finished, start learning
    1.44
    EP 461 learning finished: 0.011393415814332286, 0.0040335122652607565
    462
    EP 462 simulation finished, start learning
    1.7
    EP 462 learning finished: 0.01349888039390379, 0.004384675240464375
    463
    EP 463 simulation finished, start learning
    1.2
    EP 463 learning finished: 0.006860505276403472, 0.003286635625435667
    464
    EP 464 simulation finished, start learning
    1.01
    EP 464 learning finished: 0.0046117933179559145, 0.0027940330647805802
    465
    EP 465 simulation finished, start learning
    0.9
    EP 465 learning finished: 0.004428245836198386, 0.0026764268150714088
    466
    EP 466 simulation finished, start learning
    1.2
    EP 466 learning finished: 0.004336027055323655, 0.002589882767319196
    467
    EP 467 simulation finished, start learning
    1.03
    EP 467 learning finished: 0.0043309166849496705, 0.0024643488035923275
    468
    EP 468 simulation finished, start learning
    0.81
    EP 468 learning finished: 0.003939833687821803, 0.001998724071199239
    469
    EP 469 simulation finished, start learning
    1.0
    EP 469 learning finished: 0.0037239530041865976, 0.0020574553310706753
    470
    EP 470 simulation finished, start learning
    1.72
    EP 470 learning finished: 0.003577081236198154, 0.00219378637017531
    471
    EP 471 simulation finished, start learning
    1.23
    EP 471 learning finished: 0.006623453801328338, 0.0035024888899931974
    472
    EP 472 simulation finished, start learning
    1.17
    EP 472 learning finished: 0.007101160663124162, 0.00418583801121238
    473
    EP 473 simulation finished, start learning
    1.2
    EP 473 learning finished: 0.007063140406591728, 0.004210574740259432
    474
    EP 474 simulation finished, start learning
    1.4
    EP 474 learning finished: 0.007085653015456828, 0.003735953465717756
    475
    EP 475 simulation finished, start learning
    1.41
    EP 475 learning finished: 0.007087301755376529, 0.0034363980888514334
    476
    EP 476 simulation finished, start learning
    1.26
    EP 476 learning finished: 0.0034707764181227674, 0.0028531986647909826
    477
    EP 477 simulation finished, start learning
    1.22
    EP 477 learning finished: 0.0034614759970104103, 0.0028364842748599037
    478
    EP 478 simulation finished, start learning
    1.32
    EP 478 learning finished: 0.007066224337803462, 0.0034221150968202587
    479
    EP 479 simulation finished, start learning
    1.52
    EP 479 learning finished: 0.0034401007120798965, 0.002823457136175674
    480
    EP 480 simulation finished, start learning
    1.73
    EP 480 learning finished: 0.006959624969454126, 0.003489926315824713
    481
    EP 481 simulation finished, start learning
    1.11
    EP 481 learning finished: 0.006968361585101945, 0.0034469709520565197
    482
    EP 482 simulation finished, start learning
    1.35
    EP 482 learning finished: 0.004379859215276838, 0.0031773499356212063
    483
    EP 483 simulation finished, start learning
    1.79
    EP 483 learning finished: 0.0044036574615837765, 0.0032394555048377662
    484
    EP 484 simulation finished, start learning
    0.97
    EP 484 learning finished: 0.004280543428492933, 0.0025778784309379668
    485
    EP 485 simulation finished, start learning
    1.3
    EP 485 learning finished: 0.004270253232334653, 0.0026128454621507182
    486
    EP 486 simulation finished, start learning
    1.41
    EP 486 learning finished: 0.004225180926820116, 0.0026305881659747974
    487
    EP 487 simulation finished, start learning
    1.4
    EP 487 learning finished: 0.003411782072799241, 0.0022425304948200755
    488
    EP 488 simulation finished, start learning
    1.71
    EP 488 learning finished: 0.0042820780774235345, 0.0031944663829346354
    489
    EP 489 simulation finished, start learning
    1.74
    EP 489 learning finished: 0.006877030599266476, 0.0038257349550040822
    490
    EP 490 simulation finished, start learning
    0.93
    EP 490 learning finished: 0.00678096622231017, 0.0037658317491615634
    491
    EP 491 simulation finished, start learning
    1.1
    EP 491 learning finished: 0.00671246098602829, 0.0036961542940687764
    492
    EP 492 simulation finished, start learning
    1.2
    EP 492 learning finished: 0.0032486694720947895, 0.0027007357553244424
    493
    EP 493 simulation finished, start learning
    0.94
    EP 493 learning finished: 0.003363804855290198, 0.0020353009133806665
    494
    EP 494 simulation finished, start learning
    1.41
    EP 494 learning finished: 0.0034056112011704442, 0.0017780423951151054
    495
    EP 495 simulation finished, start learning
    1.24
    EP 495 learning finished: 0.0032543038251642287, 0.0018129783086215194
    496
    EP 496 simulation finished, start learning
    1.35
    EP 496 learning finished: 0.003069260694964271, 0.0021581957586670106
    497
    EP 497 simulation finished, start learning
    1.19
    EP 497 learning finished: 0.003097592486597023, 0.0022360177467687
    498
    EP 498 simulation finished, start learning
    1.44
    EP 498 learning finished: 0.0030060930841226963, 0.002192756064963497
    499
    EP 499 simulation finished, start learning
    1.59
    EP 499 learning finished: 0.0033672542898237623, 0.0027088925079826374
    500
    EP 500 simulation finished, start learning
    1.95
    EP 500 learning finished: 0.006821947878316692, 0.003076889927367136
    501
    EP 501 simulation finished, start learning
    1.29
    EP 501 learning finished: 0.006810985450076856, 0.0030951655847104576
    502
    EP 502 simulation finished, start learning
    2.28
    EP 502 learning finished: 0.0067614663272070885, 0.003552419707636053
    503
    EP 503 simulation finished, start learning
    1.81
    EP 503 learning finished: 0.0042409920298034365, 0.0033548828827833312
    504
    EP 504 simulation finished, start learning
    1.8
    EP 504 learning finished: 0.00406780152658115, 0.0027747837082368796
    505
    EP 505 simulation finished, start learning
    1.12
    EP 505 learning finished: 0.0040495201263865965, 0.0027300803234181124
    506
    EP 506 simulation finished, start learning
    1.09
    EP 506 learning finished: 0.0032226093329175143, 0.0022245027361214773
    507
    EP 507 simulation finished, start learning
    1.3
    EP 507 learning finished: 0.0032695439161223305, 0.0021196728599646164
    508
    EP 508 simulation finished, start learning
    1.59
    EP 508 learning finished: 0.0040669738698220215, 0.0024406709404466063
    509
    EP 509 simulation finished, start learning
    1.62
    EP 509 learning finished: 0.005968906643211369, 0.0028658080625962867
    510
    EP 510 simulation finished, start learning
    1.25
    EP 510 learning finished: 0.0079546035821636, 0.00317599145698703
    511
    EP 511 simulation finished, start learning
    1.4
    EP 511 learning finished: 0.00599313491040121, 0.0028631665449833457
    512
    EP 512 simulation finished, start learning
    1.01
    EP 512 learning finished: 0.0059437534537920556, 0.002715147617068444
    513
    EP 513 simulation finished, start learning
    1.62
    EP 513 learning finished: 0.007908212052644359, 0.003032934719779353
    514
    EP 514 simulation finished, start learning
    1.31
    EP 514 learning finished: 0.007821276247607756, 0.00305940339952393
    515
    EP 515 simulation finished, start learning
    1.49
    EP 515 learning finished: 0.011644845372906332, 0.0037302459999383566
    516
    EP 516 simulation finished, start learning
    1.32
    EP 516 learning finished: 0.009730766239741229, 0.003947422961554814
    517
    EP 517 simulation finished, start learning
    1.98
    EP 517 learning finished: 0.006551592162830744, 0.003910993097439845
    518
    EP 518 simulation finished, start learning
    1.68
    EP 518 learning finished: 0.00655482695760945, 0.003646485805489282
    519
    EP 519 simulation finished, start learning
    1.11
    EP 519 learning finished: 0.006562778197453767, 0.0029578038979115226
    520
    EP 520 simulation finished, start learning
    1.23
    EP 520 learning finished: 0.0031712181097756087, 0.002383968594021605
    521
    EP 521 simulation finished, start learning
    1.63
    EP 521 learning finished: 0.0029162837655913876, 0.0018154321200523071
    522
    EP 522 simulation finished, start learning
    1.2
    EP 522 learning finished: 0.0029769633908198756, 0.0017029596381095875
    523
    EP 523 simulation finished, start learning
    1.29
    EP 523 learning finished: 0.0029702665080364454, 0.001710640937469125
    524
    EP 524 simulation finished, start learning
    0.92
    EP 524 learning finished: 0.0030964486569544647, 0.001982980725684591
    525
    EP 525 simulation finished, start learning
    1.8
    EP 525 learning finished: 0.0032192324137536588, 0.0025159583715952394
    526
    EP 526 simulation finished, start learning
    1.4
    EP 526 learning finished: 0.0032406871703614115, 0.0025533807318346056
    527
    EP 527 simulation finished, start learning
    1.53
    EP 527 learning finished: 0.003264391889342022, 0.002575741354490605
    528
    EP 528 simulation finished, start learning
    1.6
    EP 528 learning finished: 0.00326070400643097, 0.002388535688641724
    529
    EP 529 simulation finished, start learning
    1.13
    EP 529 learning finished: 0.003842827339706645, 0.002368398389834899
    530
    EP 530 simulation finished, start learning
    1.28
    EP 530 learning finished: 0.003835352016025695, 0.0028504259281116142
    531
    EP 531 simulation finished, start learning
    1.34
    EP 531 learning finished: 0.003877102528427656, 0.00282402217514773
    532
    EP 532 simulation finished, start learning
    1.52
    EP 532 learning finished: 0.0038903765854065853, 0.0027791512034000356
    533
    EP 533 simulation finished, start learning
    2.31
    EP 533 learning finished: 0.0032109270911142573, 0.0026794506287481923
    534
    EP 534 simulation finished, start learning
    1.04
    EP 534 learning finished: 0.003899348027271543, 0.0024294628567774388
    535
    EP 535 simulation finished, start learning
    1.82
    EP 535 learning finished: 0.0038646452803590137, 0.002548602293332536
    536
    EP 536 simulation finished, start learning
    1.83
    EP 536 learning finished: 0.0039195254839548, 0.00260637199606902
    537
    EP 537 simulation finished, start learning
    1.71
    EP 537 learning finished: 0.003148302733228192, 0.0026595671215146055
    538
    EP 538 simulation finished, start learning
    1.68
    EP 538 learning finished: 0.0038731233900790585, 0.0030973660119414013
    539
    EP 539 simulation finished, start learning
    1.5
    EP 539 learning finished: 0.005657444236170761, 0.0033266814550181323
    540
    EP 540 simulation finished, start learning
    1.02
    EP 540 learning finished: 0.005597812319042106, 0.0032031995685821713
    541
    EP 541 simulation finished, start learning
    2.01
    EP 541 learning finished: 0.005601927406267748, 0.00281455246986854
    542
    EP 542 simulation finished, start learning
    2.04
    EP 542 learning finished: 0.003617168740831289, 0.002225313870836428
    543
    EP 543 simulation finished, start learning
    1.24
    EP 543 learning finished: 0.003388527943425146, 0.0018872536174650751
    544
    EP 544 simulation finished, start learning
    1.35
    EP 544 learning finished: 0.003408131513356949, 0.0018800568009007534
    545
    EP 545 simulation finished, start learning
    1.5
    EP 545 learning finished: 0.002798867618292178, 0.0017795905162900638
    546
    EP 546 simulation finished, start learning
    1.31
    EP 546 learning finished: 0.0028761813124216218, 0.0016556645502901663
    547
    EP 547 simulation finished, start learning
    1.04
    EP 547 learning finished: 0.0038060555140155827, 0.0027111395713035336
    548
    EP 548 simulation finished, start learning
    1.83
    EP 548 learning finished: 0.003814666218211471, 0.0027916591468040153
    549
    EP 549 simulation finished, start learning
    1.19
    EP 549 learning finished: 0.0038007800359878274, 0.002744409168010442
    550
    EP 550 simulation finished, start learning
    1.61
    EP 550 learning finished: 0.0037679447081764295, 0.0028118828359617134
    551
    EP 551 simulation finished, start learning
    1.32
    EP 551 learning finished: 0.0028150028146356476, 0.0020555437631646356
    552
    EP 552 simulation finished, start learning
    0.91
    EP 552 learning finished: 0.0036915270073478723, 0.0022872256039840146
    553
    EP 553 simulation finished, start learning
    0.81
    EP 553 learning finished: 0.007281350894491112, 0.003269804161187532
    554
    EP 554 simulation finished, start learning
    2.13
    EP 554 learning finished: 0.01081276729127012, 0.004412110842109129
    555
    EP 555 simulation finished, start learning
    1.52
    EP 555 learning finished: 0.010767377241563736, 0.0044446702523618
    556
    EP 556 simulation finished, start learning
    0.99
    EP 556 learning finished: 0.010692471504050616, 0.004903990686544309
    557
    EP 557 simulation finished, start learning
    1.41
    EP 557 learning finished: 0.00709599020720475, 0.0038968183501297085
    558
    EP 558 simulation finished, start learning
    2.09
    EP 558 learning finished: 0.008797768690094188, 0.00368830066498127
    559
    EP 559 simulation finished, start learning
    1.26
    EP 559 learning finished: 0.008751594082539782, 0.0036275170553482874
    560
    EP 560 simulation finished, start learning
    1.23
    EP 560 learning finished: 0.006997483739251464, 0.0029217694584398046
    561
    EP 561 simulation finished, start learning
    1.31
    EP 561 learning finished: 0.006973648077586603, 0.002881397360585384
    562
    EP 562 simulation finished, start learning
    1.11
    EP 562 learning finished: 0.0029639327971473506, 0.002299916395532916
    563
    EP 563 simulation finished, start learning
    1.94
    EP 563 learning finished: 0.0029564023200414624, 0.0023890137345604265
    564
    EP 564 simulation finished, start learning
    1.5
    EP 564 learning finished: 0.002998402571153136, 0.0024171777452102936
    565
    EP 565 simulation finished, start learning
    1.5
    EP 565 learning finished: 0.002943875251403938, 0.0024501212253831284
    566
    EP 566 simulation finished, start learning
    0.9
    EP 566 learning finished: 0.0026534043357653303, 0.0019468793979582344
    567
    EP 567 simulation finished, start learning
    1.02
    EP 567 learning finished: 0.00289090428205796, 0.001499062367752953
    568
    EP 568 simulation finished, start learning
    2.13
    EP 568 learning finished: 0.0027766416327033028, 0.0016114650005271496
    569
    EP 569 simulation finished, start learning
    1.5
    EP 569 learning finished: 0.0029858572956502627, 0.0020960668196036516
    570
    EP 570 simulation finished, start learning
    1.35
    EP 570 learning finished: 0.00604901806484115, 0.002635709673281935
    571
    EP 571 simulation finished, start learning
    1.53
    EP 571 learning finished: 0.005938463596473653, 0.003287546182882887
    572
    EP 572 simulation finished, start learning
    1.09
    EP 572 learning finished: 0.0058629940533609926, 0.0031342922373379287
    573
    EP 573 simulation finished, start learning
    1.98
    EP 573 learning finished: 0.00344853645707488, 0.0027076830098877736
    574
    EP 574 simulation finished, start learning
    1.51
    EP 574 learning finished: 0.0034352207408616664, 0.002312397476830578
    575
    EP 575 simulation finished, start learning
    1.11
    EP 575 learning finished: 0.0027387394484623684, 0.001931226698389447
    576
    EP 576 simulation finished, start learning
    1.23
    EP 576 learning finished: 0.003445695272660032, 0.002201535590958296
    577
    EP 577 simulation finished, start learning
    1.41
    EP 577 learning finished: 0.005142244749944069, 0.0028105141737399344
    578
    EP 578 simulation finished, start learning
    1.8
    EP 578 learning finished: 0.0068351126289663, 0.003149107107835114
    579
    EP 579 simulation finished, start learning
    1.28
    EP 579 learning finished: 0.0068028651194183, 0.0031703134409193368
    580
    EP 580 simulation finished, start learning
    1.41
    EP 580 learning finished: 0.006783150379785496, 0.0036852496526439773
    581
    EP 581 simulation finished, start learning
    1.3
    EP 581 learning finished: 0.005081163617219886, 0.0029131161910737426
    582
    EP 582 simulation finished, start learning
    1.59
    EP 582 learning finished: 0.005782115984365177, 0.003092014301540935
    583
    EP 583 simulation finished, start learning
    1.42
    EP 583 learning finished: 0.005814184979536563, 0.002837991237657415
    584
    EP 584 simulation finished, start learning
    1.81
    EP 584 learning finished: 0.00290136325176249, 0.0021537340194432986
    585
    EP 585 simulation finished, start learning
    1.42
    EP 585 learning finished: 0.0028707623768919888, 0.0021697925956611166
    586
    EP 586 simulation finished, start learning
    1.5
    EP 586 learning finished: 0.0027032436326503984, 0.0016601244575020029
    587
    EP 587 simulation finished, start learning
    1.02
    EP 587 learning finished: 0.003391294019166935, 0.0022230447495304637
    588
    EP 588 simulation finished, start learning
    1.51
    EP 588 learning finished: 0.005038728362535494, 0.0024170097964685646
    589
    EP 589 simulation finished, start learning
    1.31
    EP 589 learning finished: 0.00504017185586747, 0.002830247753847992
    590
    EP 590 simulation finished, start learning
    1.23
    EP 590 learning finished: 0.006695463144644975, 0.0030463056534693798
    591
    EP 591 simulation finished, start learning
    1.2
    EP 591 learning finished: 0.005639143149442916, 0.002983047642614108
    592
    EP 592 simulation finished, start learning
    1.29
    EP 592 learning finished: 0.005641711304484431, 0.0026713865667196877
    593
    EP 593 simulation finished, start learning
    1.62
    EP 593 learning finished: 0.0033291699804927077, 0.0024641508658393243
    594
    EP 594 simulation finished, start learning
    0.6
    EP 594 learning finished: 0.0028069942573427653, 0.0021291032069582863
    595
    EP 595 simulation finished, start learning
    1.52
    EP 595 learning finished: 0.002743041582606409, 0.0017342779667233188
    596
    EP 596 simulation finished, start learning
    1.71
    EP 596 learning finished: 0.0025942566827215074, 0.0018090951440011456
    597
    EP 597 simulation finished, start learning
    1.41
    EP 597 learning finished: 0.0026745068658229343, 0.001515870401966583
    598
    EP 598 simulation finished, start learning
    1.82
    EP 598 learning finished: 0.0029606690521184476, 0.0021143750585851015
    599
    EP 599 simulation finished, start learning
    1.54
    EP 599 learning finished: 0.002794129417738489, 0.0023562555929546036
    600
    EP 600 simulation finished, start learning
    1.29
    EP 600 learning finished: 0.002791060028497938, 0.0023036562040668804
    601
    EP 601 simulation finished, start learning
    1.43
    EP 601 learning finished: 0.005648776232659358, 0.0027635197448928985
    602
    EP 602 simulation finished, start learning
    1.1
    EP 602 learning finished: 0.005662505055846978, 0.002652872230259007
    603
    EP 603 simulation finished, start learning
    0.84
    EP 603 learning finished: 0.005593052193784587, 0.0023362049493436863
    604
    EP 604 simulation finished, start learning
    1.11
    EP 604 learning finished: 0.005546812798449707, 0.0025672980019617597
    605
    EP 605 simulation finished, start learning
    1.41
    EP 605 learning finished: 0.002788030892981297, 0.0021052994878948774
    606
    EP 606 simulation finished, start learning
    1.41
    EP 606 learning finished: 0.0032955494886195424, 0.002406062574271376
    607
    EP 607 simulation finished, start learning
    1.38
    EP 607 learning finished: 0.0033035288342944114, 0.0024228566359131656
    608
    EP 608 simulation finished, start learning
    1.03
    EP 608 learning finished: 0.002700419648876462, 0.0021254601459507563
    609
    EP 609 simulation finished, start learning
    1.21
    EP 609 learning finished: 0.0032784550822810843, 0.0023579832546429813
    610
    EP 610 simulation finished, start learning
    1.49
    EP 610 learning finished: 0.0025080929279397424, 0.001713076626884913
    611
    EP 611 simulation finished, start learning
    1.98
    EP 611 learning finished: 0.0026577401117595656, 0.002182656628003876
    612
    EP 612 simulation finished, start learning
    1.6
    EP 612 learning finished: 0.003249856180030264, 0.0025391247101902437
    613
    EP 613 simulation finished, start learning
    1.59
    EP 613 learning finished: 0.002866283757850141, 0.0023182621721179095
    614
    EP 614 simulation finished, start learning
    1.14
    EP 614 learning finished: 0.0026732732679555227, 0.0022655460060319044
    615
    EP 615 simulation finished, start learning
    1.02
    EP 615 learning finished: 0.0025686234164918886, 0.0017573633736682008
    616
    EP 616 simulation finished, start learning
    1.93
    EP 616 learning finished: 0.004833933721283046, 0.002326047693159102
    617
    EP 617 simulation finished, start learning
    1.22
    EP 617 learning finished: 0.004824009764771375, 0.0026404079415955906
    618
    EP 618 simulation finished, start learning
    1.2
    EP 618 learning finished: 0.007993514418156877, 0.0031524820460668104
    619
    EP 619 simulation finished, start learning
    1.11
    EP 619 learning finished: 0.009560704489967264, 0.0034113369240129643
    620
    EP 620 simulation finished, start learning
    0.99
    EP 620 learning finished: 0.00794089904600881, 0.0030501608760810464
    621
    EP 621 simulation finished, start learning
    1.8
    EP 621 learning finished: 0.011084011757183987, 0.0033540786698337635
    622
    EP 622 simulation finished, start learning
    1.84
    EP 622 learning finished: 0.00788769933145872, 0.002924357669560488
    623
    EP 623 simulation finished, start learning
    1.08
    EP 623 learning finished: 0.007869965885820562, 0.0029242563800758368
    624
    EP 624 simulation finished, start learning
    0.99
    EP 624 learning finished: 0.006272301242095324, 0.002632100610802842
    625
    EP 625 simulation finished, start learning
    1.33
    EP 625 learning finished: 0.0030956476676493594, 0.0020655082211324207
    626
    EP 626 simulation finished, start learning
    1.2
    EP 626 learning finished: 0.0030818486891455166, 0.0019442960876864628
    627
    EP 627 simulation finished, start learning
    1.81
    EP 627 learning finished: 0.002445684540182569, 0.0017499022687494186
    628
    EP 628 simulation finished, start learning
    0.97
    EP 628 learning finished: 0.0030812444111774084, 0.0019609364800296327
    629
    EP 629 simulation finished, start learning
    0.8
    EP 629 learning finished: 0.003086598705710486, 0.0018150476010158438
    630
    EP 630 simulation finished, start learning
    1.39
    EP 630 learning finished: 0.003087656478435008, 0.0018677680100556888
    631
    EP 631 simulation finished, start learning
    2.31
    EP 631 learning finished: 0.0030828641499596188, 0.0023532242967458927
    632
    EP 632 simulation finished, start learning
    0.93
    EP 632 learning finished: 0.002650690087978802, 0.0018149838946642423
    633
    EP 633 simulation finished, start learning
    2.25
    EP 633 learning finished: 0.003044827010498352, 0.0025195301782326616
    634
    EP 634 simulation finished, start learning
    1.53
    EP 634 learning finished: 0.003029637188113554, 0.0025015283914398743
    635
    EP 635 simulation finished, start learning
    1.04
    EP 635 learning finished: 0.0045478001909060595, 0.0022097950506256105
    636
    EP 636 simulation finished, start learning
    1.62
    EP 636 learning finished: 0.004536990176465139, 0.002323091363632438
    637
    EP 637 simulation finished, start learning
    1.32
    EP 637 learning finished: 0.003009300939122994, 0.0019115245849429623
    638
    EP 638 simulation finished, start learning
    1.42
    EP 638 learning finished: 0.002994381737877916, 0.0019080739609780295
    639
    EP 639 simulation finished, start learning
    0.63
    EP 639 learning finished: 0.004529278810036175, 0.0020804657881908404
    640
    EP 640 simulation finished, start learning
    1.3
    EP 640 learning finished: 0.006031153420476443, 0.0022924037658581162
    641
    EP 641 simulation finished, start learning
    1.44
    EP 641 learning finished: 0.00599926439214471, 0.002344842458888626
    642
    EP 642 simulation finished, start learning
    1.23
    EP 642 learning finished: 0.005986237325561916, 0.0022933350569613133
    643
    EP 643 simulation finished, start learning
    1.11
    EP 643 learning finished: 0.0029431007101372586, 0.0018513897355309321
    644
    EP 644 simulation finished, start learning
    1.95
    EP 644 learning finished: 0.002308019697210284, 0.0016803788886393296
    645
    EP 645 simulation finished, start learning
    0.99
    EP 645 learning finished: 0.0025580080211931033, 0.0019994486236942777
    646
    EP 646 simulation finished, start learning
    2.0
    EP 646 learning finished: 0.005192210950527354, 0.0028051177380756798
    647
    EP 647 simulation finished, start learning
    1.25
    EP 647 learning finished: 0.007767950476983643, 0.003723015513137954
    648
    EP 648 simulation finished, start learning
    0.81
    EP 648 learning finished: 0.007731233068200893, 0.0035666709926820417
    649
    EP 649 simulation finished, start learning
    1.3
    EP 649 learning finished: 0.005915159904283485, 0.0031787983269596274
    650
    EP 650 simulation finished, start learning
    1.0
    EP 650 learning finished: 0.005886274928482038, 0.002610924655122915
    651
    EP 651 simulation finished, start learning
    1.43
    EP 651 learning finished: 0.00295025114991887, 0.0021333269597404816
    652
    EP 652 simulation finished, start learning
    1.71
    EP 652 learning finished: 0.0029203723070416743, 0.0022408950999786015
    653
    EP 653 simulation finished, start learning
    1.39
    EP 653 learning finished: 0.002916166134944831, 0.002250867122455702
    654
    EP 654 simulation finished, start learning
    1.82
    EP 654 learning finished: 0.0029253627205605426, 0.0023576992197108146
    655
    EP 655 simulation finished, start learning
    1.12
    EP 655 learning finished: 0.004373801835107827, 0.0022307421381559203
    656
    EP 656 simulation finished, start learning
    1.39
    EP 656 learning finished: 0.004379644756550505, 0.0022261686306895476
    657
    EP 657 simulation finished, start learning
    1.4
    EP 657 learning finished: 0.002880125283283519, 0.002329361320064116
    658
    EP 658 simulation finished, start learning
    1.6
    EP 658 learning finished: 0.002423358500301731, 0.0020460234352424798
    659
    EP 659 simulation finished, start learning
    1.59
    EP 659 learning finished: 0.002461699569678899, 0.002085006376534674
    660
    EP 660 simulation finished, start learning
    1.83
    EP 660 learning finished: 0.0025037872245032844, 0.0021131360225578413
    661
    EP 661 simulation finished, start learning
    1.71
    EP 661 learning finished: 0.0029001743632655064, 0.0020002163444313346
    662
    EP 662 simulation finished, start learning
    0.9
    EP 662 learning finished: 0.0058499891529528725, 0.0023979169239671805
    663
    EP 663 simulation finished, start learning
    1.37
    EP 663 learning finished: 0.007307044178364919, 0.0026740827437838157
    664
    EP 664 simulation finished, start learning
    1.42
    EP 664 learning finished: 0.007290777889053857, 0.002614936803395492
    665
    EP 665 simulation finished, start learning
    1.48
    EP 665 learning finished: 0.007313147957770383, 0.002582997056341796
    666
    EP 666 simulation finished, start learning
    1.5
    EP 666 learning finished: 0.004363832906246756, 0.0021961839979223186
    667
    EP 667 simulation finished, start learning
    0.81
    EP 667 learning finished: 0.002447914445216526, 0.001965853690871988
    668
    EP 668 simulation finished, start learning
    1.41
    EP 668 learning finished: 0.004957043515675608, 0.002372816729054724
    669
    EP 669 simulation finished, start learning
    1.29
    EP 669 learning finished: 0.00494061766258902, 0.002342157154787768
    670
    EP 670 simulation finished, start learning
    0.82
    EP 670 learning finished: 0.004924028726085439, 0.0022201584418895712
    671
    EP 671 simulation finished, start learning
    1.29
    EP 671 learning finished: 0.002830888345425109, 0.0021057080812158326
    672
    EP 672 simulation finished, start learning
    1.31
    EP 672 learning finished: 0.0028638806864051958, 0.0017514345802362996
    673
    EP 673 simulation finished, start learning
    0.77
    EP 673 learning finished: 0.0024611239707947874, 0.0014305511016932115
    674
    EP 674 simulation finished, start learning
    1.52
    EP 674 learning finished: 0.00240475601462542, 0.001885985584945793
    675
    EP 675 simulation finished, start learning
    0.72
    EP 675 learning finished: 0.0049224479100136165, 0.0019801437436983704
    676
    EP 676 simulation finished, start learning
    1.41
    EP 676 learning finished: 0.004955070710630949, 0.002176225745462131
    677
    EP 677 simulation finished, start learning
    0.91
    EP 677 learning finished: 0.004946431421798181, 0.0021848047467412832
    678
    EP 678 simulation finished, start learning
    0.93
    EP 678 learning finished: 0.002423097282841624, 0.0016970773880933265
    679
    EP 679 simulation finished, start learning
    1.81
    EP 679 learning finished: 0.0022973294851240667, 0.0014752579721761805
    680
    EP 680 simulation finished, start learning
    1.74
    EP 680 learning finished: 0.0021953391805933364, 0.001341381963117468
    681
    EP 681 simulation finished, start learning
    1.29
    EP 681 learning finished: 0.0021425398277461483, 0.0015774225126140944
    682
    EP 682 simulation finished, start learning
    1.65
    EP 682 learning finished: 0.0027482943409609106, 0.0016782463541839359
    683
    EP 683 simulation finished, start learning
    1.32
    EP 683 learning finished: 0.0023297074177293098, 0.0015949861877950038
    684
    EP 684 simulation finished, start learning
    0.53
    EP 684 learning finished: 0.0027056782018597403, 0.0016516042192756448
    685
    EP 685 simulation finished, start learning
    1.2
    EP 685 learning finished: 0.002372090077697564, 0.0017581761935193303
    686
    EP 686 simulation finished, start learning
    1.62
    EP 686 learning finished: 0.0026610682546959197, 0.0019755429178559916
    687
    EP 687 simulation finished, start learning
    1.4
    EP 687 learning finished: 0.0048271052533106484, 0.002636154561769762
    688
    EP 688 simulation finished, start learning
    1.02
    EP 688 learning finished: 0.004836058901355855, 0.0024631788586188343
    689
    EP 689 simulation finished, start learning
    1.5
    EP 689 learning finished: 0.004824366552309556, 0.0024980410049219395
    690
    EP 690 simulation finished, start learning
    1.89
    EP 690 learning finished: 0.007263107864558194, 0.0027297174592989157
    691
    EP 691 simulation finished, start learning
    1.09
    EP 691 learning finished: 0.004847551864049393, 0.0020612314557736746
    692
    EP 692 simulation finished, start learning
    1.01
    EP 692 learning finished: 0.0047867442411160255, 0.00228556002175632
    693
    EP 693 simulation finished, start learning
    1.02
    EP 693 learning finished: 0.0023362512096856136, 0.0018150829777801765
    694
    EP 694 simulation finished, start learning
    1.41
    EP 694 learning finished: 0.002299548510125149, 0.0013827389586588505
    695
    EP 695 simulation finished, start learning
    1.41
    EP 695 learning finished: 0.002199972432622488, 0.001432514576412025
    696
    EP 696 simulation finished, start learning
    1.63
    EP 696 learning finished: 0.0020528937296653282, 0.0015124594585526434
    697
    EP 697 simulation finished, start learning
    1.72
    EP 697 learning finished: 0.0024807902640233692, 0.0016242808822219956
    698
    EP 698 simulation finished, start learning
    1.61
    EP 698 learning finished: 0.00270483553776657, 0.0018570828217718827
    699
    EP 699 simulation finished, start learning
    1.59
    EP 699 learning finished: 0.0041142940066905, 0.0021072048872330384
    700
    EP 700 simulation finished, start learning
    1.81
    EP 700 learning finished: 0.004071438854349907, 0.0021225469249820015
    701
    EP 701 simulation finished, start learning
    1.73
    EP 701 learning finished: 0.005435800441119909, 0.0023353368602636593
    702
    EP 702 simulation finished, start learning
    0.98
    EP 702 learning finished: 0.0040647781152785, 0.0024092950918402543
    703
    EP 703 simulation finished, start learning
    2.24
    EP 703 learning finished: 0.004648792410458569, 0.0028753567551810613
    704
    EP 704 simulation finished, start learning
    1.5
    EP 704 learning finished: 0.004676973128809436, 0.002595759105199195
    705
    EP 705 simulation finished, start learning
    1.91
    EP 705 learning finished: 0.007087017501628878, 0.0027832568245328795
    706
    EP 706 simulation finished, start learning
    1.13
    EP 706 learning finished: 0.004720107276650509, 0.002374514536170954
    707
    EP 707 simulation finished, start learning
    1.23
    EP 707 learning finished: 0.0023697808233968942, 0.0016841897680691293
    708
    EP 708 simulation finished, start learning
    1.34
    EP 708 learning finished: 0.0022921374954112395, 0.001838146048969887
    709
    EP 709 simulation finished, start learning
    1.31
    EP 709 learning finished: 0.002236535167181586, 0.0017717035942161157
    710
    EP 710 simulation finished, start learning
    1.62
    EP 710 learning finished: 0.0022320099512622704, 0.0018445897830320728
    711
    EP 711 simulation finished, start learning
    1.99
    EP 711 learning finished: 0.0026757607702272423, 0.002183218057307812
    712
    EP 712 simulation finished, start learning
    1.79
    EP 712 learning finished: 0.004621150423144371, 0.0024010682213120305
    713
    EP 713 simulation finished, start learning
    1.82
    EP 713 learning finished: 0.003117350466109059, 0.002067234788954172
    714
    EP 714 simulation finished, start learning
    1.74
    EP 714 learning finished: 0.0032564159606636167, 0.00229219509583641
    715
    EP 715 simulation finished, start learning
    1.44
    EP 715 learning finished: 0.002772382269714768, 0.0021899057604356877
    716
    EP 716 simulation finished, start learning
    1.2
    EP 716 learning finished: 0.002607088947151226, 0.0018336210611019272
    717
    EP 717 simulation finished, start learning
    1.08
    EP 717 learning finished: 0.0026201811825785626, 0.0017287052572256083
    718
    EP 718 simulation finished, start learning
    1.02
    EP 718 learning finished: 0.0026533693413242077, 0.0016286056056410796
    719
    EP 719 simulation finished, start learning
    1.44
    EP 719 learning finished: 0.002197522211226044, 0.001724891531216836
    720
    EP 720 simulation finished, start learning
    1.29
    EP 720 learning finished: 0.0026482434567434154, 0.0019396117400691064
    721
    EP 721 simulation finished, start learning
    1.32
    EP 721 learning finished: 0.0025881759822858896, 0.001958111097791241
    722
    EP 722 simulation finished, start learning
    1.49
    EP 722 learning finished: 0.0045911082148499816, 0.002174572659904802
    723
    EP 723 simulation finished, start learning
    2.01
    EP 723 learning finished: 0.005288690363450144, 0.0025389072047392602
    724
    EP 724 simulation finished, start learning
    1.22
    EP 724 learning finished: 0.003902505755622343, 0.0022975704154832854
    725
    EP 725 simulation finished, start learning
    2.31
    EP 725 learning finished: 0.005260093512688968, 0.00263227650845869
    726
    EP 726 simulation finished, start learning
    0.91
    EP 726 learning finished: 0.005224442322937506, 0.0022065217306967235
    727
    EP 727 simulation finished, start learning
    1.28
    EP 727 learning finished: 0.0025433579632831993, 0.002017736331136888
    728
    EP 728 simulation finished, start learning
    0.9
    EP 728 learning finished: 0.004581202746761548, 0.002359382805561694
    729
    EP 729 simulation finished, start learning
    1.93
    EP 729 learning finished: 0.004573694240502135, 0.0020860904473207023
    730
    EP 730 simulation finished, start learning
    0.82
    EP 730 learning finished: 0.004539066378187684, 0.00208332788670725
    731
    EP 731 simulation finished, start learning
    1.31
    EP 731 learning finished: 0.0021978242391131886, 0.001705236476624646
    732
    EP 732 simulation finished, start learning
    1.64
    EP 732 learning finished: 0.002194754966938106, 0.0017720464546489012
    733
    EP 733 simulation finished, start learning
    1.83
    EP 733 learning finished: 0.00449023470667953, 0.0021305029284723886
    734
    EP 734 simulation finished, start learning
    1.02
    EP 734 learning finished: 0.004480394579770675, 0.0021568128160828572
    735
    EP 735 simulation finished, start learning
    1.34
    EP 735 learning finished: 0.0044991199626540634, 0.0019532766219609123
    736
    EP 736 simulation finished, start learning
    1.14
    EP 736 learning finished: 0.0021429284577827445, 0.0017235013789701863
    737
    EP 737 simulation finished, start learning
    1.32
    EP 737 learning finished: 0.002027648929951553, 0.0013237948548567857
    738
    EP 738 simulation finished, start learning
    1.39
    EP 738 learning finished: 0.001968410943115728, 0.0013634732877886673
    739
    EP 739 simulation finished, start learning
    1.21
    EP 739 learning finished: 0.0020659881090221432, 0.0013599847074425853
    740
    EP 740 simulation finished, start learning
    1.4
    EP 740 learning finished: 0.00219643078455789, 0.0017225845384825078
    741
    EP 741 simulation finished, start learning
    0.51
    EP 741 learning finished: 0.004500059161683723, 0.0020143562975591203
    742
    EP 742 simulation finished, start learning
    1.55
    EP 742 learning finished: 0.0044327911438077905, 0.001998175039729766
    743
    EP 743 simulation finished, start learning
    1.52
    EP 743 learning finished: 0.006661370408492573, 0.002610897168419335
    744
    EP 744 simulation finished, start learning
    1.52
    EP 744 learning finished: 0.006574657439960774, 0.002833946130862112
    745
    EP 745 simulation finished, start learning
    1.8
    EP 745 learning finished: 0.004260308915645439, 0.0026298658988478016
    746
    EP 746 simulation finished, start learning
    1.32
    EP 746 learning finished: 0.00427836498641021, 0.0026114682695951665
    747
    EP 747 simulation finished, start learning
    1.71
    EP 747 learning finished: 0.0037794401167907665, 0.0022641168663239952
    748
    EP 748 simulation finished, start learning
    1.69
    EP 748 learning finished: 0.0025514686697825084, 0.0015497426132196068
    749
    EP 749 simulation finished, start learning
    1.4
    EP 749 learning finished: 0.002498362494009143, 0.0016973946385059335
    750
    EP 750 simulation finished, start learning
    1.24
    EP 750 learning finished: 0.005087225690193264, 0.00213877328879312
    751
    EP 751 simulation finished, start learning
    1.64
    EP 751 learning finished: 0.003789558369920343, 0.0018950802619578748
    752
    EP 752 simulation finished, start learning
    1.42
    EP 752 learning finished: 0.00508407379028597, 0.0020541543523776204
    753
    EP 753 simulation finished, start learning
    0.9
    EP 753 learning finished: 0.00509282870597616, 0.001985877236374852
    754
    EP 754 simulation finished, start learning
    0.92
    EP 754 learning finished: 0.0038065462063308075, 0.0017198689914842784
    755
    EP 755 simulation finished, start learning
    1.92
    EP 755 learning finished: 0.0038001227117128167, 0.0017540385394907432
    756
    EP 756 simulation finished, start learning
    1.8
    EP 756 learning finished: 0.0037702488508344345, 0.0018363605805718681
    757
    EP 757 simulation finished, start learning
    2.01
    EP 757 learning finished: 0.002656003020724995, 0.0017601223122666692
    758
    EP 758 simulation finished, start learning
    1.51
    EP 758 learning finished: 0.0029833407005916968, 0.001591091777816504
    759
    EP 759 simulation finished, start learning
    1.62
    EP 759 learning finished: 0.002664974059126024, 0.0015371016670930142
    760
    EP 760 simulation finished, start learning
    0.99
    EP 760 learning finished: 0.001996795663227857, 0.0012518790835270484
    761
    EP 761 simulation finished, start learning
    1.76
    EP 761 learning finished: 0.0021603982406599442, 0.001527464830240947
    762
    EP 762 simulation finished, start learning
    1.09
    EP 762 learning finished: 0.002131125211873197, 0.0015009642937394336
    763
    EP 763 simulation finished, start learning
    1.41
    EP 763 learning finished: 0.0024369048695653817, 0.0018543902558256775
    764
    EP 764 simulation finished, start learning
    1.21
    EP 764 learning finished: 0.0024097459512759656, 0.0018925293992883379
    765
    EP 765 simulation finished, start learning
    1.29
    EP 765 learning finished: 0.002428501087038626, 0.0015420923352446418
    766
    EP 766 simulation finished, start learning
    2.13
    EP 766 learning finished: 0.00370350467404279, 0.0018567538862415134
    767
    EP 767 simulation finished, start learning
    1.24
    EP 767 learning finished: 0.002349105871058299, 0.001609547594254579
    768
    EP 768 simulation finished, start learning
    1.11
    EP 768 learning finished: 0.00489177947717323, 0.0020202698800003325
    769
    EP 769 simulation finished, start learning
    1.43
    EP 769 learning finished: 0.004875931786180843, 0.0020371389130463178
    770
    EP 770 simulation finished, start learning
    1.23
    EP 770 learning finished: 0.0035923633346079953, 0.0017078399414571048
    771
    EP 771 simulation finished, start learning
    1.3
    EP 771 learning finished: 0.002389342899370206, 0.0015119999346489739
    772
    EP 772 simulation finished, start learning
    1.14
    EP 772 learning finished: 0.002021965157720018, 0.0011170885391944552
    773
    EP 773 simulation finished, start learning
    0.91
    EP 773 learning finished: 0.0021248670869482567, 0.0010405903278819888
    774
    EP 774 simulation finished, start learning
    1.13
    EP 774 learning finished: 0.002099886636415249, 0.0010530670678804792
    775
    EP 775 simulation finished, start learning
    1.51
    EP 775 learning finished: 0.0019799457976379387, 0.0012264174091172937
    776
    EP 776 simulation finished, start learning
    1.94
    EP 776 learning finished: 0.0036179190171701237, 0.002049894218521864
    777
    EP 777 simulation finished, start learning
    1.03
    EP 777 learning finished: 0.0036333958260001964, 0.0020875254591109873
    778
    EP 778 simulation finished, start learning
    1.44
    EP 778 learning finished: 0.003601538389482073, 0.002098238009639708
    779
    EP 779 simulation finished, start learning
    1.31
    EP 779 learning finished: 0.002340496640609757, 0.0018690454265497536
    780
    EP 780 simulation finished, start learning
    1.23
    EP 780 learning finished: 0.001980686285104838, 0.001585436293035573
    781
    EP 781 simulation finished, start learning
    1.21
    EP 781 learning finished: 0.002045213995183204, 0.00157649571081316
    782
    EP 782 simulation finished, start learning
    1.55
    EP 782 learning finished: 0.0020420657076325455, 0.0015857004094993836
    783
    EP 783 simulation finished, start learning
    1.32
    EP 783 learning finished: 0.0020619944635065224, 0.001586449985245153
    784
    EP 784 simulation finished, start learning
    1.62
    EP 784 learning finished: 0.0020833378757387777, 0.0015073466539841836
    785
    EP 785 simulation finished, start learning
    1.2
    EP 785 learning finished: 0.002056916543800469, 0.0015212953672151623
    786
    EP 786 simulation finished, start learning
    0.93
    EP 786 learning finished: 0.0020783689539101147, 0.0014250148570299122
    787
    EP 787 simulation finished, start learning
    1.1
    EP 787 learning finished: 0.004197079799261871, 0.0018971569864116116
    788
    EP 788 simulation finished, start learning
    1.5
    EP 788 learning finished: 0.004187893554034261, 0.002083668259028985
    789
    EP 789 simulation finished, start learning
    0.93
    EP 789 learning finished: 0.004817604704278722, 0.0024391724725158306
    790
    EP 790 simulation finished, start learning
    1.74
    EP 790 learning finished: 0.0047823087474938575, 0.002527011429101241
    791
    EP 791 simulation finished, start learning
    1.91
    EP 791 learning finished: 0.00474514622975164, 0.0022459038309552426
    792
    EP 792 simulation finished, start learning
    1.55
    EP 792 learning finished: 0.004706450572356857, 0.0019455956033578303
    793
    EP 793 simulation finished, start learning
    1.14
    EP 793 learning finished: 0.0034153875073741347, 0.0017542300361347667
    794
    EP 794 simulation finished, start learning
    1.71
    EP 794 learning finished: 0.004653532000894245, 0.0019532337180375356
    795
    EP 795 simulation finished, start learning
    1.23
    EP 795 learning finished: 0.004637488926977795, 0.001884999281463055
    796
    EP 796 simulation finished, start learning
    1.1
    EP 796 learning finished: 0.004621391693550692, 0.0018311569265104428
    797
    EP 797 simulation finished, start learning
    1.8
    EP 797 learning finished: 0.0034425430884077457, 0.002034951396498267
    798
    EP 798 simulation finished, start learning
    1.5
    EP 798 learning finished: 0.003487475515147054, 0.0020133483734721125
    799
    EP 799 simulation finished, start learning
    1.91
    EP 799 learning finished: 0.0022947282758778442, 0.001896306585499256
    800
    EP 800 simulation finished, start learning
    1.2
    EP 800 learning finished: 0.004076539512533728, 0.002270023800044522
    801
    EP 801 simulation finished, start learning
    1.31
    EP 801 learning finished: 0.0023422018681422685, 0.001856083689756538
    802
    EP 802 simulation finished, start learning
    1.75
    EP 802 learning finished: 0.003441420627428471, 0.0020575549420159454
    803
    EP 803 simulation finished, start learning
    1.92
    EP 803 learning finished: 0.003433410726298547, 0.002041452188136598
    804
    EP 804 simulation finished, start learning
    1.41
    EP 804 learning finished: 0.0033978845258353374, 0.0017831540016940057
    805
    EP 805 simulation finished, start learning
    1.71
    EP 805 learning finished: 0.0033898284647477794, 0.0021104237750106947
    806
    EP 806 simulation finished, start learning
    1.8
    EP 806 learning finished: 0.002487979530197748, 0.0017331678958859935
    807
    EP 807 simulation finished, start learning
    1.01
    EP 807 learning finished: 0.004001458318061424, 0.0019725680287227356
    808
    EP 808 simulation finished, start learning
    1.8
    EP 808 learning finished: 0.00411658822176264, 0.0018411502939903457
    809
    EP 809 simulation finished, start learning
    1.72
    EP 809 learning finished: 0.002005564861498499, 0.0016512677172491053
    810
    EP 810 simulation finished, start learning
    1.71
    EP 810 learning finished: 0.002029214928683132, 0.001655628836231789
    811
    EP 811 simulation finished, start learning
    0.93
    EP 811 learning finished: 0.0021771968089758786, 0.0015381922798850904
    812
    EP 812 simulation finished, start learning
    2.19
    EP 812 learning finished: 0.0023747535879365955, 0.0016120033495493995
    813
    EP 813 simulation finished, start learning
    0.63
    EP 813 learning finished: 0.002169532293486043, 0.0014818180133309156
    814
    EP 814 simulation finished, start learning
    1.92
    EP 814 learning finished: 0.002083300457798032, 0.0014945189541209676
    815
    EP 815 simulation finished, start learning
    0.99
    EP 815 learning finished: 0.0019265463904005742, 0.0016076814881937906
    816
    EP 816 simulation finished, start learning
    1.61
    EP 816 learning finished: 0.002213922877423613, 0.001713880474184748
    817
    EP 817 simulation finished, start learning
    1.21
    EP 817 learning finished: 0.0021835471698530284, 0.0018077071292760631
    818
    EP 818 simulation finished, start learning
    1.73
    EP 818 learning finished: 0.0022345011977797924, 0.0017820407705049963
    819
    EP 819 simulation finished, start learning
    1.29
    EP 819 learning finished: 0.00220426997455192, 0.0018055962176652212
    820
    EP 820 simulation finished, start learning
    1.11
    EP 820 learning finished: 0.00338245642066326, 0.0019368024201037077
    821
    EP 821 simulation finished, start learning
    2.02
    EP 821 learning finished: 0.00219574313527815, 0.0018032439119461212
    822
    EP 822 simulation finished, start learning
    0.92
    EP 822 learning finished: 0.004023575329265228, 0.0020376900826729973
    823
    EP 823 simulation finished, start learning
    1.29
    EP 823 learning finished: 0.003940691965986787, 0.0020491177311692255
    824
    EP 824 simulation finished, start learning
    0.92
    EP 824 learning finished: 0.003981102011104909, 0.0017375283765431366
    825
    EP 825 simulation finished, start learning
    1.29
    EP 825 learning finished: 0.0038418885825670657, 0.0019681538663401574
    826
    EP 826 simulation finished, start learning
    1.44
    EP 826 learning finished: 0.002040340127880569, 0.0016938081096865194
    827
    EP 827 simulation finished, start learning
    1.2
    EP 827 learning finished: 0.002091198788214399, 0.0013976180855118276
    828
    EP 828 simulation finished, start learning
    1.91
    EP 828 learning finished: 0.0021424857449687806, 0.0017896870044455209
    829
    EP 829 simulation finished, start learning
    1.62
    EP 829 learning finished: 0.0019467675757081082, 0.0016094873271453498
    830
    EP 830 simulation finished, start learning
    1.42
    EP 830 learning finished: 0.001918758506370466, 0.0016223584188856204
    831
    EP 831 simulation finished, start learning
    1.53
    EP 831 learning finished: 0.0021395031882178728, 0.0016424214456420099
    832
    EP 832 simulation finished, start learning
    2.22
    EP 832 learning finished: 0.0023851894093951387, 0.0016793555740450827
    833
    EP 833 simulation finished, start learning
    1.08
    EP 833 learning finished: 0.00387371729862096, 0.0023362269847829373
    834
    EP 834 simulation finished, start learning
    1.43
    EP 834 learning finished: 0.004387125672920433, 0.0025087308229441163
    835
    EP 835 simulation finished, start learning
    1.77
    EP 835 learning finished: 0.005802374428217405, 0.003060092240904607
    836
    EP 836 simulation finished, start learning
    1.52
    EP 836 learning finished: 0.00557232971467414, 0.0026317752112804036
    837
    EP 837 simulation finished, start learning
    1.47
    EP 837 learning finished: 0.0019853970037573276, 0.0017459684072242965
    838
    EP 838 simulation finished, start learning
    1.72
    EP 838 learning finished: 0.002029255303515498, 0.0015883423601981879
    839
    EP 839 simulation finished, start learning
    2.0
    EP 839 learning finished: 0.0020791027517006455, 0.0015262708444773547
    840
    EP 840 simulation finished, start learning
    1.2
    EP 840 learning finished: 0.0019703831708746122, 0.0015816759112260808
    841
    EP 841 simulation finished, start learning
    0.99
    EP 841 learning finished: 0.003895635161947908, 0.0018836594041387045
    842
    EP 842 simulation finished, start learning
    1.33
    EP 842 learning finished: 0.0038598479098084226, 0.002026191961196725
    843
    EP 843 simulation finished, start learning
    1.39
    EP 843 learning finished: 0.003285623343721511, 0.001824336661002827
    844
    EP 844 simulation finished, start learning
    1.59
    EP 844 learning finished: 0.0038773270312102426, 0.0020046240275929015
    845
    EP 845 simulation finished, start learning
    2.02
    EP 845 learning finished: 0.0021953066709983204, 0.0017762437653742562
    846
    EP 846 simulation finished, start learning
    0.84
    EP 846 learning finished: 0.003867955508646048, 0.0018508495125454754
    847
    EP 847 simulation finished, start learning
    1.7
    EP 847 learning finished: 0.005774615955902007, 0.002053142920669752
    848
    EP 848 simulation finished, start learning
    0.99
    EP 848 learning finished: 0.005789935469033836, 0.0019907295962659313
    849
    EP 849 simulation finished, start learning
    0.81
    EP 849 learning finished: 0.005743262703681751, 0.0018503260984115144
    850
    EP 850 simulation finished, start learning
    1.8
    EP 850 learning finished: 0.005760198259645577, 0.002112371473659872
    851
    EP 851 simulation finished, start learning
    1.47
    EP 851 learning finished: 0.003821051067232517, 0.0019484293424711179
    852
    EP 852 simulation finished, start learning
    1.69
    EP 852 learning finished: 0.002189831885466731, 0.0017046813914752452
    853
    EP 853 simulation finished, start learning
    1.14
    EP 853 learning finished: 0.002187724384484113, 0.0017415289566861485
    854
    EP 854 simulation finished, start learning
    1.32
    EP 854 learning finished: 0.0016725970159594539, 0.0012338500893145365
    855
    EP 855 simulation finished, start learning
    1.29
    EP 855 learning finished: 0.0017839360925897552, 0.001465258316536359
    856
    EP 856 simulation finished, start learning
    1.29
    EP 856 learning finished: 0.0032592843196432852, 0.0017884989899010033
    857
    EP 857 simulation finished, start learning
    1.05
    EP 857 learning finished: 0.004240647312940382, 0.0019475926046202592
    858
    EP 858 simulation finished, start learning
    1.43
    EP 858 learning finished: 0.004314354993926628, 0.0019619237988385643
    859
    EP 859 simulation finished, start learning
    1.84
    EP 859 learning finished: 0.0030897096356845566, 0.0015724242140144895
    860
    EP 860 simulation finished, start learning
    1.53
    EP 860 learning finished: 0.001804570419935203, 0.0014751568898011176
    861
    EP 861 simulation finished, start learning
    0.9
    EP 861 learning finished: 0.0017700495232286784, 0.0013541946177122455
    862
    EP 862 simulation finished, start learning
    1.8
    EP 862 learning finished: 0.0019364931133445396, 0.0015344135440914879
    863
    EP 863 simulation finished, start learning
    0.92
    EP 863 learning finished: 0.0021289300293787505, 0.0016228610385370008
    864
    EP 864 simulation finished, start learning
    1.93
    EP 864 learning finished: 0.00318826635741587, 0.001564568163289574
    865
    EP 865 simulation finished, start learning
    0.86
    EP 865 learning finished: 0.0030362280048024104, 0.001565324886243798
    866
    EP 866 simulation finished, start learning
    1.11
    EP 866 learning finished: 0.0018536413841978572, 0.0015287716113316918
    867
    EP 867 simulation finished, start learning
    1.3
    EP 867 learning finished: 0.0018492958638880452, 0.0015878709291075783
    868
    EP 868 simulation finished, start learning
    1.8
    EP 868 learning finished: 0.0017571061522350107, 0.0013998453332521403
    869
    EP 869 simulation finished, start learning
    1.24
    EP 869 learning finished: 0.0019884442517366097, 0.00163569272934172
    870
    EP 870 simulation finished, start learning
    0.89
    EP 870 learning finished: 0.0031226132976343024, 0.001575065872131039
    871
    EP 871 simulation finished, start learning
    1.23
    EP 871 learning finished: 0.0020094859324792935, 0.0015960850981227754
    872
    EP 872 simulation finished, start learning
    0.92
    EP 872 learning finished: 0.001977157505343996, 0.0014766067484165053
    873
    EP 873 simulation finished, start learning
    1.53
    EP 873 learning finished: 0.003547226855530994, 0.0016375083292912151
    874
    EP 874 simulation finished, start learning
    1.12
    EP 874 learning finished: 0.003602425776449536, 0.0015668397047973552
    875
    EP 875 simulation finished, start learning
    1.65
    EP 875 learning finished: 0.0016971818294386765, 0.0013018618766137042
    876
    EP 876 simulation finished, start learning
    1.11
    EP 876 learning finished: 0.001694261992488677, 0.0013039165161708628
    877
    EP 877 simulation finished, start learning
    1.2
    EP 877 learning finished: 0.0016525135322478235, 0.0010010935817324669
    878
    EP 878 simulation finished, start learning
    1.22
    EP 878 learning finished: 0.0016220356162044425, 0.000979109884694888
    879
    EP 879 simulation finished, start learning
    1.32
    EP 879 learning finished: 0.0017578042843489072, 0.0011852130939079923
    880
    EP 880 simulation finished, start learning
    1.29
    EP 880 learning finished: 0.0017561674365040858, 0.0013789650932031587
    881
    EP 881 simulation finished, start learning
    1.41
    EP 881 learning finished: 0.003603893817102133, 0.001716613071771178
    882
    EP 882 simulation finished, start learning
    1.32
    EP 882 learning finished: 0.0035958018897375155, 0.0017313471595192809
    883
    EP 883 simulation finished, start learning
    0.85
    EP 883 learning finished: 0.0019714332687392755, 0.0015379837000410272
    884
    EP 884 simulation finished, start learning
    1.92
    EP 884 learning finished: 0.0019402683736480242, 0.0015799395309989233
    885
    EP 885 simulation finished, start learning
    1.94
    EP 885 learning finished: 0.004091583525333675, 0.0017435479348210474
    886
    EP 886 simulation finished, start learning
    2.07
    EP 886 learning finished: 0.00530994455948175, 0.002238771507004541
    887
    EP 887 simulation finished, start learning
    0.9
    EP 887 learning finished: 0.0051532424002047605, 0.0022025053403507875
    888
    EP 888 simulation finished, start learning
    0.81
    EP 888 learning finished: 0.004051780355045054, 0.001916572149675962
    889
    EP 889 simulation finished, start learning
    1.31
    EP 889 learning finished: 0.003488918220396542, 0.001818591783134001
    890
    EP 890 simulation finished, start learning
    1.32
    EP 890 learning finished: 0.0019306170028805525, 0.0014244242284323267
    891
    EP 891 simulation finished, start learning
    1.2
    EP 891 learning finished: 0.0031542317217798654, 0.0016792321192988051
    892
    EP 892 simulation finished, start learning
    1.02
    EP 892 learning finished: 0.0034320002950334327, 0.001990037660558978
    893
    EP 893 simulation finished, start learning
    1.39
    EP 893 learning finished: 0.0030865025387960543, 0.0016986585328388297
    894
    EP 894 simulation finished, start learning
    1.52
    EP 894 learning finished: 0.003021370586630212, 0.001712455556936258
    895
    EP 895 simulation finished, start learning
    1.62
    EP 895 learning finished: 0.0019407550568999454, 0.001568727543909058
    896
    EP 896 simulation finished, start learning
    1.19
    EP 896 learning finished: 0.001937725890538644, 0.0013289873134970283
    897
    EP 897 simulation finished, start learning
    1.33
    EP 897 learning finished: 0.0018443656668551967, 0.0012798114334409878
    898
    EP 898 simulation finished, start learning
    1.89
    EP 898 learning finished: 0.0015994239468959393, 0.001154519388877932
    899
    EP 899 simulation finished, start learning
    1.92
    EP 899 learning finished: 0.00176173483718145, 0.0010788862752152777
    900
    EP 900 simulation finished, start learning
    1.2
    EP 900 learning finished: 0.0018744046898291344, 0.0012053098712696553
    901
    EP 901 simulation finished, start learning
    1.49
    EP 901 learning finished: 0.0020523609472640203, 0.0014423968698965932
    902
    EP 902 simulation finished, start learning
    0.81
    EP 902 learning finished: 0.0019258232598282632, 0.0013439954157030698
    903
    EP 903 simulation finished, start learning
    0.92
    EP 903 learning finished: 0.0030177180605690523, 0.0014294631497430526
    904
    EP 904 simulation finished, start learning
    0.72
    EP 904 learning finished: 0.0019490350918225374, 0.0012097619934913577
    905
    EP 905 simulation finished, start learning
    1.43
    EP 905 learning finished: 0.0017097846025144747, 0.0009870064601719501
    906
    EP 906 simulation finished, start learning
    1.39
    EP 906 learning finished: 0.0016719068884221144, 0.00102970564522753
    907
    EP 907 simulation finished, start learning
    1.9
    EP 907 learning finished: 0.0015632825566882502, 0.0010209245235370072
    908
    EP 908 simulation finished, start learning
    0.6
    EP 908 learning finished: 0.0015966666427900122, 0.0011218118914841924
    909
    EP 909 simulation finished, start learning
    1.59
    EP 909 learning finished: 0.001642519912881113, 0.0014014273632376728
    910
    EP 910 simulation finished, start learning
    1.2
    EP 910 learning finished: 0.002001097795485407, 0.0015419375662938013
    911
    EP 911 simulation finished, start learning
    0.89
    EP 911 learning finished: 0.004144022409022341, 0.0017831663372570954
    912
    EP 912 simulation finished, start learning
    1.02
    EP 912 learning finished: 0.0034175402126876975, 0.0019281701386256362
    913
    EP 913 simulation finished, start learning
    1.35
    EP 913 learning finished: 0.0039496875712536905, 0.001760519042271665
    914
    EP 914 simulation finished, start learning
    1.24
    EP 914 learning finished: 0.003938577845396196, 0.001763504092020381
    915
    EP 915 simulation finished, start learning
    1.62
    EP 915 learning finished: 0.001766770290430486, 0.0014928048013983879
    916
    EP 916 simulation finished, start learning
    0.93
    EP 916 learning finished: 0.001793646334633081, 0.0012457523208310544
    917
    EP 917 simulation finished, start learning
    1.41
    EP 917 learning finished: 0.00280596137965361, 0.0014083650617345473
    918
    EP 918 simulation finished, start learning
    1.38
    EP 918 learning finished: 0.0018155451744199524, 0.001504092385970009
    919
    EP 919 simulation finished, start learning
    1.42
    EP 919 learning finished: 0.001843377887155384, 0.001476802448898103
    920
    EP 920 simulation finished, start learning
    1.41
    EP 920 learning finished: 0.0028819241437574306, 0.001702138670131544
    921
    EP 921 simulation finished, start learning
    1.11
    EP 921 learning finished: 0.0035293088841434894, 0.0016289936633431996
    922
    EP 922 simulation finished, start learning
    1.83
    EP 922 learning finished: 0.0018347028200054183, 0.001524476138008879
    923
    EP 923 simulation finished, start learning
    1.34
    EP 923 learning finished: 0.003491771213283812, 0.0017978422180657816
    924
    EP 924 simulation finished, start learning
    1.81
    EP 924 learning finished: 0.003517849992196176, 0.0016655994312981075
    925
    EP 925 simulation finished, start learning
    1.94
    EP 925 learning finished: 0.0035143415672198586, 0.0017415193115759564
    926
    EP 926 simulation finished, start learning
    1.19
    EP 926 learning finished: 0.005197876971999804, 0.002001528492820389
    927
    EP 927 simulation finished, start learning
    1.23
    EP 927 learning finished: 0.0034361166427032686, 0.0016996735968287287
    928
    EP 928 simulation finished, start learning
    1.4
    EP 928 learning finished: 0.0033640482023784557, 0.0016681767102605012
    929
    EP 929 simulation finished, start learning
    0.92
    EP 929 learning finished: 0.0019159098311156697, 0.0014508889934872944
    930
    EP 930 simulation finished, start learning
    2.01
    EP 930 learning finished: 0.0019137006743790153, 0.0012664476634052235
    931
    EP 931 simulation finished, start learning
    1.83
    EP 931 learning finished: 0.002912157895103462, 0.00153677717125363
    932
    EP 932 simulation finished, start learning
    1.53
    EP 932 learning finished: 0.002906658572470633, 0.0015159461521700261
    933
    EP 933 simulation finished, start learning
    1.39
    EP 933 learning finished: 0.001968117427068705, 0.0013757192596024481
    934
    EP 934 simulation finished, start learning
    1.5
    EP 934 learning finished: 0.0019020717440447754, 0.0013407415038297898
    935
    EP 935 simulation finished, start learning
    1.72
    EP 935 learning finished: 0.0019312654404544105, 0.0012943995011400016
    936
    EP 936 simulation finished, start learning
    1.42
    EP 936 learning finished: 0.0019916473937868773, 0.00131118120997827
    937
    EP 937 simulation finished, start learning
    1.61
    EP 937 learning finished: 0.004072886551675481, 0.0016885981508443966
    938
    EP 938 simulation finished, start learning
    1.52
    EP 938 learning finished: 0.0029807047517138585, 0.0017438636148785491
    939
    EP 939 simulation finished, start learning
    1.01
    EP 939 learning finished: 0.003279482283748081, 0.0019681674370698823
    940
    EP 940 simulation finished, start learning
    2.12
    EP 940 learning finished: 0.0033019525114532655, 0.0020087246335454743
    941
    EP 941 simulation finished, start learning
    1.18
    EP 941 learning finished: 0.0033572423501695425, 0.0016453000866552732
    942
    EP 942 simulation finished, start learning
    1.5
    EP 942 learning finished: 0.0015965266285391522, 0.0013563358637505019
    943
    EP 943 simulation finished, start learning
    1.62
    EP 943 learning finished: 0.0020379844595019847, 0.0011095099192018527
    944
    EP 944 simulation finished, start learning
    1.08
    EP 944 learning finished: 0.0015282864321921608, 0.001128154380250675
    945
    EP 945 simulation finished, start learning
    1.8
    EP 945 learning finished: 0.0017660312578620108, 0.001396182466089968
    946
    EP 946 simulation finished, start learning
    1.37
    EP 946 learning finished: 0.001972829070474802, 0.0015648631714834476
    947
    EP 947 simulation finished, start learning
    1.29
    EP 947 learning finished: 0.002003312124646328, 0.0015356442162639539
    948
    EP 948 simulation finished, start learning
    1.01
    EP 948 learning finished: 0.0015095809231731039, 0.001343105432939436
    949
    EP 949 simulation finished, start learning
    1.63
    EP 949 learning finished: 0.0019063988328404946, 0.001285344165003366
    950
    EP 950 simulation finished, start learning
    1.48
    EP 950 learning finished: 0.0015260041486744313, 0.0011021276881297837
    951
    EP 951 simulation finished, start learning
    0.7
    EP 951 learning finished: 0.0018397427831864377, 0.0012125805256282263
    952
    EP 952 simulation finished, start learning
    0.81
    EP 952 learning finished: 0.0018693876452253733, 0.001191389797845351
    953
    EP 953 simulation finished, start learning
    1.5
    EP 953 learning finished: 0.0018988977119889228, 0.0011888454133561766
    954
    EP 954 simulation finished, start learning
    1.29
    EP 954 learning finished: 0.001897011853738917, 0.0011479694090305397
    955
    EP 955 simulation finished, start learning
    0.9
    EP 955 learning finished: 0.001623633503571753, 0.0009610869184989804
    956
    EP 956 simulation finished, start learning
    1.38
    EP 956 learning finished: 0.0018619117433722465, 0.0012004885352775811
    957
    EP 957 simulation finished, start learning
    1.0
    EP 957 learning finished: 0.0016359689604890093, 0.0009864566309204885
    958
    EP 958 simulation finished, start learning
    0.92
    EP 958 learning finished: 0.0016669338621504437, 0.0009370056986931866
    959
    EP 959 simulation finished, start learning
    2.13
    EP 959 learning finished: 0.0018258010394163193, 0.0012234496645386155
    960
    EP 960 simulation finished, start learning
    1.8
    EP 960 learning finished: 0.0017918318782648157, 0.0014885033871205877
    961
    EP 961 simulation finished, start learning
    1.13
    EP 961 learning finished: 0.002730218691691988, 0.0016548528060977101
    962
    EP 962 simulation finished, start learning
    2.1
    EP 962 learning finished: 0.0032347279071167634, 0.00205463215825219
    963
    EP 963 simulation finished, start learning
    0.88
    EP 963 learning finished: 0.00381662904412039, 0.0021054303739672284
    964
    EP 964 simulation finished, start learning
    1.64
    EP 964 learning finished: 0.0032179404857348715, 0.0019096326104153978
    965
    EP 965 simulation finished, start learning
    1.7
    EP 965 learning finished: 0.0032978030476655817, 0.0018021159717607205
    966
    EP 966 simulation finished, start learning
    1.61
    EP 966 learning finished: 0.001769616591751344, 0.001492573510110932
    967
    EP 967 simulation finished, start learning
    1.29
    EP 967 learning finished: 0.003169689717836377, 0.0016474983095632705
    968
    EP 968 simulation finished, start learning
    1.34
    EP 968 learning finished: 0.001707169226513372, 0.0013410804502168954
    969
    EP 969 simulation finished, start learning
    1.23
    EP 969 learning finished: 0.003096480353882214, 0.0015622149649295498
    970
    EP 970 simulation finished, start learning
    1.53
    EP 970 learning finished: 0.0031459043849544344, 0.0016878398613654848
    971
    EP 971 simulation finished, start learning
    1.14
    EP 971 learning finished: 0.0015100342919680543, 0.001217566922860532
    972
    EP 972 simulation finished, start learning
    0.69
    EP 972 learning finished: 0.0017598598306456381, 0.0013351558561263133
    973
    EP 973 simulation finished, start learning
    1.12
    EP 973 learning finished: 0.0027476462887001674, 0.0012758885974069045
    974
    EP 974 simulation finished, start learning
    1.19
    EP 974 learning finished: 0.0028074804009570187, 0.0012484596669729935
    975
    EP 975 simulation finished, start learning
    1.41
    EP 975 learning finished: 0.005798658482714103, 0.00199228839436986
    976
    EP 976 simulation finished, start learning
    1.21
    EP 976 learning finished: 0.004800439425221015, 0.001889031606428541
    977
    EP 977 simulation finished, start learning
    1.32
    EP 977 learning finished: 0.0047576693521911295, 0.001888360854861067
    978
    EP 978 simulation finished, start learning
    1.32
    EP 978 learning finished: 0.0036635101657402913, 0.0017201730643735165
    979
    EP 979 simulation finished, start learning
    1.84
    EP 979 learning finished: 0.0015390882035362097, 0.0010553617664944188
    980
    EP 980 simulation finished, start learning
    0.72
    EP 980 learning finished: 0.0014168453124376867, 0.0010218053306922197
    981
    EP 981 simulation finished, start learning
    1.82
    EP 981 learning finished: 0.0015202390325939064, 0.000985037904629375
    982
    EP 982 simulation finished, start learning
    1.32
    EP 982 learning finished: 0.0016237276330421935, 0.0009987768853149486
    983
    EP 983 simulation finished, start learning
    0.99
    EP 983 learning finished: 0.0014097075143002513, 0.00103137872084987
    984
    EP 984 simulation finished, start learning
    2.12
    EP 984 learning finished: 0.0017880538448835637, 0.0014959732218612725
    985
    EP 985 simulation finished, start learning
    1.71
    EP 985 learning finished: 0.003094529766645593, 0.0019127001066001625
    986
    EP 986 simulation finished, start learning
    2.01
    EP 986 learning finished: 0.00311535020577125, 0.0019725018137792816
    987
    EP 987 simulation finished, start learning
    2.0
    EP 987 learning finished: 0.0031366246234050266, 0.001909564973957482
    988
    EP 988 simulation finished, start learning
    0.92
    EP 988 learning finished: 0.003159767428215332, 0.0016365440298820033
    989
    EP 989 simulation finished, start learning
    1.65
    EP 989 learning finished: 0.0017718719256585302, 0.0012445599292944347
    990
    EP 990 simulation finished, start learning
    1.31
    EP 990 learning finished: 0.003209656426939489, 0.0014527160092117815
    991
    EP 991 simulation finished, start learning
    1.93
    EP 991 learning finished: 0.003264851133916835, 0.0014592642543463559
    992
    EP 992 simulation finished, start learning
    1.57
    EP 992 learning finished: 0.0016462871526279343, 0.0012865047157844718
    993
    EP 993 simulation finished, start learning
    1.2
    EP 993 learning finished: 0.0016162665404770848, 0.0012429568511942435
    994
    EP 994 simulation finished, start learning
    1.32
    EP 994 learning finished: 0.0015994060879940264, 0.0012828074840109949
    995
    EP 995 simulation finished, start learning
    1.4
    EP 995 learning finished: 0.0016104698254792993, 0.001395891312028219
    996
    EP 996 simulation finished, start learning
    1.23
    EP 996 learning finished: 0.003042650767319799, 0.0016421624577095766
    997
    EP 997 simulation finished, start learning
    1.81
    EP 997 learning finished: 0.0030039541328896746, 0.0018432934278195627
    998
    EP 998 simulation finished, start learning
    0.99
    EP 998 learning finished: 0.0018624682336856324, 0.0014082796588654745
    999
    EP 999 simulation finished, start learning
    1.29
    EP 999 learning finished: 0.0029595472839005765, 0.0015098791929168225



```python
v_model, pi_model= q_model.q.yield_v_pi()
```


```python
v_model
```




    0    18.249646
    1    19.498898
    2    27.241977
    dtype: float64




```python
pi_model
```




    0    1
    1    1
    2   -1
    dtype: int64




```python

```


```python
for s in state_space:
    for a in control_space:
        print(f'state: {s}, action_buffer: {[a]}, action to take: {q_model.q.max_value_act(model(s, [a]))[1]}')
        
def pi_model(i):
    s, a = i
    return q_model.q.max_value_act(model_model(s, [a]))[1]
```

    state: 0, action_buffer: [-1], action to take: 1
    state: 0, action_buffer: [1], action to take: 1
    state: 1, action_buffer: [-1], action to take: 1
    state: 1, action_buffer: [1], action to take: 1
    state: 2, action_buffer: [-1], action to take: 1
    state: 2, action_buffer: [1], action to take: -1



```python
def pi_wrapper(pi):
    def ret(x):
        return pi[x]
    return ret

def eval_pi_model(env, pi, model, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = pi(model(obs, action_buffer))
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        
    return np.mean(r_ls), r_ls.mean(axis = 1)

def cummean(arr):
    return np.cumsum(arr)/np.arange(1, len(arr)+1)
```


```python
val, v_samples_lin = eval_pi_model(env, pi_wrapper(pi_l), model, 0.95)
val
```




    0.25403037079411306




```python
exp2_model = v_samples_lin
```


```python
sns.distplot(v_samples_lin, label = 'model')
plt.legend()
```


```python
val, v_samples_lin = eval_pi_model(env, lambda x: np.random.choice([-1,1]), model, 0.95)
val
```




    -0.005780531808926421




```python
exp2_random = v_samples_lin
```


```python
sns.distplot(v_samples_lin, label = 'random benchmark')
plt.legend()
```


```python
# Q-learning using model, async reward collection and optimal action selection



class Q_learning:
    def __init__(self, q, env, delay_step, model, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(400)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        model = self.model
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            for _ in range(100):
                
                
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (get_opt_act(q, model, obs, self.action_memory), False)
                last_action = self.action_memory.pop(0)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q

def get_opt_act(q, model, x, acts):
    act_idx = np.argmax([np.sum([q[s, u]*model(s, x, acts) for s in q.state_space]) for u in q.control_space])
    return q.control_space[act_idx]
```


```python
# P(s|x, a_1, a_2, ...)
def model(s, x, acts):
    return trans_prob(s, x, acts[0])
```


```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
# lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_opt = Q_learning(q, env, 1, model, MAX_ITER = 1000)
learned_q = q_opt.train(lr_fn)
```

    0
    1
    2
    3
    EP 3 simulation finished, start learning
    -0.77
    EP 3 learning finished: 13.162958545093007, 6.382037234246667
    4
    EP 4 simulation finished, start learning
    1.34
    EP 4 learning finished: 4.83825463967513, 2.952703893906135
    5
    EP 5 simulation finished, start learning
    1.01
    EP 5 learning finished: 3.4173293874326385, 1.5549239122405396
    6
    EP 6 simulation finished, start learning
    1.13
    EP 6 learning finished: 1.3364963939440502, 0.8781276785358122
    7
    EP 7 simulation finished, start learning
    1.59
    EP 7 learning finished: 0.7775297227735471, 0.5661839604892069
    8
    EP 8 simulation finished, start learning
    1.02
    EP 8 learning finished: 0.8820166604487643, 0.4675474574429332
    9
    EP 9 simulation finished, start learning
    1.53
    EP 9 learning finished: 0.3757226805749063, 0.32969333057408096
    10
    EP 10 simulation finished, start learning
    1.09
    EP 10 learning finished: 0.29129878850149726, 0.27344399322143415
    11
    EP 11 simulation finished, start learning
    1.41
    EP 11 learning finished: 0.3802110082493275, 0.26081816691671955
    12
    EP 12 simulation finished, start learning
    1.23
    EP 12 learning finished: 0.3386835070100247, 0.1971958731566105
    13
    EP 13 simulation finished, start learning
    1.45
    EP 13 learning finished: 0.47571564771357977, 0.20534255112552646
    14
    EP 14 simulation finished, start learning
    1.11
    EP 14 learning finished: 0.38503897258815023, 0.20630054021901945
    15
    EP 15 simulation finished, start learning
    1.4
    EP 15 learning finished: 0.23944660388822214, 0.16731717893530154
    16
    EP 16 simulation finished, start learning
    0.93
    EP 16 learning finished: 0.18714045456970752, 0.1426425919653352
    17
    EP 17 simulation finished, start learning
    1.53
    EP 17 learning finished: 0.1750113241657747, 0.12181685161099558
    18
    EP 18 simulation finished, start learning
    1.17
    EP 18 learning finished: 0.16517463276441546, 0.09231591625169895
    19
    EP 19 simulation finished, start learning
    1.18
    EP 19 learning finished: 0.20313917399666437, 0.10479227685253623
    20
    EP 20 simulation finished, start learning
    1.39
    EP 20 learning finished: 0.2315450798319194, 0.1081141447149836
    21
    EP 21 simulation finished, start learning
    1.41
    EP 21 learning finished: 0.2456670661152156, 0.1254832157655089
    22
    EP 22 simulation finished, start learning
    1.24
    EP 22 learning finished: 0.26752222997478015, 0.12271724700801911
    23
    EP 23 simulation finished, start learning
    1.5
    EP 23 learning finished: 0.17533796339330365, 0.10310629070441883
    24
    EP 24 simulation finished, start learning
    1.32
    EP 24 learning finished: 0.14175188127006422, 0.09515112830649297
    25
    EP 25 simulation finished, start learning
    1.14
    EP 25 learning finished: 0.12993132498526627, 0.07206429515758088
    26
    EP 26 simulation finished, start learning
    1.42
    EP 26 learning finished: 0.09884792334635506, 0.06688031839491693
    27
    EP 27 simulation finished, start learning
    0.81
    EP 27 learning finished: 0.10180679498052214, 0.06272723264884543
    28
    EP 28 simulation finished, start learning
    1.43
    EP 28 learning finished: 0.10640246628868155, 0.061619218319676726
    29
    EP 29 simulation finished, start learning
    1.41
    EP 29 learning finished: 0.08898923726314578, 0.059754375660407234
    30
    EP 30 simulation finished, start learning
    1.92
    EP 30 learning finished: 0.08815127736246353, 0.06955957817581677
    31
    EP 31 simulation finished, start learning
    1.51
    EP 31 learning finished: 0.08372513512260404, 0.06681162980612321
    32
    EP 32 simulation finished, start learning
    1.22
    EP 32 learning finished: 0.08281686975163716, 0.06318920373817309
    33
    EP 33 simulation finished, start learning
    1.41
    EP 33 learning finished: 0.07435935592393328, 0.05335079193418283
    34
    EP 34 simulation finished, start learning
    1.23
    EP 34 learning finished: 0.07481996057540741, 0.04381766189694147
    35
    EP 35 simulation finished, start learning
    1.7
    EP 35 learning finished: 0.0729345549250251, 0.05344476755774963
    36
    EP 36 simulation finished, start learning
    1.11
    EP 36 learning finished: 0.13041029284908312, 0.061752280139883396
    37
    EP 37 simulation finished, start learning
    1.11
    EP 37 learning finished: 0.12198842788888165, 0.05918745869189893
    38
    EP 38 simulation finished, start learning
    1.57
    EP 38 learning finished: 0.11009064668750135, 0.05300965671456422
    39
    EP 39 simulation finished, start learning
    1.21
    EP 39 learning finished: 0.0654316037771352, 0.045989005948117466
    40
    EP 40 simulation finished, start learning
    0.94
    EP 40 learning finished: 0.06583329903834922, 0.03710847360257515
    41
    EP 41 simulation finished, start learning
    2.33
    EP 41 learning finished: 0.059708439318873374, 0.039575958263154
    42
    EP 42 simulation finished, start learning
    1.53
    EP 42 learning finished: 0.08822171632003162, 0.0507435884016895
    43
    EP 43 simulation finished, start learning
    1.43
    EP 43 learning finished: 0.09038243644630484, 0.05036516247915331
    44
    EP 44 simulation finished, start learning
    1.53
    EP 44 learning finished: 0.10441830865711488, 0.06027403301545468
    45
    EP 45 simulation finished, start learning
    1.12
    EP 45 learning finished: 0.09122869619496399, 0.054309556525601245
    46
    EP 46 simulation finished, start learning
    1.32
    EP 46 learning finished: 0.06682143395566342, 0.042057865643931734
    47
    EP 47 simulation finished, start learning
    1.92
    EP 47 learning finished: 0.04690975240140283, 0.038389505800503265
    48
    EP 48 simulation finished, start learning
    0.93
    EP 48 learning finished: 0.04793187772642682, 0.026994853444932392
    49
    EP 49 simulation finished, start learning
    1.32
    EP 49 learning finished: 0.046304697668874795, 0.0321418439127689
    50
    EP 50 simulation finished, start learning
    1.68
    EP 50 learning finished: 0.04413570770431363, 0.03520674097930184
    51
    EP 51 simulation finished, start learning
    0.93
    EP 51 learning finished: 0.06014756751793904, 0.03623804933777711
    52
    EP 52 simulation finished, start learning
    2.31
    EP 52 learning finished: 0.07724246001586899, 0.04073570050693457
    53
    EP 53 simulation finished, start learning
    1.83
    EP 53 learning finished: 0.11735997932659892, 0.04315743874126188
    54
    EP 54 simulation finished, start learning
    1.11
    EP 54 learning finished: 0.09587882971913153, 0.03762603360371308
    55
    EP 55 simulation finished, start learning
    2.01
    EP 55 learning finished: 0.06949122599685253, 0.04026063254991108
    56
    EP 56 simulation finished, start learning
    0.91
    EP 56 learning finished: 0.0686571028753793, 0.03733052101887768
    57
    EP 57 simulation finished, start learning
    1.02
    EP 57 learning finished: 0.039612223496765964, 0.028853166393878027
    58
    EP 58 simulation finished, start learning
    1.31
    EP 58 learning finished: 0.037921188618811996, 0.028163475741972494
    59
    EP 59 simulation finished, start learning
    1.71
    EP 59 learning finished: 0.04009889487000429, 0.02680063447984467
    60
    EP 60 simulation finished, start learning
    1.14
    EP 60 learning finished: 0.05462613969329411, 0.031105952571818918
    61
    EP 61 simulation finished, start learning
    1.14
    EP 61 learning finished: 0.05343794146946301, 0.030218293265669754
    62
    EP 62 simulation finished, start learning
    0.91
    EP 62 learning finished: 0.05239306358976137, 0.02918772828818768
    63
    EP 63 simulation finished, start learning
    1.31
    EP 63 learning finished: 0.04094630750211081, 0.02594282972988034
    64
    EP 64 simulation finished, start learning
    2.01
    EP 64 learning finished: 0.03497506118192817, 0.020307972204410756
    65
    EP 65 simulation finished, start learning
    1.23
    EP 65 learning finished: 0.034271852895701116, 0.020156781183085393
    66
    EP 66 simulation finished, start learning
    0.89
    EP 66 learning finished: 0.03363214972080897, 0.01957902442932576
    67
    EP 67 simulation finished, start learning
    0.54
    EP 67 learning finished: 0.07146697950512504, 0.027065347925775935
    68
    EP 68 simulation finished, start learning
    1.33
    EP 68 learning finished: 0.06926649165526655, 0.02547474021053168
    69
    EP 69 simulation finished, start learning
    1.91
    EP 69 learning finished: 0.06708690380222748, 0.0260896713264831
    70
    EP 70 simulation finished, start learning
    1.92
    EP 70 learning finished: 0.06504885008171435, 0.030031647501504406
    71
    EP 71 simulation finished, start learning
    1.71
    EP 71 learning finished: 0.04574556472370084, 0.024730905985661522
    72
    EP 72 simulation finished, start learning
    1.52
    EP 72 learning finished: 0.044855390681250995, 0.026863273867407205
    73
    EP 73 simulation finished, start learning
    1.35
    EP 73 learning finished: 0.04441864527050399, 0.02578502482612303
    74
    EP 74 simulation finished, start learning
    1.11
    EP 74 learning finished: 0.04378083587338466, 0.024614465028658827
    75
    EP 75 simulation finished, start learning
    1.44
    EP 75 learning finished: 0.03502677325707637, 0.02280554774184444
    76
    EP 76 simulation finished, start learning
    1.61
    EP 76 learning finished: 0.04192017051217789, 0.023737853268401093
    77
    EP 77 simulation finished, start learning
    1.32
    EP 77 learning finished: 0.04109050697487859, 0.0233017442240057
    78
    EP 78 simulation finished, start learning
    1.39
    EP 78 learning finished: 0.02702787396330386, 0.01991143447926221
    79
    EP 79 simulation finished, start learning
    1.68
    EP 79 learning finished: 0.027766322145460265, 0.01762786073722437
    80
    EP 80 simulation finished, start learning
    1.17
    EP 80 learning finished: 0.028754911139023065, 0.014254542759946531
    81
    EP 81 simulation finished, start learning
    1.41
    EP 81 learning finished: 0.028212752286602694, 0.013900725106663478
    82
    EP 82 simulation finished, start learning
    0.93
    EP 82 learning finished: 0.027985222051166048, 0.01337277743082499
    83
    EP 83 simulation finished, start learning
    1.31
    EP 83 learning finished: 0.027203037737383085, 0.0128631995139085
    84
    EP 84 simulation finished, start learning
    1.53
    EP 84 learning finished: 0.025391115736169922, 0.016079303790714167
    85
    EP 85 simulation finished, start learning
    2.01
    EP 85 learning finished: 0.024193403203462083, 0.016506318738863435
    86
    EP 86 simulation finished, start learning
    1.94
    EP 86 learning finished: 0.021721480000236326, 0.016934487147556965
    87
    EP 87 simulation finished, start learning
    1.41
    EP 87 learning finished: 0.021572538425861154, 0.01858012525148922
    88
    EP 88 simulation finished, start learning
    1.61
    EP 88 learning finished: 0.02685692359350611, 0.017590295825604823
    89
    EP 89 simulation finished, start learning
    1.25
    EP 89 learning finished: 0.029431222101628407, 0.019895582004288315
    90
    EP 90 simulation finished, start learning
    2.02
    EP 90 learning finished: 0.0407847272655566, 0.021665043114787192
    91
    EP 91 simulation finished, start learning
    0.74
    EP 91 learning finished: 0.053424452652773624, 0.022954157606016306
    92
    EP 92 simulation finished, start learning
    1.22
    EP 92 learning finished: 0.0415596163900851, 0.02042010821706815
    93
    EP 93 simulation finished, start learning
    1.82
    EP 93 learning finished: 0.03922633502381245, 0.017982312490121426
    94
    EP 94 simulation finished, start learning
    1.38
    EP 94 learning finished: 0.026778481766996265, 0.015323004478521677
    95
    EP 95 simulation finished, start learning
    1.53
    EP 95 learning finished: 0.020817917715946876, 0.011641451709597739
    96
    EP 96 simulation finished, start learning
    1.11
    EP 96 learning finished: 0.020769306701675205, 0.015604889371715261
    97
    EP 97 simulation finished, start learning
    1.74
    EP 97 learning finished: 0.02026756956407283, 0.015179726727874451
    98
    EP 98 simulation finished, start learning
    1.23
    EP 98 learning finished: 0.03468875461443233, 0.018442961195729524
    99
    EP 99 simulation finished, start learning
    1.89
    EP 99 learning finished: 0.03414219437386734, 0.01872625385537881
    100
    EP 100 simulation finished, start learning
    1.71
    EP 100 learning finished: 0.023222889071892183, 0.01733894623031862
    101
    EP 101 simulation finished, start learning
    1.04
    EP 101 learning finished: 0.03405611082129667, 0.01844156487757065
    102
    EP 102 simulation finished, start learning
    1.29
    EP 102 learning finished: 0.0328542423674687, 0.0193463483809575
    103
    EP 103 simulation finished, start learning
    1.23
    EP 103 learning finished: 0.03300916300229062, 0.02008560163256708
    104
    EP 104 simulation finished, start learning
    1.62
    EP 104 learning finished: 0.04295893698337494, 0.018817279477374882
    105
    EP 105 simulation finished, start learning
    1.5
    EP 105 learning finished: 0.03162348565866324, 0.01987450869804726
    106
    EP 106 simulation finished, start learning
    1.62
    EP 106 learning finished: 0.021989834967428123, 0.015668459267629053
    107
    EP 107 simulation finished, start learning
    1.31
    EP 107 learning finished: 0.017391017240553808, 0.014070876685631006
    108
    EP 108 simulation finished, start learning
    1.01
    EP 108 learning finished: 0.02995827708749843, 0.015861494319043896
    109
    EP 109 simulation finished, start learning
    1.01
    EP 109 learning finished: 0.020435040705962138, 0.014184044690040606
    110
    EP 110 simulation finished, start learning
    1.49
    EP 110 learning finished: 0.019426142737408725, 0.013935229449702232
    111
    EP 111 simulation finished, start learning
    2.15
    EP 111 learning finished: 0.028740492988195143, 0.015897709549821393
    112
    EP 112 simulation finished, start learning
    1.71
    EP 112 learning finished: 0.019271457810972947, 0.012743404046101858
    113
    EP 113 simulation finished, start learning
    1.31
    EP 113 learning finished: 0.017329298350482247, 0.01160190248321887
    114
    EP 114 simulation finished, start learning
    1.71
    EP 114 learning finished: 0.01860564158796052, 0.013892186037980517
    115
    EP 115 simulation finished, start learning
    1.11
    EP 115 learning finished: 0.016678488673374403, 0.011556028762832385
    116
    EP 116 simulation finished, start learning
    1.52
    EP 116 learning finished: 0.016861355073796602, 0.011226979553106512
    117
    EP 117 simulation finished, start learning
    1.3
    EP 117 learning finished: 0.01684342200847233, 0.012519120192669616
    118
    EP 118 simulation finished, start learning
    1.8
    EP 118 learning finished: 0.01814782595580411, 0.011637149980154563
    119
    EP 119 simulation finished, start learning
    1.34
    EP 119 learning finished: 0.018478420649145733, 0.013686423457535568
    120
    EP 120 simulation finished, start learning
    1.81
    EP 120 learning finished: 0.018301706690891706, 0.013853294414451156
    121
    EP 121 simulation finished, start learning
    1.6
    EP 121 learning finished: 0.016074308226336598, 0.012531917175435817
    122
    EP 122 simulation finished, start learning
    1.51
    EP 122 learning finished: 0.015000901278430234, 0.010923184879363093
    123
    EP 123 simulation finished, start learning
    1.3
    EP 123 learning finished: 0.015222702808122435, 0.00908405042361012
    124
    EP 124 simulation finished, start learning
    1.14
    EP 124 learning finished: 0.01828476080631347, 0.01127859354971831
    125
    EP 125 simulation finished, start learning
    1.03
    EP 125 learning finished: 0.01851925994751724, 0.010818798165530552
    126
    EP 126 simulation finished, start learning
    1.23
    EP 126 learning finished: 0.026500170629986997, 0.013542617791676648
    127
    EP 127 simulation finished, start learning
    1.44
    EP 127 learning finished: 0.02590761678790443, 0.013289749979632207
    128
    EP 128 simulation finished, start learning
    2.1
    EP 128 learning finished: 0.025263037928747067, 0.013377040747962829
    129
    EP 129 simulation finished, start learning
    2.3
    EP 129 learning finished: 0.025329627405543675, 0.014143556233181087
    130
    EP 130 simulation finished, start learning
    1.74
    EP 130 learning finished: 0.02577903240768542, 0.014365624199826676
    131
    EP 131 simulation finished, start learning
    1.78
    EP 131 learning finished: 0.025347718304240985, 0.014581067353288013
    132
    EP 132 simulation finished, start learning
    1.5
    EP 132 learning finished: 0.025154618353598224, 0.014094589775091334
    133
    EP 133 simulation finished, start learning
    1.23
    EP 133 learning finished: 0.024942321108074594, 0.013271241348867887
    134
    EP 134 simulation finished, start learning
    1.74
    EP 134 learning finished: 0.013046911419696272, 0.009825267110838531
    135
    EP 135 simulation finished, start learning
    0.89
    EP 135 learning finished: 0.024269664610477193, 0.011249227352344052
    136
    EP 136 simulation finished, start learning
    1.34
    EP 136 learning finished: 0.02334982376047634, 0.012149086201057221
    137
    EP 137 simulation finished, start learning
    1.13
    EP 137 learning finished: 0.03418572995709113, 0.013847092910679907
    138
    EP 138 simulation finished, start learning
    1.02
    EP 138 learning finished: 0.032990842174223545, 0.01442151049218913
    139
    EP 139 simulation finished, start learning
    1.8
    EP 139 learning finished: 0.03215744080385363, 0.014791802376408055
    140
    EP 140 simulation finished, start learning
    1.3
    EP 140 learning finished: 0.020195253694108573, 0.011450924582294242
    141
    EP 141 simulation finished, start learning
    1.61
    EP 141 learning finished: 0.012411733609948072, 0.009699051339709127
    142
    EP 142 simulation finished, start learning
    0.7
    EP 142 learning finished: 0.012743717860537629, 0.00828982805452553
    143
    EP 143 simulation finished, start learning
    1.44
    EP 143 learning finished: 0.012629672584333562, 0.0070428636771685165
    144
    EP 144 simulation finished, start learning
    1.33
    EP 144 learning finished: 0.01254987099749627, 0.00836942783430672
    145
    EP 145 simulation finished, start learning
    1.32
    EP 145 learning finished: 0.0165391454502668, 0.009305891117358023
    146
    EP 146 simulation finished, start learning
    1.32
    EP 146 learning finished: 0.016653691913472102, 0.010748411610362515
    147
    EP 147 simulation finished, start learning
    1.54
    EP 147 learning finished: 0.03817405488662473, 0.014232771282902235
    148
    EP 148 simulation finished, start learning
    1.21
    EP 148 learning finished: 0.03691403877438759, 0.013963543900533004
    149
    EP 149 simulation finished, start learning
    0.63
    EP 149 learning finished: 0.029288948802083326, 0.012311880386347637
    150
    EP 150 simulation finished, start learning
    1.2
    EP 150 learning finished: 0.03554177354125798, 0.012206638953812169
    151
    EP 151 simulation finished, start learning
    0.81
    EP 151 learning finished: 0.014281620888784374, 0.009260254982773239
    152
    EP 152 simulation finished, start learning
    1.5
    EP 152 learning finished: 0.018195283888712055, 0.010956314114399568
    153
    EP 153 simulation finished, start learning
    1.11
    EP 153 learning finished: 0.029040838249059675, 0.013915071017279068
    154
    EP 154 simulation finished, start learning
    0.91
    EP 154 learning finished: 0.02847584498770317, 0.013614233508806528
    155
    EP 155 simulation finished, start learning
    1.71
    EP 155 learning finished: 0.026852433894497096, 0.013370358077857544
    156
    EP 156 simulation finished, start learning
    1.11
    EP 156 learning finished: 0.01990566355389589, 0.010379957331832834
    157
    EP 157 simulation finished, start learning
    1.21
    EP 157 learning finished: 0.013193396128102108, 0.008379276354817536
    158
    EP 158 simulation finished, start learning
    1.21
    EP 158 learning finished: 0.011477815235375388, 0.007276154696035597
    159
    EP 159 simulation finished, start learning
    1.61
    EP 159 learning finished: 0.011507486508799403, 0.006074627078066719
    160
    EP 160 simulation finished, start learning
    1.23
    EP 160 learning finished: 0.01297953582019673, 0.009129360620478385
    161
    EP 161 simulation finished, start learning
    1.33
    EP 161 learning finished: 0.012810315993174726, 0.009176631726957337
    162
    EP 162 simulation finished, start learning
    0.93
    EP 162 learning finished: 0.01898312415777781, 0.009832470110768968
    163
    EP 163 simulation finished, start learning
    1.29
    EP 163 learning finished: 0.024964615763252596, 0.010535887629099664
    164
    EP 164 simulation finished, start learning
    1.23
    EP 164 learning finished: 0.01237356864247019, 0.007534464746650353
    165
    EP 165 simulation finished, start learning
    1.79
    EP 165 learning finished: 0.018443043920218116, 0.008950016823060439
    166
    EP 166 simulation finished, start learning
    1.42
    EP 166 learning finished: 0.012221712900757353, 0.008010135338134106
    167
    EP 167 simulation finished, start learning
    0.83
    EP 167 learning finished: 0.010322089765814724, 0.006600456300443962
    168
    EP 168 simulation finished, start learning
    1.08
    EP 168 learning finished: 0.010710757651846592, 0.006605766155725945
    169
    EP 169 simulation finished, start learning
    0.93
    EP 169 learning finished: 0.011200241998002625, 0.00498228776360173
    170
    EP 170 simulation finished, start learning
    1.2
    EP 170 learning finished: 0.011339787282851788, 0.006011688091688505
    171
    EP 171 simulation finished, start learning
    1.65
    EP 171 learning finished: 0.010563895897469422, 0.007410723353869386
    172
    EP 172 simulation finished, start learning
    1.51
    EP 172 learning finished: 0.009921339000669605, 0.007501971190408814
    173
    EP 173 simulation finished, start learning
    1.21
    EP 173 learning finished: 0.011738178470206861, 0.008558150787815558
    174
    EP 174 simulation finished, start learning
    1.51
    EP 174 learning finished: 0.011606983537241433, 0.00742581034144892
    175
    EP 175 simulation finished, start learning
    1.75
    EP 175 learning finished: 0.017229486278043993, 0.00845403718808028
    176
    EP 176 simulation finished, start learning
    1.41
    EP 176 learning finished: 0.02268368789511399, 0.009485174771063143
    177
    EP 177 simulation finished, start learning
    2.06
    EP 177 learning finished: 0.016677029440099744, 0.00900156710683575
    178
    EP 178 simulation finished, start learning
    1.09
    EP 178 learning finished: 0.021995691141860618, 0.009705230478279722
    179
    EP 179 simulation finished, start learning
    1.32
    EP 179 learning finished: 0.010683023506199874, 0.00744898375063574
    180
    EP 180 simulation finished, start learning
    1.12
    EP 180 learning finished: 0.010552708348633644, 0.007243422963593223
    181
    EP 181 simulation finished, start learning
    1.6
    EP 181 learning finished: 0.010637431924276797, 0.006864130290879326
    182
    EP 182 simulation finished, start learning
    1.2
    EP 182 learning finished: 0.015812974464008533, 0.007889240051998906
    183
    EP 183 simulation finished, start learning
    1.13
    EP 183 learning finished: 0.01561996946003319, 0.007880333625989463
    184
    EP 184 simulation finished, start learning
    1.65
    EP 184 learning finished: 0.010138642626507988, 0.008038931900657964
    185
    EP 185 simulation finished, start learning
    1.51
    EP 185 learning finished: 0.010035198270209378, 0.007909929223796
    186
    EP 186 simulation finished, start learning
    1.62
    EP 186 learning finished: 0.008475628822321113, 0.006471127853798701
    187
    EP 187 simulation finished, start learning
    1.41
    EP 187 learning finished: 0.009635670378113304, 0.006621295380189333
    188
    EP 188 simulation finished, start learning
    1.31
    EP 188 learning finished: 0.008895143374781256, 0.005476522652847926
    189
    EP 189 simulation finished, start learning
    1.32
    EP 189 learning finished: 0.008835078051696144, 0.005395160949441073
    190
    EP 190 simulation finished, start learning
    1.4
    EP 190 learning finished: 0.009176586890632166, 0.0061741640338522545
    191
    EP 191 simulation finished, start learning
    1.41
    EP 191 learning finished: 0.009183563096527791, 0.006047168112945182
    192
    EP 192 simulation finished, start learning
    1.53
    EP 192 learning finished: 0.00882711103476197, 0.00688740047131849
    193
    EP 193 simulation finished, start learning
    1.43
    EP 193 learning finished: 0.015865059158809203, 0.00904320278856178
    194
    EP 194 simulation finished, start learning
    1.61
    EP 194 learning finished: 0.01007862622256539, 0.007711562014545296
    195
    EP 195 simulation finished, start learning
    1.13
    EP 195 learning finished: 0.015690892108771948, 0.008863579295057397
    196
    EP 196 simulation finished, start learning
    1.62
    EP 196 learning finished: 0.01576926676195356, 0.008067973493623981
    197
    EP 197 simulation finished, start learning
    1.53
    EP 197 learning finished: 0.008146327524126917, 0.005992784055524207
    198
    EP 198 simulation finished, start learning
    1.53
    EP 198 learning finished: 0.007814334599284933, 0.005915239535747574
    199
    EP 199 simulation finished, start learning
    1.2
    EP 199 learning finished: 0.008081761903117268, 0.005049761193432071
    200
    EP 200 simulation finished, start learning
    2.1
    EP 200 learning finished: 0.009228224065338964, 0.006204815221362144
    201
    EP 201 simulation finished, start learning
    2.11
    EP 201 learning finished: 0.010817018056552996, 0.006483248478411419
    202
    EP 202 simulation finished, start learning
    1.89
    EP 202 learning finished: 0.011707003554860762, 0.00740709293750393
    203
    EP 203 simulation finished, start learning
    1.38
    EP 203 learning finished: 0.014838833715629107, 0.009478127854454973
    204
    EP 204 simulation finished, start learning
    1.1
    EP 204 learning finished: 0.019800519106325964, 0.009341205873211855
    205
    EP 205 simulation finished, start learning
    1.52
    EP 205 learning finished: 0.019728526051441264, 0.00898563000701517
    206
    EP 206 simulation finished, start learning
    1.29
    EP 206 learning finished: 0.0146737809020685, 0.009126788397129696
    207
    EP 207 simulation finished, start learning
    1.08
    EP 207 learning finished: 0.009715000163961207, 0.006857569241017103
    208
    EP 208 simulation finished, start learning
    1.33
    EP 208 learning finished: 0.008384868991015537, 0.0061386110512048715
    209
    EP 209 simulation finished, start learning
    1.61
    EP 209 learning finished: 0.00824254189626572, 0.006119383653069595
    210
    EP 210 simulation finished, start learning
    1.51
    EP 210 learning finished: 0.008014204270374137, 0.005255881634920338
    211
    EP 211 simulation finished, start learning
    1.37
    EP 211 learning finished: 0.009413985154818505, 0.0062190450868389036
    212
    EP 212 simulation finished, start learning
    1.14
    EP 212 learning finished: 0.007538225371742868, 0.00535272653665982
    213
    EP 213 simulation finished, start learning
    1.19
    EP 213 learning finished: 0.009425311867520492, 0.005930795923409467
    214
    EP 214 simulation finished, start learning
    1.83
    EP 214 learning finished: 0.014155231840696558, 0.006877939314834454
    215
    EP 215 simulation finished, start learning
    0.97
    EP 215 learning finished: 0.009261228400117005, 0.005910120314891169
    216
    EP 216 simulation finished, start learning
    1.52
    EP 216 learning finished: 0.00909916290382995, 0.0058457516119831965
    217
    EP 217 simulation finished, start learning
    0.72
    EP 217 learning finished: 0.00802816104481785, 0.005888390842460882
    218
    EP 218 simulation finished, start learning
    1.13
    EP 218 learning finished: 0.008314950057542347, 0.005487777038556037
    219
    EP 219 simulation finished, start learning
    1.8
    EP 219 learning finished: 0.009027254428549014, 0.006489920674720651
    220
    EP 220 simulation finished, start learning
    1.11
    EP 220 learning finished: 0.00897091951496165, 0.0063377126266711175
    221
    EP 221 simulation finished, start learning
    0.82
    EP 221 learning finished: 0.018008469931658055, 0.007070725518292242
    222
    EP 222 simulation finished, start learning
    1.2
    EP 222 learning finished: 0.013254695886317336, 0.006277414237813694
    223
    EP 223 simulation finished, start learning
    0.9
    EP 223 learning finished: 0.013122659668834302, 0.005860693154802791
    224
    EP 224 simulation finished, start learning
    1.31
    EP 224 learning finished: 0.017654762884454556, 0.006586713488455705
    225
    EP 225 simulation finished, start learning
    1.53
    EP 225 learning finished: 0.013094533887269222, 0.005953727980250474
    226
    EP 226 simulation finished, start learning
    1.73
    EP 226 learning finished: 0.01738063611264451, 0.006863800628058729
    227
    EP 227 simulation finished, start learning
    1.58
    EP 227 learning finished: 0.02164255115387448, 0.008789953720969912
    228
    EP 228 simulation finished, start learning
    0.87
    EP 228 learning finished: 0.017143178280758065, 0.007923323756093245
    229
    EP 229 simulation finished, start learning
    1.62
    EP 229 learning finished: 0.012641725222373168, 0.007245644763970975
    230
    EP 230 simulation finished, start learning
    0.93
    EP 230 learning finished: 0.012659951322358864, 0.006907482605689926
    231
    EP 231 simulation finished, start learning
    1.17
    EP 231 learning finished: 0.008149342183298103, 0.005271198791976535
    232
    EP 232 simulation finished, start learning
    1.81
    EP 232 learning finished: 0.007890975980895831, 0.005476315095107746
    233
    EP 233 simulation finished, start learning
    1.82
    EP 233 learning finished: 0.01223717780908018, 0.006122054578472517
    234
    EP 234 simulation finished, start learning
    1.21
    EP 234 learning finished: 0.01203579318550041, 0.006167386993537249
    235
    EP 235 simulation finished, start learning
    0.63
    EP 235 learning finished: 0.007817213457503414, 0.005059767250126986
    236
    EP 236 simulation finished, start learning
    1.29
    EP 236 learning finished: 0.00789098274305644, 0.004930551546195557
    237
    EP 237 simulation finished, start learning
    2.1
    EP 237 learning finished: 0.007182596174708067, 0.005279758562010646
    238
    EP 238 simulation finished, start learning
    1.7
    EP 238 learning finished: 0.0070673524564917045, 0.00491520162287884
    239
    EP 239 simulation finished, start learning
    2.03
    EP 239 learning finished: 0.012797499010691382, 0.007064075728032471
    240
    EP 240 simulation finished, start learning
    1.62
    EP 240 learning finished: 0.012828401619561447, 0.007084161469755064
    241
    EP 241 simulation finished, start learning
    0.97
    EP 241 learning finished: 0.008044209264216562, 0.006308296051307079
    242
    EP 242 simulation finished, start learning
    1.72
    EP 242 learning finished: 0.007900479549864059, 0.006268577947783349
    243
    EP 243 simulation finished, start learning
    1.81
    EP 243 learning finished: 0.00645166532052599, 0.004635615010048315
    244
    EP 244 simulation finished, start learning
    1.31
    EP 244 learning finished: 0.006572735704853727, 0.004532392202392543
    245
    EP 245 simulation finished, start learning
    1.41
    EP 245 learning finished: 0.006322971260829036, 0.004937422973315897
    246
    EP 246 simulation finished, start learning
    1.11
    EP 246 learning finished: 0.006185273115182355, 0.0052394778570287555
    247
    EP 247 simulation finished, start learning
    1.05
    EP 247 learning finished: 0.007624794832274517, 0.005500983751275316
    248
    EP 248 simulation finished, start learning
    1.33
    EP 248 learning finished: 0.00758204477168789, 0.005464910118492483
    249
    EP 249 simulation finished, start learning
    1.89
    EP 249 learning finished: 0.007652004067775664, 0.004734945606704659
    250
    EP 250 simulation finished, start learning
    1.74
    EP 250 learning finished: 0.007390620669198711, 0.005824953897007212
    251
    EP 251 simulation finished, start learning
    1.44
    EP 251 learning finished: 0.006535954650885856, 0.005318117150488059
    252
    EP 252 simulation finished, start learning
    1.25
    EP 252 learning finished: 0.0073317223801048215, 0.005853806132645485
    253
    EP 253 simulation finished, start learning
    1.53
    EP 253 learning finished: 0.01521592223744861, 0.008031924507277152
    254
    EP 254 simulation finished, start learning
    1.55
    EP 254 learning finished: 0.014952500303536453, 0.006856839422340795
    255
    EP 255 simulation finished, start learning
    1.08
    EP 255 learning finished: 0.01491713969874553, 0.006805621458203663
    256
    EP 256 simulation finished, start learning
    1.62
    EP 256 learning finished: 0.010843754069394862, 0.006298133174055032
    257
    EP 257 simulation finished, start learning
    1.21
    EP 257 learning finished: 0.006714873441740821, 0.004624539096209783
    258
    EP 258 simulation finished, start learning
    1.56
    EP 258 learning finished: 0.0061763418791116464, 0.004832230973324414
    259
    EP 259 simulation finished, start learning
    0.83
    EP 259 learning finished: 0.006780621283645871, 0.005232295606485415
    260
    EP 260 simulation finished, start learning
    1.79
    EP 260 learning finished: 0.011520920544167979, 0.00633031030789688
    261
    EP 261 simulation finished, start learning
    2.1
    EP 261 learning finished: 0.017533705122943388, 0.00697466847477705
    262
    EP 262 simulation finished, start learning
    0.81
    EP 262 learning finished: 0.017500882850683297, 0.0067673861525001655
    263
    EP 263 simulation finished, start learning
    0.99
    EP 263 learning finished: 0.017005632923591918, 0.007505225381699769
    264
    EP 264 simulation finished, start learning
    1.93
    EP 264 learning finished: 0.010947014584697712, 0.00649514217536377
    265
    EP 265 simulation finished, start learning
    1.22
    EP 265 learning finished: 0.010819529322183996, 0.005828633807320867
    266
    EP 266 simulation finished, start learning
    0.92
    EP 266 learning finished: 0.010624571370982494, 0.005164406687018304
    267
    EP 267 simulation finished, start learning
    0.61
    EP 267 learning finished: 0.006913838915359349, 0.004240125465816054
    268
    EP 268 simulation finished, start learning
    1.38
    EP 268 learning finished: 0.010914715121590035, 0.004746006604560667
    269
    EP 269 simulation finished, start learning
    1.71
    EP 269 learning finished: 0.006843239830331527, 0.004163563607619676
    270
    EP 270 simulation finished, start learning
    1.29
    EP 270 learning finished: 0.007089695480694047, 0.00508553740571364
    271
    EP 271 simulation finished, start learning
    1.3
    EP 271 learning finished: 0.0065184550742891645, 0.005266308117078881
    272
    EP 272 simulation finished, start learning
    1.52
    EP 272 learning finished: 0.009773989722475562, 0.005851084405416114
    273
    EP 273 simulation finished, start learning
    1.12
    EP 273 learning finished: 0.009687260516788143, 0.005631723671390236
    274
    EP 274 simulation finished, start learning
    1.59
    EP 274 learning finished: 0.009597958943032836, 0.005677133757498787
    275
    EP 275 simulation finished, start learning
    2.02
    EP 275 learning finished: 0.013285824717659267, 0.00645529762232903
    276
    EP 276 simulation finished, start learning
    1.64
    EP 276 learning finished: 0.009553573998498877, 0.005838861910957209
    277
    EP 277 simulation finished, start learning
    0.92
    EP 277 learning finished: 0.01335744146068052, 0.006360171122544796
    278
    EP 278 simulation finished, start learning
    1.11
    EP 278 learning finished: 0.01310096540720096, 0.005581345959123851
    279
    EP 279 simulation finished, start learning
    1.21
    EP 279 learning finished: 0.009750915297171758, 0.004724561601891016
    280
    EP 280 simulation finished, start learning
    1.22
    EP 280 learning finished: 0.0063164698777598005, 0.004714833860195593
    281
    EP 281 simulation finished, start learning
    1.14
    EP 281 learning finished: 0.005878123278629133, 0.004179524246880284
    282
    EP 282 simulation finished, start learning
    1.02
    EP 282 learning finished: 0.005914246892871233, 0.004021355153478605
    283
    EP 283 simulation finished, start learning
    1.14
    EP 283 learning finished: 0.010742101658838266, 0.004627535399802578
    284
    EP 284 simulation finished, start learning
    1.04
    EP 284 learning finished: 0.005654300060895068, 0.003756183858392307
    285
    EP 285 simulation finished, start learning
    1.01
    EP 285 learning finished: 0.005811711576857448, 0.004259122190571674
    286
    EP 286 simulation finished, start learning
    1.8
    EP 286 learning finished: 0.005758483760846644, 0.00462388627256427
    287
    EP 287 simulation finished, start learning
    1.5
    EP 287 learning finished: 0.01289224910973985, 0.005905461105854807
    288
    EP 288 simulation finished, start learning
    1.53
    EP 288 learning finished: 0.013105990847599003, 0.006968294741760417
    289
    EP 289 simulation finished, start learning
    1.32
    EP 289 learning finished: 0.019988566180600742, 0.008175705437215797
    290
    EP 290 simulation finished, start learning
    1.63
    EP 290 learning finished: 0.01936470096548959, 0.007935049322304858
    291
    EP 291 simulation finished, start learning
    1.02
    EP 291 learning finished: 0.01550494416635928, 0.006328797648319136
    292
    EP 292 simulation finished, start learning
    0.72
    EP 292 learning finished: 0.015189794342656882, 0.0055385055614095196
    293
    EP 293 simulation finished, start learning
    1.42
    EP 293 learning finished: 0.00541074997032176, 0.0036830515806511124
    294
    EP 294 simulation finished, start learning
    1.57
    EP 294 learning finished: 0.005766168210906031, 0.003788252993125537
    295
    EP 295 simulation finished, start learning
    0.75
    EP 295 learning finished: 0.0060581068447636, 0.002994366267523733
    296
    EP 296 simulation finished, start learning
    1.92
    EP 296 learning finished: 0.005620255664428697, 0.003986436007406742
    297
    EP 297 simulation finished, start learning
    1.38
    EP 297 learning finished: 0.0055699510604831914, 0.003960237896433701
    298
    EP 298 simulation finished, start learning
    1.32
    EP 298 learning finished: 0.005347801996768098, 0.003843545705121502
    299
    EP 299 simulation finished, start learning
    2.11
    EP 299 learning finished: 0.00597615149628794, 0.004216091493906227
    300
    EP 300 simulation finished, start learning
    1.23
    EP 300 learning finished: 0.0050507691229473295, 0.003201532582386276
    301
    EP 301 simulation finished, start learning
    1.32
    EP 301 learning finished: 0.005565870841166998, 0.00391950929269728
    302
    EP 302 simulation finished, start learning
    1.35
    EP 302 learning finished: 0.004915645944841174, 0.0038139893010583257
    303
    EP 303 simulation finished, start learning
    1.41
    EP 303 learning finished: 0.008258569954229245, 0.004282461370168041
    304
    EP 304 simulation finished, start learning
    1.63
    EP 304 learning finished: 0.008042225921538915, 0.004394459049546384
    305
    EP 305 simulation finished, start learning
    1.35
    EP 305 learning finished: 0.004910949340981574, 0.003323917106849983
    306
    EP 306 simulation finished, start learning
    1.3
    EP 306 learning finished: 0.008427105760436149, 0.004581603579276876
    307
    EP 307 simulation finished, start learning
    1.95
    EP 307 learning finished: 0.005747342447737225, 0.004056376765125454
    308
    EP 308 simulation finished, start learning
    2.22
    EP 308 learning finished: 0.006351249296546513, 0.004817290756307102
    309
    EP 309 simulation finished, start learning
    1.41
    EP 309 learning finished: 0.008302541556943765, 0.005334043879514934
    310
    EP 310 simulation finished, start learning
    1.64
    EP 310 learning finished: 0.006965607040001487, 0.004354988124791959
    311
    EP 311 simulation finished, start learning
    1.61
    EP 311 learning finished: 0.009532975790637721, 0.005071682949191174
    312
    EP 312 simulation finished, start learning
    1.73
    EP 312 learning finished: 0.00595266350849144, 0.004562735651351642
    313
    EP 313 simulation finished, start learning
    1.69
    EP 313 learning finished: 0.006353748046869612, 0.004178836757780857
    314
    EP 314 simulation finished, start learning
    1.5
    EP 314 learning finished: 0.009483285672338582, 0.004948998997934773
    315
    EP 315 simulation finished, start learning
    1.83
    EP 315 learning finished: 0.005961912242479883, 0.004203089217882668
    316
    EP 316 simulation finished, start learning
    1.51
    EP 316 learning finished: 0.005217980619633522, 0.0038989076256648048
    317
    EP 317 simulation finished, start learning
    1.42
    EP 317 learning finished: 0.004934428410336977, 0.003792759007896329
    318
    EP 318 simulation finished, start learning
    1.34
    EP 318 learning finished: 0.004710350380054962, 0.0029726565056371612
    319
    EP 319 simulation finished, start learning
    1.58
    EP 319 learning finished: 0.004907526431185971, 0.0036915141901889825
    320
    EP 320 simulation finished, start learning
    1.32
    EP 320 learning finished: 0.004650245223368543, 0.0038020311071974198
    321
    EP 321 simulation finished, start learning
    1.8
    EP 321 learning finished: 0.009357027029263776, 0.004769513586422747
    322
    EP 322 simulation finished, start learning
    1.41
    EP 322 learning finished: 0.00920462677506606, 0.0048434878418657235
    323
    EP 323 simulation finished, start learning
    0.91
    EP 323 learning finished: 0.004623327405470334, 0.003849434153939827
    324
    EP 324 simulation finished, start learning
    1.81
    EP 324 learning finished: 0.004611300809738594, 0.003597428835622196
    325
    EP 325 simulation finished, start learning
    1.11
    EP 325 learning finished: 0.00520062435287727, 0.003574321130164352
    326
    EP 326 simulation finished, start learning
    1.23
    EP 326 learning finished: 0.00790027529794557, 0.0039727434751523445
    327
    EP 327 simulation finished, start learning
    1.6
    EP 327 learning finished: 0.007566011322701627, 0.004002011241917162
    328
    EP 328 simulation finished, start learning
    1.32
    EP 328 learning finished: 0.00737038395597267, 0.004554925494579069
    329
    EP 329 simulation finished, start learning
    1.18
    EP 329 learning finished: 0.00774267271006579, 0.004656250789105994
    330
    EP 330 simulation finished, start learning
    2.03
    EP 330 learning finished: 0.005256163330898289, 0.0044220555903082825
    331
    EP 331 simulation finished, start learning
    1.33
    EP 331 learning finished: 0.005310890075243435, 0.004348641612786253
    332
    EP 332 simulation finished, start learning
    1.21
    EP 332 learning finished: 0.005443411236651485, 0.003708366749719675
    333
    EP 333 simulation finished, start learning
    1.11
    EP 333 learning finished: 0.004535219940791535, 0.0031549919320023414
    334
    EP 334 simulation finished, start learning
    1.55
    EP 334 learning finished: 0.004568843136464551, 0.00301779932259727
    335
    EP 335 simulation finished, start learning
    1.58
    EP 335 learning finished: 0.004653815122503957, 0.0037123730429788893
    336
    EP 336 simulation finished, start learning
    1.09
    EP 336 learning finished: 0.0046619441575153076, 0.003638692715954406
    337
    EP 337 simulation finished, start learning
    1.14
    EP 337 learning finished: 0.004691193446362263, 0.004013642935929873
    338
    EP 338 simulation finished, start learning
    1.2
    EP 338 learning finished: 0.0048103542581969805, 0.00394951638015358
    339
    EP 339 simulation finished, start learning
    1.81
    EP 339 learning finished: 0.004506300678123409, 0.0033715906858858466
    340
    EP 340 simulation finished, start learning
    2.19
    EP 340 learning finished: 0.004781529443054211, 0.00420764489956819
    341
    EP 341 simulation finished, start learning
    1.89
    EP 341 learning finished: 0.009087486176529325, 0.004534992353020388
    342
    EP 342 simulation finished, start learning
    0.92
    EP 342 learning finished: 0.008648715865877676, 0.004674486284945563
    343
    EP 343 simulation finished, start learning
    1.36
    EP 343 learning finished: 0.008598279915432272, 0.004509583042377289
    344
    EP 344 simulation finished, start learning
    1.11
    EP 344 learning finished: 0.0045950379131198815, 0.0038970471996157365
    345
    EP 345 simulation finished, start learning
    1.02
    EP 345 learning finished: 0.013069598474139354, 0.004546397542743996
    346
    EP 346 simulation finished, start learning
    1.5
    EP 346 learning finished: 0.016214140031483026, 0.005063234677657273
    347
    EP 347 simulation finished, start learning
    0.9
    EP 347 learning finished: 0.016404066476731316, 0.004975975831617833
    348
    EP 348 simulation finished, start learning
    1.62
    EP 348 learning finished: 0.013521403696309164, 0.004600015205919128
    349
    EP 349 simulation finished, start learning
    1.41
    EP 349 learning finished: 0.005116237016586922, 0.003893703424480736
    350
    EP 350 simulation finished, start learning
    1.18
    EP 350 learning finished: 0.004467758629138174, 0.0030696851680488044
    351
    EP 351 simulation finished, start learning
    1.23
    EP 351 learning finished: 0.004260004897377101, 0.0031815676517969274
    352
    EP 352 simulation finished, start learning
    1.34
    EP 352 learning finished: 0.00837223245980212, 0.003946408514663879
    353
    EP 353 simulation finished, start learning
    1.5
    EP 353 learning finished: 0.004278958334030136, 0.0032598953857808275
    354
    EP 354 simulation finished, start learning
    1.41
    EP 354 learning finished: 0.008548755671629493, 0.004078076278521081
    355
    EP 355 simulation finished, start learning
    1.31
    EP 355 learning finished: 0.008433664242417649, 0.004445451210763156
    356
    EP 356 simulation finished, start learning
    0.87
    EP 356 learning finished: 0.004634343364397608, 0.003721450644962173
    357
    EP 357 simulation finished, start learning
    1.54
    EP 357 learning finished: 0.004359916884842718, 0.0036373162833115793
    358
    EP 358 simulation finished, start learning
    1.52
    EP 358 learning finished: 0.00445860376055407, 0.0036879882178212733
    359
    EP 359 simulation finished, start learning
    1.69
    EP 359 learning finished: 0.004189421524994685, 0.0032442363362198043
    360
    EP 360 simulation finished, start learning
    0.77
    EP 360 learning finished: 0.004176321804548522, 0.003193575866053363
    361
    EP 361 simulation finished, start learning
    1.41
    EP 361 learning finished: 0.0082146048417151, 0.003762560318169387
    362
    EP 362 simulation finished, start learning
    2.01
    EP 362 learning finished: 0.008178064091019621, 0.003957871053746824
    363
    EP 363 simulation finished, start learning
    1.11
    EP 363 learning finished: 0.008044394819052059, 0.004009873921069366
    364
    EP 364 simulation finished, start learning
    1.11
    EP 364 learning finished: 0.008093047622903526, 0.0040011728316094635
    365
    EP 365 simulation finished, start learning
    1.63
    EP 365 learning finished: 0.003990738427363638, 0.0033174356682961204
    366
    EP 366 simulation finished, start learning
    1.38
    EP 366 learning finished: 0.004702648385325858, 0.0036605210979325045
    367
    EP 367 simulation finished, start learning
    1.53
    EP 367 learning finished: 0.004423155195521389, 0.0036940200067900144
    368
    EP 368 simulation finished, start learning
    1.52
    EP 368 learning finished: 0.004539342960104165, 0.003800958691221249
    369
    EP 369 simulation finished, start learning
    1.49
    EP 369 learning finished: 0.007965762907719665, 0.004513200826541978
    370
    EP 370 simulation finished, start learning
    1.29
    EP 370 learning finished: 0.003948397467993914, 0.0033326621490719077
    371
    EP 371 simulation finished, start learning
    1.63
    EP 371 learning finished: 0.004047249345294546, 0.003039335594862654
    372
    EP 372 simulation finished, start learning
    1.09
    EP 372 learning finished: 0.004367845296622264, 0.002950178713884514
    373
    EP 373 simulation finished, start learning
    1.25
    EP 373 learning finished: 0.00421924537071483, 0.002581045603755605
    374
    EP 374 simulation finished, start learning
    1.54
    EP 374 learning finished: 0.0040288855284895675, 0.0025268170077904864
    375
    EP 375 simulation finished, start learning
    1.08
    EP 375 learning finished: 0.0039924010876468685, 0.002580420723183726
    376
    EP 376 simulation finished, start learning
    2.31
    EP 376 learning finished: 0.004398349325676065, 0.002830303360038459
    377
    EP 377 simulation finished, start learning
    1.5
    EP 377 learning finished: 0.004535071788119183, 0.0026316088508560043
    378
    EP 378 simulation finished, start learning
    1.34
    EP 378 learning finished: 0.004285096880046524, 0.0031178604048855973
    379
    EP 379 simulation finished, start learning
    1.14
    EP 379 learning finished: 0.004053063370275112, 0.0031345070235436387
    380
    EP 380 simulation finished, start learning
    1.62
    EP 380 learning finished: 0.003960051310560431, 0.002958624029342142
    381
    EP 381 simulation finished, start learning
    1.42
    EP 381 learning finished: 0.003598659576411478, 0.0031301612344050156
    382
    EP 382 simulation finished, start learning
    2.13
    EP 382 learning finished: 0.004037878509606685, 0.00330221754022908
    383
    EP 383 simulation finished, start learning
    1.93
    EP 383 learning finished: 0.0052310477664114785, 0.0034545135763461112
    384
    EP 384 simulation finished, start learning
    1.09
    EP 384 learning finished: 0.00696003331313122, 0.004192521229777609
    385
    EP 385 simulation finished, start learning
    1.42
    EP 385 learning finished: 0.004465067527483768, 0.003760950986775017
    386
    EP 386 simulation finished, start learning
    1.22
    EP 386 learning finished: 0.0068964078167539355, 0.003588192615848603
    387
    EP 387 simulation finished, start learning
    2.12
    EP 387 learning finished: 0.007111120516871239, 0.0036637874738731924
    388
    EP 388 simulation finished, start learning
    1.83
    EP 388 learning finished: 0.0048031791735603235, 0.0032798099384795756
    389
    EP 389 simulation finished, start learning
    1.77
    EP 389 learning finished: 0.004899179809655152, 0.0032101329449251517
    390
    EP 390 simulation finished, start learning
    1.3
    EP 390 learning finished: 0.005074603633499208, 0.003226064484842084
    391
    EP 391 simulation finished, start learning
    1.86
    EP 391 learning finished: 0.006135163854668946, 0.004032034121661236
    392
    EP 392 simulation finished, start learning
    1.62
    EP 392 learning finished: 0.007237667228100264, 0.00466546468912495
    393
    EP 393 simulation finished, start learning
    1.02
    EP 393 learning finished: 0.007021647242037687, 0.004520917799152239
    394
    EP 394 simulation finished, start learning
    1.12
    EP 394 learning finished: 0.006878874145375136, 0.004021384982866157
    395
    EP 395 simulation finished, start learning
    1.23
    EP 395 learning finished: 0.0038702647684196734, 0.0032641557032725053
    396
    EP 396 simulation finished, start learning
    0.63
    EP 396 learning finished: 0.003712115814675343, 0.0022216929221900252
    397
    EP 397 simulation finished, start learning
    1.62
    EP 397 learning finished: 0.003576780876809238, 0.0023679954230259406
    398
    EP 398 simulation finished, start learning
    1.92
    EP 398 learning finished: 0.0035244139267653907, 0.002898577114570718
    399
    EP 399 simulation finished, start learning
    1.55
    EP 399 learning finished: 0.0034997725286700643, 0.002981054403313088
    400
    EP 400 simulation finished, start learning
    1.38
    EP 400 learning finished: 0.007517383420214685, 0.00375495013679276
    401
    EP 401 simulation finished, start learning
    1.18
    EP 401 learning finished: 0.007496480440089215, 0.0036592355983223377
    402
    EP 402 simulation finished, start learning
    1.03
    EP 402 learning finished: 0.003795972263052505, 0.003221133685596437
    403
    EP 403 simulation finished, start learning
    1.41
    EP 403 learning finished: 0.0039324375932352496, 0.0032377671089755347
    404
    EP 404 simulation finished, start learning
    1.51
    EP 404 learning finished: 0.004034722074155894, 0.0028354249120828476
    405
    EP 405 simulation finished, start learning
    1.14
    EP 405 learning finished: 0.005729427542707555, 0.003123792593534939
    406
    EP 406 simulation finished, start learning
    1.41
    EP 406 learning finished: 0.006042959586423535, 0.0032232659536640305
    407
    EP 407 simulation finished, start learning
    1.09
    EP 407 learning finished: 0.008517050393137993, 0.0036305837524019204
    408
    EP 408 simulation finished, start learning
    1.18
    EP 408 learning finished: 0.010962665562992413, 0.003952822634651317
    409
    EP 409 simulation finished, start learning
    1.41
    EP 409 learning finished: 0.006641429949098665, 0.003224618870458625
    410
    EP 410 simulation finished, start learning
    1.61
    EP 410 learning finished: 0.0041316539980407185, 0.0027936976367464914
    411
    EP 411 simulation finished, start learning
    1.42
    EP 411 learning finished: 0.003690342468116725, 0.0023455274687872083
    412
    EP 412 simulation finished, start learning
    1.05
    EP 412 learning finished: 0.0037368973362283953, 0.0021661991900187396
    413
    EP 413 simulation finished, start learning
    1.49
    EP 413 learning finished: 0.0037647334251094122, 0.0027650490246348505
    414
    EP 414 simulation finished, start learning
    1.41
    EP 414 learning finished: 0.003791592818330969, 0.0026877766304682638
    415
    EP 415 simulation finished, start learning
    1.92
    EP 415 learning finished: 0.0035829856630957835, 0.003060602293615785
    416
    EP 416 simulation finished, start learning
    1.23
    EP 416 learning finished: 0.003454144098956391, 0.003020211388194305
    417
    EP 417 simulation finished, start learning
    1.62
    EP 417 learning finished: 0.003368481235497711, 0.002462609321763528
    418
    EP 418 simulation finished, start learning
    1.1
    EP 418 learning finished: 0.004221180229460941, 0.002803999209792648
    419
    EP 419 simulation finished, start learning
    2.04
    EP 419 learning finished: 0.003403076131206717, 0.002903623357971341
    420
    EP 420 simulation finished, start learning
    1.41
    EP 420 learning finished: 0.0033934599752463157, 0.0029226027943668833
    421
    EP 421 simulation finished, start learning
    1.12
    EP 421 learning finished: 0.0064125834257175995, 0.0036074737308358826
    422
    EP 422 simulation finished, start learning
    1.03
    EP 422 learning finished: 0.003950798428704161, 0.0031934982720036587
    423
    EP 423 simulation finished, start learning
    1.0
    EP 423 learning finished: 0.003940373155625565, 0.0025514948781086844
    424
    EP 424 simulation finished, start learning
    1.55
    EP 424 learning finished: 0.003928398351252582, 0.0026725723142781987
    425
    EP 425 simulation finished, start learning
    1.53
    EP 425 learning finished: 0.0033380631444277498, 0.002327266276696882
    426
    EP 426 simulation finished, start learning
    1.77
    EP 426 learning finished: 0.003619201057428967, 0.0029015462523451774
    427
    EP 427 simulation finished, start learning
    1.62
    EP 427 learning finished: 0.006985531309744886, 0.0035624642596584857
    428
    EP 428 simulation finished, start learning
    1.41
    EP 428 learning finished: 0.006958430724200326, 0.0039056944652546832
    429
    EP 429 simulation finished, start learning
    0.82
    EP 429 learning finished: 0.006795408966123517, 0.004121202520860716
    430
    EP 430 simulation finished, start learning
    1.71
    EP 430 learning finished: 0.006027240927576116, 0.0034825363202285096
    431
    EP 431 simulation finished, start learning
    1.83
    EP 431 learning finished: 0.010534343278980174, 0.003804834223707291
    432
    EP 432 simulation finished, start learning
    1.02
    EP 432 learning finished: 0.010354867031082904, 0.003595300158286167
    433
    EP 433 simulation finished, start learning
    1.5
    EP 433 learning finished: 0.00816408433783522, 0.0033552403675469003
    434
    EP 434 simulation finished, start learning
    1.6
    EP 434 learning finished: 0.008246529514593703, 0.0034581358542483307
    435
    EP 435 simulation finished, start learning
    1.59
    EP 435 learning finished: 0.003865826363792735, 0.0031801961892936603
    436
    EP 436 simulation finished, start learning
    1.0
    EP 436 learning finished: 0.0033717052853816654, 0.002867662568018403
    437
    EP 437 simulation finished, start learning
    1.41
    EP 437 learning finished: 0.004219872761026977, 0.003194984074424608
    438
    EP 438 simulation finished, start learning
    1.38
    EP 438 learning finished: 0.003968343064698843, 0.0031066211666382535
    439
    EP 439 simulation finished, start learning
    1.5
    EP 439 learning finished: 0.003960496587428253, 0.002622236810417178
    440
    EP 440 simulation finished, start learning
    1.11
    EP 440 learning finished: 0.0035837312272093413, 0.0026683060286002793
    441
    EP 441 simulation finished, start learning
    1.43
    EP 441 learning finished: 0.0032673684645061485, 0.002210144034719974
    442
    EP 442 simulation finished, start learning
    1.31
    EP 442 learning finished: 0.0032223080329316645, 0.0021856383643286867
    443
    EP 443 simulation finished, start learning
    1.42
    EP 443 learning finished: 0.0031773120523901355, 0.002158323889149235
    444
    EP 444 simulation finished, start learning
    1.01
    EP 444 learning finished: 0.0032781224772087114, 0.0020613610508550075
    445
    EP 445 simulation finished, start learning
    1.53
    EP 445 learning finished: 0.003233296793101914, 0.002087558825511001
    446
    EP 446 simulation finished, start learning
    1.39
    EP 446 learning finished: 0.003261047683107421, 0.002089569405230307
    447
    EP 447 simulation finished, start learning
    1.55
    EP 447 learning finished: 0.00314374430771025, 0.0020385651084916154
    448
    EP 448 simulation finished, start learning
    2.1
    EP 448 learning finished: 0.0036618056716974934, 0.002805617421142159
    449
    EP 449 simulation finished, start learning
    1.4
    EP 449 learning finished: 0.0034488131018832746, 0.0026565372181277525
    450
    EP 450 simulation finished, start learning
    1.51
    EP 450 learning finished: 0.0035710790220662147, 0.0028256865621217506
    451
    EP 451 simulation finished, start learning
    0.91
    EP 451 learning finished: 0.0033192600845204367, 0.0027852310503953723
    452
    EP 452 simulation finished, start learning
    1.11
    EP 452 learning finished: 0.0033749246481313833, 0.002042684075117762
    453
    EP 453 simulation finished, start learning
    1.82
    EP 453 learning finished: 0.0032237356182207577, 0.0025981150292858266
    454
    EP 454 simulation finished, start learning
    1.94
    EP 454 learning finished: 0.0032528813336689666, 0.0026153167220564675
    455
    EP 455 simulation finished, start learning
    1.2
    EP 455 learning finished: 0.006585360336277546, 0.0032686848576949736
    456
    EP 456 simulation finished, start learning
    1.01
    EP 456 learning finished: 0.010041536497563897, 0.0038437528065919935
    457
    EP 457 simulation finished, start learning
    1.53
    EP 457 learning finished: 0.006685923973989816, 0.003174790751807303
    458
    EP 458 simulation finished, start learning
    1.92
    EP 458 learning finished: 0.006666734296242538, 0.003206169859052347
    459
    EP 459 simulation finished, start learning
    1.44
    EP 459 learning finished: 0.0033554729155049756, 0.002424252940783115
    460
    EP 460 simulation finished, start learning
    1.31
    EP 460 learning finished: 0.002974024730050928, 0.0019891423135973696
    461
    EP 461 simulation finished, start learning
    1.11
    EP 461 learning finished: 0.003071619280966331, 0.0019228953034856981
    462
    EP 462 simulation finished, start learning
    1.04
    EP 462 learning finished: 0.0031850057487936567, 0.0023994080968835854
    463
    EP 463 simulation finished, start learning
    1.14
    EP 463 learning finished: 0.0031949373682458315, 0.0026368963536498313
    464
    EP 464 simulation finished, start learning
    0.69
    EP 464 learning finished: 0.007621802789529397, 0.0032324541242735685
    465
    EP 465 simulation finished, start learning
    0.81
    EP 465 learning finished: 0.007581393210244869, 0.0031298144340675313
    466
    EP 466 simulation finished, start learning
    1.71
    EP 466 learning finished: 0.005552428058784642, 0.0024612749031014403
    467
    EP 467 simulation finished, start learning
    1.02
    EP 467 learning finished: 0.005969103786924279, 0.0025395975254619194
    468
    EP 468 simulation finished, start learning
    1.17
    EP 468 learning finished: 0.003445739419387195, 0.0018746783708151564
    469
    EP 469 simulation finished, start learning
    2.42
    EP 469 learning finished: 0.00374768814543458, 0.003050469905501174
    470
    EP 470 simulation finished, start learning
    1.38
    EP 470 learning finished: 0.005716063403816918, 0.003351966168014068
    471
    EP 471 simulation finished, start learning
    1.41
    EP 471 learning finished: 0.0035071696497190885, 0.0030526787089719676
    472
    EP 472 simulation finished, start learning
    1.17
    EP 472 learning finished: 0.005462179108786103, 0.0033712457448586974
    473
    EP 473 simulation finished, start learning
    1.11
    EP 473 learning finished: 0.005439877557023465, 0.0031158202997071762
    474
    EP 474 simulation finished, start learning
    0.99
    EP 474 learning finished: 0.0034664339575378733, 0.002694724193026706
    475
    EP 475 simulation finished, start learning
    1.55
    EP 475 learning finished: 0.006183533971206856, 0.0032335324367220863
    476
    EP 476 simulation finished, start learning
    1.78
    EP 476 learning finished: 0.006207817198049526, 0.002997995026407333
    477
    EP 477 simulation finished, start learning
    1.49
    EP 477 learning finished: 0.0029485321749120885, 0.00241003063232507
    478
    EP 478 simulation finished, start learning
    1.05
    EP 478 learning finished: 0.0031167576201127645, 0.002791526035589816
    479
    EP 479 simulation finished, start learning
    1.91
    EP 479 learning finished: 0.003314302991714868, 0.0028843380590407954
    480
    EP 480 simulation finished, start learning
    1.82
    EP 480 learning finished: 0.003612744522101252, 0.0029061076871309175
    481
    EP 481 simulation finished, start learning
    0.93
    EP 481 learning finished: 0.0033038985648943253, 0.002811372083510714
    482
    EP 482 simulation finished, start learning
    0.93
    EP 482 learning finished: 0.002965388755246323, 0.0023048449753512537
    483
    EP 483 simulation finished, start learning
    0.91
    EP 483 learning finished: 0.003089282376848246, 0.001637888296137054
    484
    EP 484 simulation finished, start learning
    1.51
    EP 484 learning finished: 0.0031469904737129184, 0.0018117769456017878
    485
    EP 485 simulation finished, start learning
    1.5
    EP 485 learning finished: 0.003073358378337332, 0.0023431952485436134
    486
    EP 486 simulation finished, start learning
    1.31
    EP 486 learning finished: 0.006056296506780967, 0.003612251043158802
    487
    EP 487 simulation finished, start learning
    1.5
    EP 487 learning finished: 0.00923417070730892, 0.004219686817057422
    488
    EP 488 simulation finished, start learning
    1.71
    EP 488 learning finished: 0.009245713087175034, 0.0038921950660307423
    489
    EP 489 simulation finished, start learning
    1.89
    EP 489 learning finished: 0.009135761280283816, 0.00462451091070785
    490
    EP 490 simulation finished, start learning
    1.42
    EP 490 learning finished: 0.00597619454528342, 0.0034438855606519305
    491
    EP 491 simulation finished, start learning
    1.23
    EP 491 learning finished: 0.007121823890269496, 0.003484388821671421
    492
    EP 492 simulation finished, start learning
    0.77
    EP 492 learning finished: 0.009188007707386703, 0.003664754242787893
    493
    EP 493 simulation finished, start learning
    2.1
    EP 493 learning finished: 0.005062402278184663, 0.0030135839628068575
    494
    EP 494 simulation finished, start learning
    1.61
    EP 494 learning finished: 0.005380095489350367, 0.0030643556133220784
    495
    EP 495 simulation finished, start learning
    1.01
    EP 495 learning finished: 0.0028531651358889576, 0.0024126665152242785
    496
    EP 496 simulation finished, start learning
    1.64
    EP 496 learning finished: 0.0032367630973340056, 0.0023120589987447224
    497
    EP 497 simulation finished, start learning
    1.41
    EP 497 learning finished: 0.0027803624901956425, 0.0018414458645844196
    498
    EP 498 simulation finished, start learning
    0.81
    EP 498 learning finished: 0.0028655716609620185, 0.001918926189321842
    499
    EP 499 simulation finished, start learning
    2.0
    EP 499 learning finished: 0.0037258289614250373, 0.0023887805451951274
    500
    EP 500 simulation finished, start learning
    1.34
    EP 500 learning finished: 0.005355180646411384, 0.0026020575775209664
    501
    EP 501 simulation finished, start learning
    1.52
    EP 501 learning finished: 0.005225224235527293, 0.0026090064054429263
    502
    EP 502 simulation finished, start learning
    0.57
    EP 502 learning finished: 0.005094231298830465, 0.0025597559196022246
    503
    EP 503 simulation finished, start learning
    1.61
    EP 503 learning finished: 0.004963150577307118, 0.0024886944042587325
    504
    EP 504 simulation finished, start learning
    1.91
    EP 504 learning finished: 0.0031094299992737717, 0.0022788010591051253
    505
    EP 505 simulation finished, start learning
    1.4
    EP 505 learning finished: 0.005042457650265675, 0.0029783024022383606
    506
    EP 506 simulation finished, start learning
    1.17
    EP 506 learning finished: 0.005925010768500272, 0.003241055555488851
    507
    EP 507 simulation finished, start learning
    1.02
    EP 507 learning finished: 0.008969917223947022, 0.003629919700382326
    508
    EP 508 simulation finished, start learning
    0.72
    EP 508 learning finished: 0.008874758663100835, 0.0037198905192212806
    509
    EP 509 simulation finished, start learning
    1.11
    EP 509 learning finished: 0.008961208715049906, 0.0033442044615335695
    510
    EP 510 simulation finished, start learning
    0.7
    EP 510 learning finished: 0.005893413563228478, 0.002759427070574342
    511
    EP 511 simulation finished, start learning
    0.94
    EP 511 learning finished: 0.0032751381850211203, 0.0018954313264133067
    512
    EP 512 simulation finished, start learning
    1.34
    EP 512 learning finished: 0.0030817142939731923, 0.0019759257864434923
    513
    EP 513 simulation finished, start learning
    1.93
    EP 513 learning finished: 0.0028593799791352126, 0.00168346070758633
    514
    EP 514 simulation finished, start learning
    1.38
    EP 514 learning finished: 0.002640820129393262, 0.0021973811014124323
    515
    EP 515 simulation finished, start learning
    1.13
    EP 515 learning finished: 0.005709222121129898, 0.0027228478997406804
    516
    EP 516 simulation finished, start learning
    1.59
    EP 516 learning finished: 0.0058230115351030065, 0.0027645858956078704
    517
    EP 517 simulation finished, start learning
    1.1
    EP 517 learning finished: 0.005606757697552922, 0.0026033300922616576
    518
    EP 518 simulation finished, start learning
    1.53
    EP 518 learning finished: 0.002639972542638702, 0.0022368020776989774
    519
    EP 519 simulation finished, start learning
    1.08
    EP 519 learning finished: 0.0027247203351201676, 0.00193241993366892
    520
    EP 520 simulation finished, start learning
    1.73
    EP 520 learning finished: 0.0034500546639470997, 0.002224965449091935
    521
    EP 521 simulation finished, start learning
    1.34
    EP 521 learning finished: 0.006664448223869357, 0.0031098656389498083
    522
    EP 522 simulation finished, start learning
    1.41
    EP 522 learning finished: 0.006631728602666698, 0.0030844582215075653
    523
    EP 523 simulation finished, start learning
    1.62
    EP 523 learning finished: 0.0084683738656679, 0.003472986951360705
    524
    EP 524 simulation finished, start learning
    1.28
    EP 524 learning finished: 0.010395424809956921, 0.003739801461057013
    525
    EP 525 simulation finished, start learning
    0.84
    EP 525 learning finished: 0.012190772674362194, 0.003926800739283347
    526
    EP 526 simulation finished, start learning
    1.41
    EP 526 learning finished: 0.010393025063716976, 0.003631868817273324
    527
    EP 527 simulation finished, start learning
    0.84
    EP 527 learning finished: 0.00999000429573016, 0.0034581338727175392
    528
    EP 528 simulation finished, start learning
    1.61
    EP 528 learning finished: 0.00615533823720682, 0.0028721687233529045
    529
    EP 529 simulation finished, start learning
    1.63
    EP 529 learning finished: 0.0026020877261920816, 0.001868343367731858
    530
    EP 530 simulation finished, start learning
    1.41
    EP 530 learning finished: 0.004768114560725678, 0.0024931825795982596
    531
    EP 531 simulation finished, start learning
    1.47
    EP 531 learning finished: 0.0030971636111942757, 0.002278582001236984
    532
    EP 532 simulation finished, start learning
    1.31
    EP 532 learning finished: 0.004980530213810397, 0.0025767588410137896
    533
    EP 533 simulation finished, start learning
    1.02
    EP 533 learning finished: 0.0050861990604857965, 0.0028169278809772416
    534
    EP 534 simulation finished, start learning
    1.82
    EP 534 learning finished: 0.002711862205803328, 0.0022848713403339125
    535
    EP 535 simulation finished, start learning
    1.08
    EP 535 learning finished: 0.0036205322485187708, 0.0025140385172518855
    536
    EP 536 simulation finished, start learning
    0.81
    EP 536 learning finished: 0.0027164653482998347, 0.002073144760684542
    537
    EP 537 simulation finished, start learning
    1.8
    EP 537 learning finished: 0.0026223990789198126, 0.0018108537293960818
    538
    EP 538 simulation finished, start learning
    1.82
    EP 538 learning finished: 0.0026454324308353705, 0.0021704062449903936
    539
    EP 539 simulation finished, start learning
    1.23
    EP 539 learning finished: 0.0026020109938151847, 0.0021419002370723916
    540
    EP 540 simulation finished, start learning
    1.12
    EP 540 learning finished: 0.0025972568279044594, 0.0021540584113513006
    541
    EP 541 simulation finished, start learning
    0.8
    EP 541 learning finished: 0.0026576147427164187, 0.001975917415612353
    542
    EP 542 simulation finished, start learning
    1.2
    EP 542 learning finished: 0.0028969751577854197, 0.0018037745347925238
    543
    EP 543 simulation finished, start learning
    1.19
    EP 543 learning finished: 0.003237537537895463, 0.002258689844753784
    544
    EP 544 simulation finished, start learning
    0.82
    EP 544 learning finished: 0.0031163303419461386, 0.0022185555189230635
    545
    EP 545 simulation finished, start learning
    1.8
    EP 545 learning finished: 0.002678825326920986, 0.002118091584062398
    546
    EP 546 simulation finished, start learning
    1.5
    EP 546 learning finished: 0.0051244344107068684, 0.002587365980158888
    547
    EP 547 simulation finished, start learning
    1.31
    EP 547 learning finished: 0.005223274467169858, 0.002551220418501643
    548
    EP 548 simulation finished, start learning
    1.38
    EP 548 learning finished: 0.005325044282390223, 0.0026740092544415006
    549
    EP 549 simulation finished, start learning
    2.11
    EP 549 learning finished: 0.005311121976394162, 0.0026948847760029557
    550
    EP 550 simulation finished, start learning
    1.69
    EP 550 learning finished: 0.002932122867715492, 0.0022976955483660078
    551
    EP 551 simulation finished, start learning
    2.33
    EP 551 learning finished: 0.003760653450420648, 0.002401011527429849
    552
    EP 552 simulation finished, start learning
    1.62
    EP 552 learning finished: 0.004124963990594921, 0.002440752464697956
    553
    EP 553 simulation finished, start learning
    0.53
    EP 553 learning finished: 0.0027764992911691877, 0.0021544995755663856
    554
    EP 554 simulation finished, start learning
    1.68
    EP 554 learning finished: 0.002741977915107441, 0.0021572232569742056
    555
    EP 555 simulation finished, start learning
    1.23
    EP 555 learning finished: 0.002593800226811993, 0.0016940273373039123
    556
    EP 556 simulation finished, start learning
    1.2
    EP 556 learning finished: 0.0026998728670406535, 0.001604552837621398
    557
    EP 557 simulation finished, start learning
    0.48
    EP 557 learning finished: 0.0028325689251857966, 0.0015958031410357638
    558
    EP 558 simulation finished, start learning
    1.53
    EP 558 learning finished: 0.0027701064389660246, 0.0015019020103809548
    559
    EP 559 simulation finished, start learning
    1.6
    EP 559 learning finished: 0.002733099299060626, 0.0018464900337527013
    560
    EP 560 simulation finished, start learning
    1.44
    EP 560 learning finished: 0.002519070351162611, 0.0018066213806378073
    561
    EP 561 simulation finished, start learning
    1.5
    EP 561 learning finished: 0.0029309211559755965, 0.0020192293046636203
    562
    EP 562 simulation finished, start learning
    1.59
    EP 562 learning finished: 0.0030528951458208553, 0.0018083342218151444
    563
    EP 563 simulation finished, start learning
    1.32
    EP 563 learning finished: 0.0028895249696745395, 0.0017648872753071032
    564
    EP 564 simulation finished, start learning
    0.81
    EP 564 learning finished: 0.0023897529372760573, 0.0017017852804907345
    565
    EP 565 simulation finished, start learning
    1.59
    EP 565 learning finished: 0.0029172708078668563, 0.0019551812628974354
    566
    EP 566 simulation finished, start learning
    1.41
    EP 566 learning finished: 0.0026943034569271163, 0.0018648895627422963
    567
    EP 567 simulation finished, start learning
    1.8
    EP 567 learning finished: 0.002528159898734117, 0.0016663534498627304
    568
    EP 568 simulation finished, start learning
    0.9
    EP 568 learning finished: 0.002495083545557719, 0.0017123826942183247
    569
    EP 569 simulation finished, start learning
    0.93
    EP 569 learning finished: 0.0025164565760604773, 0.0019249652402351103
    570
    EP 570 simulation finished, start learning
    2.1
    EP 570 learning finished: 0.0027974874251661674, 0.002055243493139495
    571
    EP 571 simulation finished, start learning
    1.53
    EP 571 learning finished: 0.004913601379321619, 0.0024326117624428414
    572
    EP 572 simulation finished, start learning
    0.9
    EP 572 learning finished: 0.004733660313483767, 0.002980379774657038
    573
    EP 573 simulation finished, start learning
    1.11
    EP 573 learning finished: 0.004379890428435118, 0.0025725713898854252
    574
    EP 574 simulation finished, start learning
    1.22
    EP 574 learning finished: 0.00617259936800707, 0.0027194521474616096
    575
    EP 575 simulation finished, start learning
    1.56
    EP 575 learning finished: 0.006257914891065752, 0.002477748783447827
    576
    EP 576 simulation finished, start learning
    1.43
    EP 576 learning finished: 0.002638805341472761, 0.0018885887999567548
    577
    EP 577 simulation finished, start learning
    2.39
    EP 577 learning finished: 0.003173843936583154, 0.0024728747158023387
    578
    EP 578 simulation finished, start learning
    1.53
    EP 578 learning finished: 0.0035508704454514373, 0.0022233311342017004
    579
    EP 579 simulation finished, start learning
    1.23
    EP 579 learning finished: 0.0033366644539540857, 0.002413252378509275
    580
    EP 580 simulation finished, start learning
    1.11
    EP 580 learning finished: 0.0029670045946232904, 0.00211449226922511
    581
    EP 581 simulation finished, start learning
    1.11
    EP 581 learning finished: 0.002362956706829067, 0.001526556718485376
    582
    EP 582 simulation finished, start learning
    1.15
    EP 582 learning finished: 0.002507268903976012, 0.0014149051560400732
    583
    EP 583 simulation finished, start learning
    1.5
    EP 583 learning finished: 0.0025546993973577514, 0.0013112941712307702
    584
    EP 584 simulation finished, start learning
    1.34
    EP 584 learning finished: 0.002496605995354173, 0.001332706105898751
    585
    EP 585 simulation finished, start learning
    1.02
    EP 585 learning finished: 0.002626076039589975, 0.0017964220600656944
    586
    EP 586 simulation finished, start learning
    1.31
    EP 586 learning finished: 0.002575612617942369, 0.0018168440635673022
    587
    EP 587 simulation finished, start learning
    1.32
    EP 587 learning finished: 0.002573732685075214, 0.0018188289587334576
    588
    EP 588 simulation finished, start learning
    1.44
    EP 588 learning finished: 0.002572254659771289, 0.0018463614178655756
    589
    EP 589 simulation finished, start learning
    1.63
    EP 589 learning finished: 0.0022725698500138947, 0.0014442552255940673
    590
    EP 590 simulation finished, start learning
    1.58
    EP 590 learning finished: 0.0022680642369721227, 0.00147580724517186
    591
    EP 591 simulation finished, start learning
    1.65
    EP 591 learning finished: 0.002358569507713071, 0.0015275976523785317
    592
    EP 592 simulation finished, start learning
    1.4
    EP 592 learning finished: 0.0024062305555929697, 0.001524921289734138
    593
    EP 593 simulation finished, start learning
    1.24
    EP 593 learning finished: 0.002305797851878566, 0.0015576791390638316
    594
    EP 594 simulation finished, start learning
    1.21
    EP 594 learning finished: 0.0022936163421611866, 0.0015067719758080262
    595
    EP 595 simulation finished, start learning
    1.32
    EP 595 learning finished: 0.0023927045988969553, 0.0014471458958471572
    596
    EP 596 simulation finished, start learning
    0.98
    EP 596 learning finished: 0.002490760197222386, 0.0013867903555464522
    597
    EP 597 simulation finished, start learning
    1.73
    EP 597 learning finished: 0.002474265605652448, 0.001789645759865029
    598
    EP 598 simulation finished, start learning
    1.68
    EP 598 learning finished: 0.005028559430694912, 0.0022501220542210376
    599
    EP 599 simulation finished, start learning
    1.53
    EP 599 learning finished: 0.005015400061950004, 0.002448210541057705
    600
    EP 600 simulation finished, start learning
    1.41
    EP 600 learning finished: 0.005050473614303996, 0.002454470317861871
    601
    EP 601 simulation finished, start learning
    1.11
    EP 601 learning finished: 0.0024505394037426242, 0.0019552645686970274
    602
    EP 602 simulation finished, start learning
    1.2
    EP 602 learning finished: 0.0023846909243516734, 0.0014864525725369522
    603
    EP 603 simulation finished, start learning
    1.61
    EP 603 learning finished: 0.00240512223655287, 0.0015293293599203395
    604
    EP 604 simulation finished, start learning
    0.71
    EP 604 learning finished: 0.0025013647509979364, 0.001457962912478905
    605
    EP 605 simulation finished, start learning
    1.81
    EP 605 learning finished: 0.0022776186362953865, 0.0015666166597032571
    606
    EP 606 simulation finished, start learning
    1.53
    EP 606 learning finished: 0.0023530621816441055, 0.001958801401931017
    607
    EP 607 simulation finished, start learning
    1.8
    EP 607 learning finished: 0.002395013055075168, 0.001966320338635086
    608
    EP 608 simulation finished, start learning
    1.8
    EP 608 learning finished: 0.004914878955748492, 0.002565697306337119
    609
    EP 609 simulation finished, start learning
    1.04
    EP 609 learning finished: 0.004900976589404138, 0.0026637577043343206
    610
    EP 610 simulation finished, start learning
    1.11
    EP 610 learning finished: 0.0025191904385053476, 0.0021937372813016034
    611
    EP 611 simulation finished, start learning
    1.09
    EP 611 learning finished: 0.002512383140814478, 0.002103289186165943
    612
    EP 612 simulation finished, start learning
    1.89
    EP 612 learning finished: 0.0024049016269067636, 0.001522999830487611
    613
    EP 613 simulation finished, start learning
    1.41
    EP 613 learning finished: 0.002424565600890105, 0.0014913377000394708
    614
    EP 614 simulation finished, start learning
    1.17
    EP 614 learning finished: 0.002374521605386093, 0.0019734748631693813
    615
    EP 615 simulation finished, start learning
    1.11
    EP 615 learning finished: 0.0023653795437432734, 0.0020083578041217485
    616
    EP 616 simulation finished, start learning
    1.51
    EP 616 learning finished: 0.0022293575358816042, 0.0018899121607116598
    617
    EP 617 simulation finished, start learning
    1.02
    EP 617 learning finished: 0.00277118604838833, 0.0020929020757201044
    618
    EP 618 simulation finished, start learning
    1.51
    EP 618 learning finished: 0.0022857672018723463, 0.0015018741639174493
    619
    EP 619 simulation finished, start learning
    0.81
    EP 619 learning finished: 0.0025841767742598165, 0.0017561774222561628
    620
    EP 620 simulation finished, start learning
    1.16
    EP 620 learning finished: 0.0024884724940363867, 0.0016761035539722968
    621
    EP 621 simulation finished, start learning
    1.62
    EP 621 learning finished: 0.004258637113299102, 0.0020567217039832997
    622
    EP 622 simulation finished, start learning
    1.8
    EP 622 learning finished: 0.0059306584638996185, 0.0023588037834170925
    623
    EP 623 simulation finished, start learning
    1.29
    EP 623 learning finished: 0.006004286711245754, 0.0023629081174287117
    624
    EP 624 simulation finished, start learning
    1.23
    EP 624 learning finished: 0.006172048377482753, 0.002730254478120031
    625
    EP 625 simulation finished, start learning
    1.03
    EP 625 learning finished: 0.002897575609193126, 0.0020931451540147825
    626
    EP 626 simulation finished, start learning
    1.29
    EP 626 learning finished: 0.005966640144460911, 0.002534481281898593
    627
    EP 627 simulation finished, start learning
    1.32
    EP 627 learning finished: 0.0042785549119628286, 0.002268594518800645
    628
    EP 628 simulation finished, start learning
    1.0
    EP 628 learning finished: 0.004361530941334735, 0.0019442776452146167
    629
    EP 629 simulation finished, start learning
    2.36
    EP 629 learning finished: 0.004163847386948305, 0.0020963447056683293
    630
    EP 630 simulation finished, start learning
    1.33
    EP 630 learning finished: 0.0024061693056758315, 0.00154594988001359
    631
    EP 631 simulation finished, start learning
    1.5
    EP 631 learning finished: 0.0025646460112973557, 0.0015871428185991003
    632
    EP 632 simulation finished, start learning
    1.5
    EP 632 learning finished: 0.0028856589169770075, 0.001637686922667451
    633
    EP 633 simulation finished, start learning
    1.69
    EP 633 learning finished: 0.0023413367496054605, 0.0015928114458757154
    634
    EP 634 simulation finished, start learning
    1.2
    EP 634 learning finished: 0.002315659453305585, 0.0019189988613158941
    635
    EP 635 simulation finished, start learning
    1.52
    EP 635 learning finished: 0.0025702540659366946, 0.002152355825894977
    636
    EP 636 simulation finished, start learning
    1.51
    EP 636 learning finished: 0.0023802030099844274, 0.002107027496111987
    637
    EP 637 simulation finished, start learning
    1.75
    EP 637 learning finished: 0.0038356579581915184, 0.002341509931170549
    638
    EP 638 simulation finished, start learning
    0.9
    EP 638 learning finished: 0.0022805348307990414, 0.001739294717445124
    639
    EP 639 simulation finished, start learning
    1.73
    EP 639 learning finished: 0.003812967136539669, 0.002021168936073181
    640
    EP 640 simulation finished, start learning
    1.23
    EP 640 learning finished: 0.003892677555267454, 0.002038062672047166
    641
    EP 641 simulation finished, start learning
    1.44
    EP 641 learning finished: 0.0022477390691229004, 0.0017172308655434658
    642
    EP 642 simulation finished, start learning
    1.01
    EP 642 learning finished: 0.0036846546094118082, 0.0019649642828068656
    643
    EP 643 simulation finished, start learning
    1.01
    EP 643 learning finished: 0.0020535690920446825, 0.0013758587802895998
    644
    EP 644 simulation finished, start learning
    1.22
    EP 644 learning finished: 0.002178443281060538, 0.0016419878562228263
    645
    EP 645 simulation finished, start learning
    1.08
    EP 645 learning finished: 0.0022813004309227836, 0.0016292004127027582
    646
    EP 646 simulation finished, start learning
    1.23
    EP 646 learning finished: 0.0022762193883281157, 0.0016675497755773705
    647
    EP 647 simulation finished, start learning
    1.11
    EP 647 learning finished: 0.0024075172920206, 0.0019124579207504173
    648
    EP 648 simulation finished, start learning
    1.24
    EP 648 learning finished: 0.002402509705060396, 0.0015620608490003463
    649
    EP 649 simulation finished, start learning
    1.5
    EP 649 learning finished: 0.0040042370569608465, 0.0018910671305742237
    650
    EP 650 simulation finished, start learning
    1.68
    EP 650 learning finished: 0.002480552414350967, 0.0020621579723668995
    651
    EP 651 simulation finished, start learning
    1.7
    EP 651 learning finished: 0.0024759487077830045, 0.002144010278540165
    652
    EP 652 simulation finished, start learning
    1.32
    EP 652 learning finished: 0.0046609739954455165, 0.0025466674902533035
    653
    EP 653 simulation finished, start learning
    1.28
    EP 653 learning finished: 0.004650055438041534, 0.0022416585112138656
    654
    EP 654 simulation finished, start learning
    1.91
    EP 654 learning finished: 0.004638728454754215, 0.0022586477341259106
    655
    EP 655 simulation finished, start learning
    1.53
    EP 655 learning finished: 0.004665869730750671, 0.0022108592693292906
    656
    EP 656 simulation finished, start learning
    1.62
    EP 656 learning finished: 0.002271971412440621, 0.00186727317644871
    657
    EP 657 simulation finished, start learning
    1.52
    EP 657 learning finished: 0.0046909773901155916, 0.002278449180089126
    658
    EP 658 simulation finished, start learning
    1.74
    EP 658 learning finished: 0.004723556571263998, 0.0022053025860344633
    659
    EP 659 simulation finished, start learning
    1.34
    EP 659 learning finished: 0.004717974622682419, 0.0021261794044420412
    660
    EP 660 simulation finished, start learning
    1.0
    EP 660 learning finished: 0.004588994432715765, 0.0020725370020494203
    661
    EP 661 simulation finished, start learning
    1.82
    EP 661 learning finished: 0.0021671036398700494, 0.00175226059149954
    662
    EP 662 simulation finished, start learning
    1.41
    EP 662 learning finished: 0.00394380977739317, 0.0022367937377930267
    663
    EP 663 simulation finished, start learning
    1.09
    EP 663 learning finished: 0.0069077964312462825, 0.00273409063578652
    664
    EP 664 simulation finished, start learning
    1.53
    EP 664 learning finished: 0.008274776882828405, 0.003007459749197731
    665
    EP 665 simulation finished, start learning
    1.4
    EP 665 learning finished: 0.006853003416413372, 0.00270771397928371
    666
    EP 666 simulation finished, start learning
    1.65
    EP 666 learning finished: 0.003605205407119172, 0.002205007954560282
    667
    EP 667 simulation finished, start learning
    1.32
    EP 667 learning finished: 0.0036847011644525196, 0.0022273057203581934
    668
    EP 668 simulation finished, start learning
    1.91
    EP 668 learning finished: 0.004510222604555025, 0.0024060773733012297
    669
    EP 669 simulation finished, start learning
    1.3
    EP 669 learning finished: 0.004541937271874019, 0.0023820699779193433
    670
    EP 670 simulation finished, start learning
    1.2
    EP 670 learning finished: 0.0023792657071872725, 0.001978971951778874
    671
    EP 671 simulation finished, start learning
    1.81
    EP 671 learning finished: 0.0022005668214291063, 0.0017230719343199752
    672
    EP 672 simulation finished, start learning
    1.92
    EP 672 learning finished: 0.0024122997592002093, 0.0014318304334051863
    673
    EP 673 simulation finished, start learning
    1.01
    EP 673 learning finished: 0.0022775983830207736, 0.0015255493797911395
    674
    EP 674 simulation finished, start learning
    2.12
    EP 674 learning finished: 0.0028848729708244036, 0.0016226926060826468
    675
    EP 675 simulation finished, start learning
    1.8
    EP 675 learning finished: 0.002924692754149305, 0.0019033444128737642
    676
    EP 676 simulation finished, start learning
    1.28
    EP 676 learning finished: 0.002331923493930077, 0.0018032771308273965
    677
    EP 677 simulation finished, start learning
    1.09
    EP 677 learning finished: 0.002258215013341669, 0.0017122664409345845
    678
    EP 678 simulation finished, start learning
    1.31
    EP 678 learning finished: 0.002102157962507789, 0.0015931721598872646
    679
    EP 679 simulation finished, start learning
    1.51
    EP 679 learning finished: 0.0020855103680403886, 0.0012403293318416313
    680
    EP 680 simulation finished, start learning
    0.81
    EP 680 learning finished: 0.002059382629219897, 0.001197750177648042
    681
    EP 681 simulation finished, start learning
    1.19
    EP 681 learning finished: 0.0020114963437514177, 0.0015374390455829001
    682
    EP 682 simulation finished, start learning
    1.67
    EP 682 learning finished: 0.0020531062734043815, 0.0016186678730451358
    683
    EP 683 simulation finished, start learning
    1.78
    EP 683 learning finished: 0.0020492267093956684, 0.0016273781902054196
    684
    EP 684 simulation finished, start learning
    1.32
    EP 684 learning finished: 0.0021482061016904197, 0.001767399108492403
    685
    EP 685 simulation finished, start learning
    1.69
    EP 685 learning finished: 0.0024860520163692, 0.0015370874270418018
    686
    EP 686 simulation finished, start learning
    0.79
    EP 686 learning finished: 0.002090209150697575, 0.0014842253275334134
    687
    EP 687 simulation finished, start learning
    2.01
    EP 687 learning finished: 0.002631633090839358, 0.0017312510358262756
    688
    EP 688 simulation finished, start learning
    1.32
    EP 688 learning finished: 0.002039152187247595, 0.0014322727799053563
    689
    EP 689 simulation finished, start learning
    1.62
    EP 689 learning finished: 0.00195554690369093, 0.0013894206218907357
    690
    EP 690 simulation finished, start learning
    0.66
    EP 690 learning finished: 0.0018651165262433267, 0.0013441757165324475
    691
    EP 691 simulation finished, start learning
    0.95
    EP 691 learning finished: 0.0021563394938723945, 0.0014598970353626821
    692
    EP 692 simulation finished, start learning
    1.5
    EP 692 learning finished: 0.002194507256000122, 0.0017459000969045253
    693
    EP 693 simulation finished, start learning
    1.32
    EP 693 learning finished: 0.002112829056237331, 0.0016905429090214124
    694
    EP 694 simulation finished, start learning
    1.28
    EP 694 learning finished: 0.0036148515537277603, 0.0020469623489625852
    695
    EP 695 simulation finished, start learning
    1.83
    EP 695 learning finished: 0.00368993362885206, 0.001869259400169317
    696
    EP 696 simulation finished, start learning
    1.11
    EP 696 learning finished: 0.0019404631376218617, 0.001319557358690086
    697
    EP 697 simulation finished, start learning
    1.43
    EP 697 learning finished: 0.0018723853156785708, 0.0013701520846787218
    698
    EP 698 simulation finished, start learning
    0.99
    EP 698 learning finished: 0.0019123353987176017, 0.0011712765550235815
    699
    EP 699 simulation finished, start learning
    1.32
    EP 699 learning finished: 0.00199514822026714, 0.0011061307044778108
    700
    EP 700 simulation finished, start learning
    1.33
    EP 700 learning finished: 0.0019274392974750754, 0.0011774656658225524
    701
    EP 701 simulation finished, start learning
    0.61
    EP 701 learning finished: 0.0020525015673023006, 0.001118023988532002
    702
    EP 702 simulation finished, start learning
    1.51
    EP 702 learning finished: 0.001935218903359015, 0.0012550994608406067
    703
    EP 703 simulation finished, start learning
    1.53
    EP 703 learning finished: 0.001846780606239662, 0.0012603249527470457
    704
    EP 704 simulation finished, start learning
    1.53
    EP 704 learning finished: 0.0018015201602068487, 0.0013735724261737896
    705
    EP 705 simulation finished, start learning
    0.63
    EP 705 learning finished: 0.002093110257160191, 0.0015550866210679952
    706
    EP 706 simulation finished, start learning
    1.71
    EP 706 learning finished: 0.002087746097593879, 0.001798002676983046
    707
    EP 707 simulation finished, start learning
    1.62
    EP 707 learning finished: 0.0042090090466473384, 0.002162151840627402
    708
    EP 708 simulation finished, start learning
    0.91
    EP 708 learning finished: 0.0042289057190672, 0.0018864484425626198
    709
    EP 709 simulation finished, start learning
    1.21
    EP 709 learning finished: 0.0042848528416215, 0.001926621642849587
    710
    EP 710 simulation finished, start learning
    0.89
    EP 710 learning finished: 0.002114864797896132, 0.0015449710265128165
    711
    EP 711 simulation finished, start learning
    1.03
    EP 711 learning finished: 0.002174017306057152, 0.0011306487227743152
    712
    EP 712 simulation finished, start learning
    1.02
    EP 712 learning finished: 0.002071781665616612, 0.0011344521643810168
    713
    EP 713 simulation finished, start learning
    1.22
    EP 713 learning finished: 0.002012125958742672, 0.0010928542341813359
    714
    EP 714 simulation finished, start learning
    1.09
    EP 714 learning finished: 0.0019876497993749354, 0.0011100076652699187
    715
    EP 715 simulation finished, start learning
    0.9
    EP 715 learning finished: 0.0023101811812757944, 0.0013732947286086983
    716
    EP 716 simulation finished, start learning
    1.02
    EP 716 learning finished: 0.003856369171593599, 0.0016427833701507428
    717
    EP 717 simulation finished, start learning
    0.63
    EP 717 learning finished: 0.0038487024170983375, 0.0015508272488056936
    718
    EP 718 simulation finished, start learning
    1.11
    EP 718 learning finished: 0.0037613514858314545, 0.001547049699893203
    719
    EP 719 simulation finished, start learning
    2.24
    EP 719 learning finished: 0.0022079523245359667, 0.0014475065222138521
    720
    EP 720 simulation finished, start learning
    1.44
    EP 720 learning finished: 0.0018360828242620642, 0.0011878699188751323
    721
    EP 721 simulation finished, start learning
    1.11
    EP 721 learning finished: 0.0019744740142044748, 0.001299683321409617
    722
    EP 722 simulation finished, start learning
    1.53
    EP 722 learning finished: 0.002320825521046288, 0.0016270538944572384
    723
    EP 723 simulation finished, start learning
    1.5
    EP 723 learning finished: 0.0041677175069914085, 0.001921643031043511
    724
    EP 724 simulation finished, start learning
    1.8
    EP 724 learning finished: 0.004155342961485786, 0.0019880778256252305
    725
    EP 725 simulation finished, start learning
    1.62
    EP 725 learning finished: 0.006340580876681656, 0.002389727477548694
    726
    EP 726 simulation finished, start learning
    1.62
    EP 726 learning finished: 0.004227427734257105, 0.0019506952130961726
    727
    EP 727 simulation finished, start learning
    1.56
    EP 727 learning finished: 0.0022758235670217175, 0.0016872010613795396
    728
    EP 728 simulation finished, start learning
    1.43
    EP 728 learning finished: 0.0021515828727842745, 0.0016344311546253227
    729
    EP 729 simulation finished, start learning
    0.8
    EP 729 learning finished: 0.00192384027792869, 0.0015584503316083225
    730
    EP 730 simulation finished, start learning
    1.0
    EP 730 learning finished: 0.0019198279063274981, 0.0015098325405678754
    731
    EP 731 simulation finished, start learning
    0.78
    EP 731 learning finished: 0.001997185618815678, 0.0012991814078482378
    732
    EP 732 simulation finished, start learning
    1.59
    EP 732 learning finished: 0.00411733337159248, 0.0016289782936631525
    733
    EP 733 simulation finished, start learning
    1.1
    EP 733 learning finished: 0.002050623719767941, 0.0013050910144561574
    734
    EP 734 simulation finished, start learning
    1.8
    EP 734 learning finished: 0.002001524019924261, 0.0014126084477084078
    735
    EP 735 simulation finished, start learning
    1.53
    EP 735 learning finished: 0.0020108788738966155, 0.0016519467324697483
    736
    EP 736 simulation finished, start learning
    1.02
    EP 736 learning finished: 0.0019268131705771907, 0.0015672410103887795
    737
    EP 737 simulation finished, start learning
    0.78
    EP 737 learning finished: 0.004048822263285956, 0.0018632902171796435
    738
    EP 738 simulation finished, start learning
    1.41
    EP 738 learning finished: 0.004037578546139287, 0.001807375601766618
    739
    EP 739 simulation finished, start learning
    1.44
    EP 739 learning finished: 0.004132551729178147, 0.0017063689428121382
    740
    EP 740 simulation finished, start learning
    1.23
    EP 740 learning finished: 0.0019380364037431264, 0.0014091616734190164
    741
    EP 741 simulation finished, start learning
    1.42
    EP 741 learning finished: 0.0018830645507250665, 0.0014207869443338221
    742
    EP 742 simulation finished, start learning
    1.59
    EP 742 learning finished: 0.0018033616134474073, 0.0014367269983151705
    743
    EP 743 simulation finished, start learning
    1.48
    EP 743 learning finished: 0.0032920352002143716, 0.0017184740374365977
    744
    EP 744 simulation finished, start learning
    1.33
    EP 744 learning finished: 0.0022388886557536125, 0.0015293199168837646
    745
    EP 745 simulation finished, start learning
    1.23
    EP 745 learning finished: 0.0017858193549002976, 0.0012538225950731174
    746
    EP 746 simulation finished, start learning
    1.29
    EP 746 learning finished: 0.0018225217466252275, 0.0012105914282756203
    747
    EP 747 simulation finished, start learning
    1.71
    EP 747 learning finished: 0.0016874517737015537, 0.001243951207357578
    748
    EP 748 simulation finished, start learning
    1.32
    EP 748 learning finished: 0.002303616724606883, 0.0014747342788083984
    749
    EP 749 simulation finished, start learning
    1.29
    EP 749 learning finished: 0.0024483924805203117, 0.0014951758218657574
    750
    EP 750 simulation finished, start learning
    1.41
    EP 750 learning finished: 0.005112083323620453, 0.0019774443906346306
    751
    EP 751 simulation finished, start learning
    1.5
    EP 751 learning finished: 0.0037714309758989373, 0.0017062396794583397
    752
    EP 752 simulation finished, start learning
    1.04
    EP 752 learning finished: 0.0022096498430101974, 0.0014118045947485076
    753
    EP 753 simulation finished, start learning
    1.36
    EP 753 learning finished: 0.002055325834959376, 0.001402083901156222
    754
    EP 754 simulation finished, start learning
    1.23
    EP 754 learning finished: 0.0017618733049751256, 0.0011386339396146639
    755
    EP 755 simulation finished, start learning
    2.21
    EP 755 learning finished: 0.0019917759533001345, 0.0015466336848965294
    756
    EP 756 simulation finished, start learning
    0.98
    EP 756 learning finished: 0.0020198446700696593, 0.0015195445347426784
    757
    EP 757 simulation finished, start learning
    1.09
    EP 757 learning finished: 0.001957250391122045, 0.001541362160638234
    758
    EP 758 simulation finished, start learning
    0.91
    EP 758 learning finished: 0.001954644498319169, 0.0015208494043399412
    759
    EP 759 simulation finished, start learning
    1.15
    EP 759 learning finished: 0.001975418498755488, 0.0010901145937243228
    760
    EP 760 simulation finished, start learning
    0.91
    EP 760 learning finished: 0.0019753124074810557, 0.001280764840117854
    761
    EP 761 simulation finished, start learning
    1.2
    EP 761 learning finished: 0.003210339826992481, 0.001506259422028459
    762
    EP 762 simulation finished, start learning
    1.11
    EP 762 learning finished: 0.0032727815421011996, 0.0017927300749770818
    763
    EP 763 simulation finished, start learning
    1.29
    EP 763 learning finished: 0.0034838160809940177, 0.0018395667370209405
    764
    EP 764 simulation finished, start learning
    1.71
    EP 764 learning finished: 0.003808478578449126, 0.0020518489933595654
    765
    EP 765 simulation finished, start learning
    1.5
    EP 765 learning finished: 0.0059604429649624535, 0.002093422559804973
    766
    EP 766 simulation finished, start learning
    1.31
    EP 766 learning finished: 0.003994448276344542, 0.001784724097854351
    767
    EP 767 simulation finished, start learning
    1.42
    EP 767 learning finished: 0.003987557650024343, 0.001801835581683638
    768
    EP 768 simulation finished, start learning
    1.5
    EP 768 learning finished: 0.0021692538295141617, 0.0017142123051930487
    769
    EP 769 simulation finished, start learning
    1.41
    EP 769 learning finished: 0.002165666562465418, 0.0013911571310399513
    770
    EP 770 simulation finished, start learning
    1.41
    EP 770 learning finished: 0.0034593018516098084, 0.00163444310855058
    771
    EP 771 simulation finished, start learning
    1.83
    EP 771 learning finished: 0.003305763249063176, 0.0019315104533177418
    772
    EP 772 simulation finished, start learning
    0.79
    EP 772 learning finished: 0.001968699162599563, 0.0014548374874265118
    773
    EP 773 simulation finished, start learning
    1.71
    EP 773 learning finished: 0.0020752980656304487, 0.001704575793124737
    774
    EP 774 simulation finished, start learning
    2.01
    EP 774 learning finished: 0.0019653727432427104, 0.001562216384246658
    775
    EP 775 simulation finished, start learning
    1.08
    EP 775 learning finished: 0.0017301021075013523, 0.0012313100310633966
    776
    EP 776 simulation finished, start learning
    1.92
    EP 776 learning finished: 0.00242003366925303, 0.0016287729419074044
    777
    EP 777 simulation finished, start learning
    1.42
    EP 777 learning finished: 0.002283057703927227, 0.0015601973416075055
    778
    EP 778 simulation finished, start learning
    0.93
    EP 778 learning finished: 0.003849504195358122, 0.0017649640346982427
    779
    EP 779 simulation finished, start learning
    1.84
    EP 779 learning finished: 0.0038715556073753987, 0.0018241473148279397
    780
    EP 780 simulation finished, start learning
    1.32
    EP 780 learning finished: 0.001787718902424018, 0.0014293550158305617
    781
    EP 781 simulation finished, start learning
    1.32
    EP 781 learning finished: 0.0018490561631985258, 0.0014049695293151625
    782
    EP 782 simulation finished, start learning
    0.93
    EP 782 learning finished: 0.0015821767405270748, 0.0011348628310550168
    783
    EP 783 simulation finished, start learning
    1.42
    EP 783 learning finished: 0.002132760810145129, 0.0013314835302834875
    784
    EP 784 simulation finished, start learning
    1.1
    EP 784 learning finished: 0.0017467206991810258, 0.0010774347815445395
    785
    EP 785 simulation finished, start learning
    1.4
    EP 785 learning finished: 0.001814347660801019, 0.0014234717495137754
    786
    EP 786 simulation finished, start learning
    1.4
    EP 786 learning finished: 0.0017791156724200619, 0.0015181653493575216
    787
    EP 787 simulation finished, start learning
    1.41
    EP 787 learning finished: 0.0017765104477263094, 0.001514778723549881
    788
    EP 788 simulation finished, start learning
    1.52
    EP 788 learning finished: 0.003762297922303759, 0.0018773676836930282
    789
    EP 789 simulation finished, start learning
    1.62
    EP 789 learning finished: 0.0017974747096687338, 0.0015447732835696115
    790
    EP 790 simulation finished, start learning
    1.52
    EP 790 learning finished: 0.0023355244804719177, 0.001730160213487153
    791
    EP 791 simulation finished, start learning
    2.06
    EP 791 learning finished: 0.003771892657798759, 0.002069031270156785
    792
    EP 792 simulation finished, start learning
    1.77
    EP 792 learning finished: 0.002425460000047508, 0.0017909892741367155
    793
    EP 793 simulation finished, start learning
    1.5
    EP 793 learning finished: 0.0037893981083314543, 0.0021249929852219887
    794
    EP 794 simulation finished, start learning
    1.11
    EP 794 learning finished: 0.0037815374184262396, 0.0023006296641755832
    795
    EP 795 simulation finished, start learning
    0.81
    EP 795 learning finished: 0.0023165049990900854, 0.001677463747218037
    796
    EP 796 simulation finished, start learning
    1.2
    EP 796 learning finished: 0.0021724728521093084, 0.0015782638051273779
    797
    EP 797 simulation finished, start learning
    1.2
    EP 797 learning finished: 0.002098678050188596, 0.00124125637688266
    798
    EP 798 simulation finished, start learning
    2.04
    EP 798 learning finished: 0.0019319822293333289, 0.0013543172829161658
    799
    EP 799 simulation finished, start learning
    1.14
    EP 799 learning finished: 0.0019312528985793165, 0.001412391204777208
    800
    EP 800 simulation finished, start learning
    1.82
    EP 800 learning finished: 0.001930121830898912, 0.001478889096082625
    801
    EP 801 simulation finished, start learning
    1.06
    EP 801 learning finished: 0.0018622613307783809, 0.0013960051427837072
    802
    EP 802 simulation finished, start learning
    1.64
    EP 802 learning finished: 0.0016049237202082622, 0.0010678758341828853
    803
    EP 803 simulation finished, start learning
    1.5
    EP 803 learning finished: 0.0017851010223104424, 0.0013551661602697607
    804
    EP 804 simulation finished, start learning
    1.13
    EP 804 learning finished: 0.001671978191669865, 0.00126426683209561
    805
    EP 805 simulation finished, start learning
    1.69
    EP 805 learning finished: 0.0016139250524922488, 0.0011480309440058771
    806
    EP 806 simulation finished, start learning
    1.23
    EP 806 learning finished: 0.0016030067170937912, 0.0011269701837957058
    807
    EP 807 simulation finished, start learning
    1.29
    EP 807 learning finished: 0.002210530906697983, 0.0016151766761076658
    808
    EP 808 simulation finished, start learning
    1.83
    EP 808 learning finished: 0.0034474085569478063, 0.0019118227067096427
    809
    EP 809 simulation finished, start learning
    1.73
    EP 809 learning finished: 0.0033728632029017547, 0.0019068214748637995
    810
    EP 810 simulation finished, start learning
    1.1
    EP 810 learning finished: 0.004670873905119777, 0.0024242383288098543
    811
    EP 811 simulation finished, start learning
    2.12
    EP 811 learning finished: 0.0036494903286907743, 0.0020983991593244156
    812
    EP 812 simulation finished, start learning
    1.2
    EP 812 learning finished: 0.005569692791372205, 0.002163184563282646
    813
    EP 813 simulation finished, start learning
    0.69
    EP 813 learning finished: 0.00747610666066123, 0.002746922196122862
    814
    EP 814 simulation finished, start learning
    1.49
    EP 814 learning finished: 0.007516950246611032, 0.002581761318199464
    815
    EP 815 simulation finished, start learning
    0.99
    EP 815 learning finished: 0.005613008598373881, 0.002126326684643113
    816
    EP 816 simulation finished, start learning
    1.71
    EP 816 learning finished: 0.0037222126783653664, 0.0018452895108490945
    817
    EP 817 simulation finished, start learning
    1.42
    EP 817 learning finished: 0.0018041174208711652, 0.0014293430607009345
    818
    EP 818 simulation finished, start learning
    1.13
    EP 818 learning finished: 0.0016294179463507419, 0.0010670917015804804
    819
    EP 819 simulation finished, start learning
    1.01
    EP 819 learning finished: 0.0016093002932002776, 0.0010618006432162967
    820
    EP 820 simulation finished, start learning
    1.01
    EP 820 learning finished: 0.0016605043099033878, 0.0010130094877342326
    821
    EP 821 simulation finished, start learning
    1.42
    EP 821 learning finished: 0.0016936467952213263, 0.0009751534775919168
    822
    EP 822 simulation finished, start learning
    1.74
    EP 822 learning finished: 0.001584524305172863, 0.001054005345574538
    823
    EP 823 simulation finished, start learning
    1.11
    EP 823 learning finished: 0.0016953215187776038, 0.001322703965523549
    824
    EP 824 simulation finished, start learning
    1.2
    EP 824 learning finished: 0.0017628452129514471, 0.001346349348203546
    825
    EP 825 simulation finished, start learning
    1.4
    EP 825 learning finished: 0.003625414943023486, 0.0016219794820096685
    826
    EP 826 simulation finished, start learning
    1.31
    EP 826 learning finished: 0.005549583055849183, 0.0018869082515463969
    827
    EP 827 simulation finished, start learning
    1.2
    EP 827 learning finished: 0.005573269668612824, 0.0019795615113219847
    828
    EP 828 simulation finished, start learning
    0.92
    EP 828 learning finished: 0.005523837873475657, 0.0019248628573252564
    829
    EP 829 simulation finished, start learning
    1.62
    EP 829 learning finished: 0.005549040094525282, 0.0019352029051571407
    830
    EP 830 simulation finished, start learning
    1.89
    EP 830 learning finished: 0.0036541913505239165, 0.001694125857964579
    831
    EP 831 simulation finished, start learning
    1.21
    EP 831 learning finished: 0.0018054355300449743, 0.0013425640394562066
    832
    EP 832 simulation finished, start learning
    0.9
    EP 832 learning finished: 0.001803823810742955, 0.0013551641992872732
    833
    EP 833 simulation finished, start learning
    1.5
    EP 833 learning finished: 0.0016448967640236845, 0.0013781213702544666
    834
    EP 834 simulation finished, start learning
    0.89
    EP 834 learning finished: 0.0018201726022724074, 0.0014577190831707536
    835
    EP 835 simulation finished, start learning
    0.99
    EP 835 learning finished: 0.0034726419094610605, 0.0019297447239280767
    836
    EP 836 simulation finished, start learning
    1.11
    EP 836 learning finished: 0.0033815119902200763, 0.001974096482535733
    837
    EP 837 simulation finished, start learning
    1.12
    EP 837 learning finished: 0.002004122829731614, 0.0014127537694511953
    838
    EP 838 simulation finished, start learning
    1.8
    EP 838 learning finished: 0.0020676006536461955, 0.0015151336410911636
    839
    EP 839 simulation finished, start learning
    1.71
    EP 839 learning finished: 0.0017284996329749447, 0.0011577735328357857
    840
    EP 840 simulation finished, start learning
    1.52
    EP 840 learning finished: 0.0018482821526895066, 0.0011794601883963196
    841
    EP 841 simulation finished, start learning
    1.11
    EP 841 learning finished: 0.004249710650849892, 0.0017695172262340482
    842
    EP 842 simulation finished, start learning
    1.23
    EP 842 learning finished: 0.0030460608950484414, 0.0015069928124062197
    843
    EP 843 simulation finished, start learning
    1.71
    EP 843 learning finished: 0.0031048182315807082, 0.0015533369595649067
    844
    EP 844 simulation finished, start learning
    1.2
    EP 844 learning finished: 0.0031645661532486713, 0.0015500007006371181
    845
    EP 845 simulation finished, start learning
    2.0
    EP 845 learning finished: 0.0020268479159675223, 0.0012563377509859919
    846
    EP 846 simulation finished, start learning
    1.35
    EP 846 learning finished: 0.0020419386942123197, 0.0014365192889241503
    847
    EP 847 simulation finished, start learning
    1.04
    EP 847 learning finished: 0.001653249033996218, 0.0013495517499061787
    848
    EP 848 simulation finished, start learning
    1.14
    EP 848 learning finished: 0.0029505773560281057, 0.001686441126732774
    849
    EP 849 simulation finished, start learning
    1.62
    EP 849 learning finished: 0.003443443024245596, 0.001721019607546855
    850
    EP 850 simulation finished, start learning
    0.71
    EP 850 learning finished: 0.0018257960985827992, 0.0013829529434126187
    851
    EP 851 simulation finished, start learning
    1.52
    EP 851 learning finished: 0.0032750729815411717, 0.0019301286340951407
    852
    EP 852 simulation finished, start learning
    1.9
    EP 852 learning finished: 0.0034168112733183875, 0.001647507780823787
    853
    EP 853 simulation finished, start learning
    0.95
    EP 853 learning finished: 0.0031888732712737067, 0.0016535464381295384
    854
    EP 854 simulation finished, start learning
    1.92
    EP 854 learning finished: 0.0031161759292643865, 0.0017731367034912182
    855
    EP 855 simulation finished, start learning
    1.02
    EP 855 learning finished: 0.0018735276321457661, 0.0012954572221790528
    856
    EP 856 simulation finished, start learning
    1.53
    EP 856 learning finished: 0.002836951713316438, 0.001458365325267626
    857
    EP 857 simulation finished, start learning
    1.43
    EP 857 learning finished: 0.0017566019323176363, 0.0011177543313637035
    858
    EP 858 simulation finished, start learning
    2.33
    EP 858 learning finished: 0.001960910681226835, 0.0011413090343754817
    859
    EP 859 simulation finished, start learning
    1.32
    EP 859 learning finished: 0.0019398229776079745, 0.0013446272355330062
    860
    EP 860 simulation finished, start learning
    1.33
    EP 860 learning finished: 0.0018575162381182508, 0.0013256160161697987
    861
    EP 861 simulation finished, start learning
    1.11
    EP 861 learning finished: 0.0019207408156241002, 0.0013027265872462834
    862
    EP 862 simulation finished, start learning
    2.43
    EP 862 learning finished: 0.0019848379010021233, 0.0013302642353127208
    863
    EP 863 simulation finished, start learning
    1.04
    EP 863 learning finished: 0.0015349271892262095, 0.0010681117186018458
    864
    EP 864 simulation finished, start learning
    1.63
    EP 864 learning finished: 0.0015833295999740926, 0.0010672179221185019
    865
    EP 865 simulation finished, start learning
    1.29
    EP 865 learning finished: 0.0017684130623294436, 0.001110668440516586
    866
    EP 866 simulation finished, start learning
    1.83
    EP 866 learning finished: 0.0014455471473091563, 0.0010152938096451198
    867
    EP 867 simulation finished, start learning
    0.82
    EP 867 learning finished: 0.0014603671149835407, 0.001008782646845674
    868
    EP 868 simulation finished, start learning
    1.68
    EP 868 learning finished: 0.001840165433769414, 0.0012432486414368782
    869
    EP 869 simulation finished, start learning
    1.49
    EP 869 learning finished: 0.001837485113377113, 0.0015183923167156148
    870
    EP 870 simulation finished, start learning
    2.01
    EP 870 learning finished: 0.0020971971652272714, 0.0015692765065846714
    871
    EP 871 simulation finished, start learning
    1.62
    EP 871 learning finished: 0.0020085273007488524, 0.001636659426369036
    872
    EP 872 simulation finished, start learning
    0.81
    EP 872 learning finished: 0.0016909676160459242, 0.0013389972838287652
    873
    EP 873 simulation finished, start learning
    1.91
    EP 873 learning finished: 0.0017317175753319702, 0.0011114086833542796
    874
    EP 874 simulation finished, start learning
    0.69
    EP 874 learning finished: 0.0017611910848671641, 0.0011666936258496463
    875
    EP 875 simulation finished, start learning
    1.41
    EP 875 learning finished: 0.0018233989747713508, 0.001407441956467655
    876
    EP 876 simulation finished, start learning
    1.51
    EP 876 learning finished: 0.001619142575993493, 0.0012850814608089678
    877
    EP 877 simulation finished, start learning
    1.65
    EP 877 learning finished: 0.0016231027673718756, 0.0014169783361322648
    878
    EP 878 simulation finished, start learning
    1.39
    EP 878 learning finished: 0.001648617524846685, 0.001325979382118092
    879
    EP 879 simulation finished, start learning
    2.13
    EP 879 learning finished: 0.0018364195221671764, 0.0013935685609389736
    880
    EP 880 simulation finished, start learning
    1.07
    EP 880 learning finished: 0.0034591375272761127, 0.0016319418634639742
    881
    EP 881 simulation finished, start learning
    1.3
    EP 881 learning finished: 0.003454319796450278, 0.0015184645200593632
    882
    EP 882 simulation finished, start learning
    1.71
    EP 882 learning finished: 0.005153653464027741, 0.0020994609464946024
    883
    EP 883 simulation finished, start learning
    1.55
    EP 883 learning finished: 0.0033696633385673636, 0.0017223867920090896
    884
    EP 884 simulation finished, start learning
    1.11
    EP 884 learning finished: 0.0018069524296500106, 0.0014592211539721234
    885
    EP 885 simulation finished, start learning
    1.89
    EP 885 learning finished: 0.0028659254248886157, 0.0016901003059667612
    886
    EP 886 simulation finished, start learning
    1.68
    EP 886 learning finished: 0.0016969707771430365, 0.0013313871625305989
    887
    EP 887 simulation finished, start learning
    1.61
    EP 887 learning finished: 0.0030503287332876994, 0.001728646764394881
    888
    EP 888 simulation finished, start learning
    1.64
    EP 888 learning finished: 0.003971849769630609, 0.0019387582721233088
    889
    EP 889 simulation finished, start learning
    2.12
    EP 889 learning finished: 0.00521005883548753, 0.002151681860935284
    890
    EP 890 simulation finished, start learning
    1.43
    EP 890 learning finished: 0.007297187014916773, 0.0023018535575580756
    891
    EP 891 simulation finished, start learning
    1.23
    EP 891 learning finished: 0.005176750739916969, 0.0019087537372707668
    892
    EP 892 simulation finished, start learning
    1.53
    EP 892 learning finished: 0.005297274730136081, 0.0020783782145355665
    893
    EP 893 simulation finished, start learning
    2.01
    EP 893 learning finished: 0.0029980981223616254, 0.0016883332266453739
    894
    EP 894 simulation finished, start learning
    1.21
    EP 894 learning finished: 0.003127755149293421, 0.0016851081631201008
    895
    EP 895 simulation finished, start learning
    1.44
    EP 895 learning finished: 0.004095940636346995, 0.0018563910521454119
    896
    EP 896 simulation finished, start learning
    1.77
    EP 896 learning finished: 0.0030495254055793453, 0.00156690821601894
    897
    EP 897 simulation finished, start learning
    1.31
    EP 897 learning finished: 0.004217739501559237, 0.0018506502070743853
    898
    EP 898 simulation finished, start learning
    1.3
    EP 898 learning finished: 0.00207214239020459, 0.0015022525744488036
    899
    EP 899 simulation finished, start learning
    1.41
    EP 899 learning finished: 0.001493410819680463, 0.0013361502432308232
    900
    EP 900 simulation finished, start learning
    1.49
    EP 900 learning finished: 0.001660303314451994, 0.0012882769208720408
    901
    EP 901 simulation finished, start learning
    0.98
    EP 901 learning finished: 0.0015783089124674632, 0.0008596320181920211
    902
    EP 902 simulation finished, start learning
    1.3
    EP 902 learning finished: 0.001560015872779985, 0.0009161700494952166
    903
    EP 903 simulation finished, start learning
    1.3
    EP 903 learning finished: 0.0015256071002838212, 0.001019180982925268
    904
    EP 904 simulation finished, start learning
    1.71
    EP 904 learning finished: 0.0019998415056932117, 0.0012579143269443687
    905
    EP 905 simulation finished, start learning
    1.04
    EP 905 learning finished: 0.0017938765493035191, 0.0012191086928279304
    906
    EP 906 simulation finished, start learning
    1.2
    EP 906 learning finished: 0.0019274916416200938, 0.0012094939518837815
    907
    EP 907 simulation finished, start learning
    1.49
    EP 907 learning finished: 0.0014060158872908346, 0.0010148831985124691
    908
    EP 908 simulation finished, start learning
    0.76
    EP 908 learning finished: 0.0015954416748762412, 0.0008635865818765373
    909
    EP 909 simulation finished, start learning
    1.41
    EP 909 learning finished: 0.0015612821470689653, 0.0008805229430629188
    910
    EP 910 simulation finished, start learning
    1.5
    EP 910 learning finished: 0.0015890556144491086, 0.001159486556390495
    911
    EP 911 simulation finished, start learning
    1.36
    EP 911 learning finished: 0.0016504620103567902, 0.0011699575804578661
    912
    EP 912 simulation finished, start learning
    0.89
    EP 912 learning finished: 0.0015858657763274664, 0.001158394807642793
    913
    EP 913 simulation finished, start learning
    1.61
    EP 913 learning finished: 0.003358330240796903, 0.0014741046147523917
    914
    EP 914 simulation finished, start learning
    1.72
    EP 914 learning finished: 0.0015808152790057761, 0.001220115687987781
    915
    EP 915 simulation finished, start learning
    1.32
    EP 915 learning finished: 0.0031883088412421046, 0.0015247557300774872
    916
    EP 916 simulation finished, start learning
    1.62
    EP 916 learning finished: 0.0032133320500875584, 0.0017675185467756667
    917
    EP 917 simulation finished, start learning
    1.44
    EP 917 learning finished: 0.0019005170772778968, 0.0014746097779401168
    918
    EP 918 simulation finished, start learning
    1.53
    EP 918 learning finished: 0.0017348009364717143, 0.001286498776583637
    919
    EP 919 simulation finished, start learning
    1.85
    EP 919 learning finished: 0.0018440731757785045, 0.0010848758607083415
    920
    EP 920 simulation finished, start learning
    0.89
    EP 920 learning finished: 0.0014726028232168176, 0.0010000879274934722
    921
    EP 921 simulation finished, start learning
    1.51
    EP 921 learning finished: 0.0013894649740002762, 0.001029486755344422
    922
    EP 922 simulation finished, start learning
    1.4
    EP 922 learning finished: 0.0017774924291984462, 0.0011767562001085234
    923
    EP 923 simulation finished, start learning
    1.61
    EP 923 learning finished: 0.001970302014751013, 0.00120159928086494
    924
    EP 924 simulation finished, start learning
    1.83
    EP 924 learning finished: 0.002919136827525648, 0.001671240193790761
    925
    EP 925 simulation finished, start learning
    1.4
    EP 925 learning finished: 0.0017694306831792517, 0.0014739983588647476
    926
    EP 926 simulation finished, start learning
    1.44
    EP 926 learning finished: 0.0016697563220020584, 0.0012692417467861834
    927
    EP 927 simulation finished, start learning
    1.41
    EP 927 learning finished: 0.0032426840416484026, 0.0015094347792169078
    928
    EP 928 simulation finished, start learning
    1.02
    EP 928 learning finished: 0.0014925144274116064, 0.001145499121141924
    929
    EP 929 simulation finished, start learning
    1.63
    EP 929 learning finished: 0.0014587535875634217, 0.001339039553963417
    930
    EP 930 simulation finished, start learning
    1.55
    EP 930 learning finished: 0.0026355266392634746, 0.001551134735363983
    931
    EP 931 simulation finished, start learning
    1.3
    EP 931 learning finished: 0.002758727636354763, 0.0013651646812184264
    932
    EP 932 simulation finished, start learning
    1.8
    EP 932 learning finished: 0.00181448449825794, 0.0012520539339770191
    933
    EP 933 simulation finished, start learning
    1.04
    EP 933 learning finished: 0.001316509660274079, 0.0009610556234041686
    934
    EP 934 simulation finished, start learning
    1.31
    EP 934 learning finished: 0.0016094344144264028, 0.0011364319815978519
    935
    EP 935 simulation finished, start learning
    1.73
    EP 935 learning finished: 0.003246173500453864, 0.0014604387843319226
    936
    EP 936 simulation finished, start learning
    1.52
    EP 936 learning finished: 0.003273394823750664, 0.0014368307488332732
    937
    EP 937 simulation finished, start learning
    0.99
    EP 937 learning finished: 0.0032699843838699394, 0.0013872601215378921
    938
    EP 938 simulation finished, start learning
    1.73
    EP 938 learning finished: 0.0032013896241878115, 0.001441900798068237
    939
    EP 939 simulation finished, start learning
    2.01
    EP 939 learning finished: 0.0032286447012381814, 0.00147910545038125
    940
    EP 940 simulation finished, start learning
    1.44
    EP 940 learning finished: 0.004884293428659703, 0.0017689982860880822
    941
    EP 941 simulation finished, start learning
    0.71
    EP 941 learning finished: 0.004842158268946406, 0.0017665545174641768
    942
    EP 942 simulation finished, start learning
    2.1
    EP 942 learning finished: 0.0032455298182796355, 0.0015165409345415288
    943
    EP 943 simulation finished, start learning
    1.1
    EP 943 learning finished: 0.0015532733183434289, 0.0011501026665256358
    944
    EP 944 simulation finished, start learning
    1.52
    EP 944 learning finished: 0.0014219760058153952, 0.0009172525534400696
    945
    EP 945 simulation finished, start learning
    1.41
    EP 945 learning finished: 0.0013998921068818504, 0.00096627971222496
    946
    EP 946 simulation finished, start learning
    1.23
    EP 946 learning finished: 0.0013879200559046012, 0.0009042483750398844
    947
    EP 947 simulation finished, start learning
    0.93
    EP 947 learning finished: 0.0014126171592430126, 0.0009209513767209169
    948
    EP 948 simulation finished, start learning
    1.59
    EP 948 learning finished: 0.0013956853240522094, 0.0009545937326974988
    949
    EP 949 simulation finished, start learning
    2.04
    EP 949 learning finished: 0.0017350412080361366, 0.0012492471052721517
    950
    EP 950 simulation finished, start learning
    1.13
    EP 950 learning finished: 0.001676228141963776, 0.0012043345845089704
    951
    EP 951 simulation finished, start learning
    1.11
    EP 951 learning finished: 0.002845585213105295, 0.0014124181493867145
    952
    EP 952 simulation finished, start learning
    0.93
    EP 952 learning finished: 0.0016086829682002701, 0.0011442006331983852
    953
    EP 953 simulation finished, start learning
    1.71
    EP 953 learning finished: 0.00166718058599713, 0.0010614478588326648
    954
    EP 954 simulation finished, start learning
    1.11
    EP 954 learning finished: 0.0016035801042484366, 0.0012531387200566495
    955
    EP 955 simulation finished, start learning
    1.85
    EP 955 learning finished: 0.0014799912327347897, 0.0011225308773414138
    956
    EP 956 simulation finished, start learning
    1.89
    EP 956 learning finished: 0.0031444321269447073, 0.0016789748583002269
    957
    EP 957 simulation finished, start learning
    1.22
    EP 957 learning finished: 0.003139222523422802, 0.0016345822411866588
    958
    EP 958 simulation finished, start learning
    1.7
    EP 958 learning finished: 0.0031023352347077093, 0.0018868319414977464
    959
    EP 959 simulation finished, start learning
    1.72
    EP 959 learning finished: 0.00372642956561009, 0.0020450693085726592
    960
    EP 960 simulation finished, start learning
    1.71
    EP 960 learning finished: 0.002620303273950242, 0.0015723504612233559
    961
    EP 961 simulation finished, start learning
    1.53
    EP 961 learning finished: 0.0017099681621601803, 0.0014324745725478039
    962
    EP 962 simulation finished, start learning
    1.2
    EP 962 learning finished: 0.0015720503940030994, 0.0013715476129606923
    963
    EP 963 simulation finished, start learning
    1.6
    EP 963 learning finished: 0.0014920651832497356, 0.0012141393233999764
    964
    EP 964 simulation finished, start learning
    1.31
    EP 964 learning finished: 0.0016960831073760119, 0.0013568052445004735
    965
    EP 965 simulation finished, start learning
    0.83
    EP 965 learning finished: 0.0016326381225546527, 0.0012881294067176678
    966
    EP 966 simulation finished, start learning
    1.61
    EP 966 learning finished: 0.0014205548412213886, 0.0011626775875074695
    967
    EP 967 simulation finished, start learning
    1.23
    EP 967 learning finished: 0.001418700410219742, 0.0011219181709793229
    968
    EP 968 simulation finished, start learning
    1.32
    EP 968 learning finished: 0.0017507250310551115, 0.0012887390691697458
    969
    EP 969 simulation finished, start learning
    1.1
    EP 969 learning finished: 0.0018705741924165409, 0.0013242948679224043
    970
    EP 970 simulation finished, start learning
    1.41
    EP 970 learning finished: 0.0018077961148996735, 0.0010972809099296654
    971
    EP 971 simulation finished, start learning
    1.11
    EP 971 learning finished: 0.0017450121841342536, 0.001075414908113359
    972
    EP 972 simulation finished, start learning
    1.09
    EP 972 learning finished: 0.0014365297200669147, 0.0008200400678290976
    973
    EP 973 simulation finished, start learning
    1.21
    EP 973 learning finished: 0.0015597080110758554, 0.0010176090235027857
    974
    EP 974 simulation finished, start learning
    1.5
    EP 974 learning finished: 0.0036599525760365736, 0.0013885666242003936
    975
    EP 975 simulation finished, start learning
    1.82
    EP 975 learning finished: 0.0036515382970598864, 0.0016649192640040649
    976
    EP 976 simulation finished, start learning
    1.42
    EP 976 learning finished: 0.0035199143691571066, 0.0016664531324034708
    977
    EP 977 simulation finished, start learning
    1.22
    EP 977 learning finished: 0.001541095236081702, 0.0013430171380205043
    978
    EP 978 simulation finished, start learning
    1.51
    EP 978 learning finished: 0.0014867583130104833, 0.001167048769321018
    979
    EP 979 simulation finished, start learning
    1.03
    EP 979 learning finished: 0.0012887855298444606, 0.0008952392355106772
    980
    EP 980 simulation finished, start learning
    1.41
    EP 980 learning finished: 0.0013424576131448873, 0.0011074372315048227
    981
    EP 981 simulation finished, start learning
    1.52
    EP 981 learning finished: 0.0014745220492891065, 0.0012916278477916876
    982
    EP 982 simulation finished, start learning
    1.89
    EP 982 learning finished: 0.002924750270498322, 0.001609849164513714
    983
    EP 983 simulation finished, start learning
    1.42
    EP 983 learning finished: 0.0030129506860028243, 0.0016588721779893234
    984
    EP 984 simulation finished, start learning
    0.89
    EP 984 learning finished: 0.0038485923777393793, 0.0016924077450196269
    985
    EP 985 simulation finished, start learning
    1.69
    EP 985 learning finished: 0.0028982528899028637, 0.0015478750990508179
    986
    EP 986 simulation finished, start learning
    1.91
    EP 986 learning finished: 0.0018895021315259442, 0.0013722486937878624
    987
    EP 987 simulation finished, start learning
    1.82
    EP 987 learning finished: 0.003064237169247974, 0.0016648573316461561
    988
    EP 988 simulation finished, start learning
    2.21
    EP 988 learning finished: 0.004702955195451608, 0.0017632838602465266
    989
    EP 989 simulation finished, start learning
    1.34
    EP 989 learning finished: 0.004698010514584183, 0.0017873519241327074
    990
    EP 990 simulation finished, start learning
    1.4
    EP 990 learning finished: 0.004658889138845268, 0.0017362608060391693
    991
    EP 991 simulation finished, start learning
    1.29
    EP 991 learning finished: 0.004618207670876728, 0.0020222604703648748
    992
    EP 992 simulation finished, start learning
    1.01
    EP 992 learning finished: 0.002970949455285421, 0.0016516298255755402
    993
    EP 993 simulation finished, start learning
    1.32
    EP 993 learning finished: 0.0028960465549943137, 0.0015260496094319602
    994
    EP 994 simulation finished, start learning
    0.82
    EP 994 learning finished: 0.0015678457069832064, 0.0011894811674414048
    995
    EP 995 simulation finished, start learning
    1.29
    EP 995 learning finished: 0.0013699587896418564, 0.000780262511853517
    996
    EP 996 simulation finished, start learning
    1.14
    EP 996 learning finished: 0.0013252264630345678, 0.0008166642593222898
    997
    EP 997 simulation finished, start learning
    2.08
    EP 997 learning finished: 0.002621614149013851, 0.0012198482802462014
    998
    EP 998 simulation finished, start learning
    1.31
    EP 998 learning finished: 0.0036719076330982148, 0.0016375514798869044
    999
    EP 999 simulation finished, start learning
    1.95
    EP 999 learning finished: 0.003541595082136695, 0.0019254284528559633



```python
v_opt, pi_opt= q_opt.q.yield_v_pi()
```


```python
v_opt
```




    0    20.369774
    1    21.618667
    2    29.367012
    dtype: float64




```python
pi_opt
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
for s in state_space:
    for a in control_space:
        print(f'state: {s}, action_buffer: {[a]}, action to take: {get_opt_act(q_opt.q, model, s, [a])}')
        
pi_opt = lambda x: get_opt_act(q_opt.q, model, x[0], [x[1]])
```

    state: 0, action_buffer: [-1], action to take: 1
    state: 0, action_buffer: [1], action to take: 1
    state: 1, action_buffer: [-1], action to take: 1
    state: 1, action_buffer: [1], action to take: -1
    state: 2, action_buffer: [-1], action to take: 1
    state: 2, action_buffer: [1], action to take: -1



```python
q_opt.q
```




                0          1          2
    -1  17.100144  19.850458  29.367012
     1  20.369774  21.618667  18.113938




```python
def eval_pi_model(env, q, model, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = get_opt_act(q, model, obs, action_buffer)
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
#     return np.mean(ret) ,ret
    return np.mean(r_ls), r_ls.mean(axis = 1)
```


```python
val, v_samples_lin = eval_pi_model(env, q_opt.q, model, 0.95)
val
```




    1.5311848690079488




```python
exp2_opt = v_samples_lin
```


```python
from matplotlib.pyplot import figure

figure(figsize=(8, 6), dpi=100)
sns.distplot(exp2_opt, label = 'Opt. Action Selection')
sns.distplot(exp2_model, label = 'model')
sns.distplot(exp2_random, label = 'random')
sns.distplot(exp1_opt, label = 'Augmented MDP')
plt.xlabel("average reward")
plt.ylabel("freq.")
plt.legend()
# plt.set_size_inches(18.5, 10.5)
plt.savefig('exp2.png', dpi=100)
```


```python
plt.savefig('exp1.png')
```


    <Figure size 432x288 with 0 Axes>



```python
# For a given policy function, we solve the system A^{\pi}x=b^{\pi} to get the optimal value function

def linear_system(pi, trans_prob, reward, state_space, gamma):
    n = len(state_space)
    A = np.zeros(shape = (n,n))
    b = np.zeros(n)
    for i, s in enumerate(state_space):
        for j, next_s in enumerate(state_space):
            A[i,j] = trans_prob(next_s, s, pi(s))
            
        b[i] = -reward(s, pi(s))
        print(f'policy({s}): {pi(s)}')
    A = gamma*A-np.eye(n)
    return A, b
        
```


```python
from numpy.linalg import solve
```


```python
# policy of opt action selection
A, b = linear_system(pi_opt, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): 1
    policy((1, 1)): -1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.  ,  0.76, -1.  ,  0.  ,  0.  ,  0.19],
            [ 0.57,  0.  ,  0.  , -1.  ,  0.38,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([27.27312864, 29.76118804, 27.71489735, 30.27493478, 38.76118804,
           26.82312864])




```python
pi_AMDP_func = lambda x:pi_AMDP[x]
```


```python
# policy of AMDP

A, b = linear_system(pi_AMDP_func, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): -1
    policy((1, 1)): -1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.76,  0.  , -1.  ,  0.  ,  0.19,  0.  ],
            [ 0.57,  0.  ,  0.  , -1.  ,  0.38,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([27.27312864, 29.76118804, 28.09220349, 30.27493478, 38.76118804,
           26.82312864])




```python
# policy of planning based mathod
```


```python
A, b = linear_system(pi_model, trans_prob_d, reward_d,  state_space_d, alpha)
A, b
```

    policy((0, -1)): 1
    policy((0, 1)): 1
    policy((1, -1)): 1
    policy((1, 1)): 1
    policy((2, -1)): 1
    policy((2, 1)): -1





    (array([[-1.  ,  0.95,  0.  ,  0.  ,  0.  ,  0.  ],
            [ 0.  , -1.  ,  0.  ,  0.95,  0.  ,  0.  ],
            [ 0.  ,  0.76, -1.  ,  0.  ,  0.  ,  0.19],
            [ 0.  ,  0.57,  0.  , -1.  ,  0.  ,  0.38],
            [ 0.  ,  0.  ,  0.  ,  0.95, -1.  ,  0.  ],
            [ 0.  ,  0.  ,  0.  ,  0.  ,  0.95, -1.  ]]),
     array([  1.,  -1.,  -0.,  -0., -10.,  10.]))




```python
solve(A, b)
```




    array([ 2.91797923,  4.12418866,  3.60329944,  3.28861965, 13.12418866,
            2.46797923])




```python

```


```python

```


```python

```


```python

```


```python
pi_d
```




    (0, -1)    1
    (0, 1)     1
    (1, -1)   -1
    (1, 1)    -1
    (2, -1)    1
    (2, 1)    -1
    dtype: int64




```python
def eval_pi_model(env, pi, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = pi[(obs, action_buffer[0])]
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
#     return np.mean(ret) ,ret
    return np.mean(r_ls), r_ls.mean(axis = 1)
```


```python
val, v_samples_lin = eval_pi_model(env, pi_d, 0.95)
val
```




    1.5329450309428467




```python

```


```python

```


```python

```


```python

```


```python
# Q-learning using LEARNING-BASED model, async reward collection and optimal action selection



class Q_learning:
    def __init__(self, q, env, delay_step, model, model_wrapper, epsilon = 0.1, MAX_ITER = 10000, episode_length = 100, threshold = 1e-3):
        self.q = q
        self.env = env
        self.delay_step = delay_step
        self.epsilon = epsilon
        self.MAX_ITER = MAX_ITER
        self.episode_length = episode_length
        self.threshold = threshold
        self.pool = Pool(400)
        self.qs = []
        self.action_memory = [-1]*delay_step
        self.model = model
        self.model_wrapper = model_wrapper
        
    def train(self, lr = lambda *args: 0.1):
        q = self.q
        env = self.env
        epsilon = self.epsilon
        MAX_ITER = self.MAX_ITER
        episode_length = self.episode_length
        threshold = self.threshold
        
        for i in range(MAX_ITER):
            print(i)
            obs = env.reset(random = True)
            r_ls = []
            model = self.model_wrapper(self.model)
            for _ in range(100):
                
                
                action, is_random = (np.random.choice(control_space), True) \
                                    if np.random.uniform() < epsilon else \
                                    (get_opt_act(q, model, obs, self.action_memory), False)
                last_action = self.action_memory.pop(0)
                self.action_memory.append(action)
                next_obs, reward, _, _ = env.step(action)
                self.pool.append((obs, next_obs, last_action, reward, is_random))
                r_ls.append(reward)
                obs = next_obs
            if not self.pool.isFull:
                continue
            print(f'EP {i} simulation finished, start learning')
            q_new = self.update_q(q, self.pool ,i, 0.95, lr)
            self.update_model(self.pool)
            self.q = q_new
            print(np.mean(r_ls))
            self.qs.append(q_new)
            print(f'EP {i} learning finished: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
            if q_new.difference(q).L_1() < threshold and False:
                break
            else:
                q = q_new
        return q_new
    
    def update_model(self, pool):
        tmp = []
        for ele in pool:
            tmp.append(ele)
    
        x_train = np.array(tmp)[:, [0,2]]
        y_train = np.array(tmp)[:, 1]
        self.model.fit(x_train, y_train, batch_size=64, epochs=3, validation_split=0.2)
        
    
    def update_q(self, q, pool, t ,alpha, lr):
        pool = pool.shuffle()
        for _ in range(1):
            q = deepcopy(q)
#           
            for s, next_s, a, r, is_random in pool:
#                 if is_random: continue
                q[s, a] = q[s,a] + lr(t, s, a)*(r + alpha*q.max_value_act(next_s)[0]-q[s,a])
#             print(f'Learning ep: {i}: {q_new.difference(q).L_infty()}, {q_new.difference(q).L_1()}')
#             if q_new.difference(q).L_1() < self.threshold:
#                 break
#             else:
#                 q = q_new
        return q

def get_opt_act(q,model, x, acts):
    act_idx = np.argmax([np.sum([q[s, u]*model(s, x, acts) for s in q.state_space]) for u in q.control_space])
    return q.control_space[act_idx]
```


```python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
```


```python
inputs =  keras.Input(shape=(2,))

layer1 = layers.Dense(64, activation="relu", name="layer1")(inputs)
layer1 = layers.Dropout(0.5)(layer1)
layer2 = layers.Dense(64, activation="relu", name="layer2")(layer1)
layer2 = layers.Dropout(0.5)(layer2)
outputs = layers.Dense(3, activation="softmax", name="output_layer")(layer2)

model = keras.Model(inputs = inputs, outputs = outputs)
model.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.RMSprop(),
    metrics=["accuracy"],
)


model.summary()
```

    Model: "model"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    input_1 (InputLayer)         [(None, 2)]               0         
    _________________________________________________________________
    layer1 (Dense)               (None, 64)                192       
    _________________________________________________________________
    dropout (Dropout)            (None, 64)                0         
    _________________________________________________________________
    layer2 (Dense)               (None, 64)                4160      
    _________________________________________________________________
    dropout_1 (Dropout)          (None, 64)                0         
    _________________________________________________________________
    output_layer (Dense)         (None, 3)                 195       
    =================================================================
    Total params: 4,547
    Trainable params: 4,547
    Non-trainable params: 0
    _________________________________________________________________



```python
tmp = []
for ele in q_learn.pool:
    tmp.append(ele)
    
x_train = np.array(tmp)[:, [0,2]]
y_train = np.array(tmp)[:, 1]
def one_hot_encode(y):
    y = y.astype(int)
    elements = set(y)
    n = len(elements)
    ret = np.zeros((len(y), n))
    ret[np.arange(y.size),y] = 1
    return ret
# y_train = one_hot_encode(y_train)
print(x_train.shape, y_train.shape)
```

    (400, 2) (400,)



```python
model.predict([[1, 1]])[0,0]
```




    0.28935617




```python
history = model.fit(x_train, y_train, batch_size=64, epochs=10, validation_split=0.2)
```

    Train on 320 samples, validate on 80 samples
    Epoch 1/10
    320/320 [==============================] - 0s 288us/sample - loss: 0.7200 - accuracy: 0.8156 - val_loss: 0.7028 - val_accuracy: 0.8000
    Epoch 2/10
    320/320 [==============================] - 0s 196us/sample - loss: 0.7178 - accuracy: 0.8250 - val_loss: 0.7029 - val_accuracy: 0.8000
    Epoch 3/10
    320/320 [==============================] - 0s 193us/sample - loss: 0.7206 - accuracy: 0.8156 - val_loss: 0.7034 - val_accuracy: 0.8000
    Epoch 4/10
    320/320 [==============================] - 0s 189us/sample - loss: 0.7223 - accuracy: 0.7937 - val_loss: 0.7030 - val_accuracy: 0.8000
    Epoch 5/10
    320/320 [==============================] - 0s 194us/sample - loss: 0.7185 - accuracy: 0.8156 - val_loss: 0.7026 - val_accuracy: 0.8000
    Epoch 6/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7228 - accuracy: 0.8094 - val_loss: 0.7036 - val_accuracy: 0.8000
    Epoch 7/10
    320/320 [==============================] - 0s 195us/sample - loss: 0.7166 - accuracy: 0.8094 - val_loss: 0.7030 - val_accuracy: 0.8000
    Epoch 8/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7212 - accuracy: 0.8156 - val_loss: 0.7024 - val_accuracy: 0.8000
    Epoch 9/10
    320/320 [==============================] - 0s 193us/sample - loss: 0.7305 - accuracy: 0.7719 - val_loss: 0.7032 - val_accuracy: 0.8000
    Epoch 10/10
    320/320 [==============================] - 0s 191us/sample - loss: 0.7183 - accuracy: 0.8188 - val_loss: 0.7020 - val_accuracy: 0.8000



```python
class Learning_Rate:
    def __init__(self, state_space, control_space):
        self.counter = dict(
                zip(list(product(state_space, control_space)), [0]*len(state_space)*len(control_space))
        )
    def __call__(self, i, s, a):
        self.counter[(s, a)] += 1
        return 1/self.counter[(s, a)]
# lr_fn = Learning_Rate(state_space, control_space)
# lr_fn = lambda i,s,a: 1/(i+1)
def model_wrapper(model):
    def m(s, x, acts):
        inputs = np.array([[x, acts[0]]])
        return model.predict(inputs)[0, int(s)]
    return m
lr_fn = lambda i,s,a: 0.1
env = delayed_wrapper(Env(), 1)
q = Q_table(random = False)
q_learn = Q_learning(q, env, 1, model, model_wrapper, MAX_ITER = 1000)
learned_q = q_learn.train(lr_fn)
```


```python
m = model_wrapper(model)
m(0, 1, [1])
```




    0.35112822




```python
v_l, pi_l= q_learn.q.yield_v_pi()
```


```python
v_l
```




    0    41.123725
    1    42.282631
    2    50.209974
    dtype: float64




```python
pi_l
```




    0    1
    1    1
    2   -1
    dtype: int64




```python
def eval_pi_model(env, q, model, model_wrapper, discount_factor, MAX_ITER = 1000000):

    path_len = int(np.log(1e-3)/np.log(discount_factor))
    path_num = int(MAX_ITER/path_len)
    obs = 0
    disc_arr = np.array([discount_factor**n for n in range(path_len)])
    r_ls = np.array([[np.nan]*path_len for _ in range(path_num)])
    ret = []
    delay_step = env.num
    action_buffer = [-1]*delay_step
    model = model_wrapper(model)
    for k in range(path_num):
        obs = env.reset(random= True)
        for i in range(path_len):
            action = get_opt_act(q, model, obs, action_buffer)
            action_buffer.pop(0)
            action_buffer.append(action)
            next_obs, reward, _, _ = env.step(action)
    #         discounted_cum_r = reward + discount_factor*discounted_cum_r
            r_ls[k, i] = reward
            obs = next_obs
        ret.append(np.cumsum(disc_arr*r_ls[k, :]))
    return np.mean(ret) ,ret
```


```python
val, v_samples_lin = eval_pi_model(env, q_learn.q, model, model_wrapper, 0.95)
val
```


```python
sns.distplot(v_samples_lin, label = 'model')
plt.legend()
```


```python

```


```python

```


```python

```


```python
df = np.arange(0, 100).reshape(25, -1)
df
```




    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11],
           [12, 13, 14, 15],
           [16, 17, 18, 19],
           [20, 21, 22, 23],
           [24, 25, 26, 27],
           [28, 29, 30, 31],
           [32, 33, 34, 35],
           [36, 37, 38, 39],
           [40, 41, 42, 43],
           [44, 45, 46, 47],
           [48, 49, 50, 51],
           [52, 53, 54, 55],
           [56, 57, 58, 59],
           [60, 61, 62, 63],
           [64, 65, 66, 67],
           [68, 69, 70, 71],
           [72, 73, 74, 75],
           [76, 77, 78, 79],
           [80, 81, 82, 83],
           [84, 85, 86, 87],
           [88, 89, 90, 91],
           [92, 93, 94, 95],
           [96, 97, 98, 99]])




```python
# T = 3
T = 3
[N, D] = df.shape
dataX = np.zeros((N - T + 1, T, D))
for i in range(T, N + 1):
    dataX[i - T] = df[i - T:i, :]
    
dataX

# unfolded LSTM
```




    array([[[ 0.,  1.,  2.,  3.],
            [ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.]],
    
           [[ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.],
            [12., 13., 14., 15.]],
    
           [[ 8.,  9., 10., 11.],
            [12., 13., 14., 15.],
            [16., 17., 18., 19.]],
    
           [[12., 13., 14., 15.],
            [16., 17., 18., 19.],
            [20., 21., 22., 23.]],
    
           [[16., 17., 18., 19.],
            [20., 21., 22., 23.],
            [24., 25., 26., 27.]],
    
           [[20., 21., 22., 23.],
            [24., 25., 26., 27.],
            [28., 29., 30., 31.]],
    
           [[24., 25., 26., 27.],
            [28., 29., 30., 31.],
            [32., 33., 34., 35.]],
    
           [[28., 29., 30., 31.],
            [32., 33., 34., 35.],
            [36., 37., 38., 39.]],
    
           [[32., 33., 34., 35.],
            [36., 37., 38., 39.],
            [40., 41., 42., 43.]],
    
           [[36., 37., 38., 39.],
            [40., 41., 42., 43.],
            [44., 45., 46., 47.]],
    
           [[40., 41., 42., 43.],
            [44., 45., 46., 47.],
            [48., 49., 50., 51.]],
    
           [[44., 45., 46., 47.],
            [48., 49., 50., 51.],
            [52., 53., 54., 55.]],
    
           [[48., 49., 50., 51.],
            [52., 53., 54., 55.],
            [56., 57., 58., 59.]],
    
           [[52., 53., 54., 55.],
            [56., 57., 58., 59.],
            [60., 61., 62., 63.]],
    
           [[56., 57., 58., 59.],
            [60., 61., 62., 63.],
            [64., 65., 66., 67.]],
    
           [[60., 61., 62., 63.],
            [64., 65., 66., 67.],
            [68., 69., 70., 71.]],
    
           [[64., 65., 66., 67.],
            [68., 69., 70., 71.],
            [72., 73., 74., 75.]],
    
           [[68., 69., 70., 71.],
            [72., 73., 74., 75.],
            [76., 77., 78., 79.]],
    
           [[72., 73., 74., 75.],
            [76., 77., 78., 79.],
            [80., 81., 82., 83.]],
    
           [[76., 77., 78., 79.],
            [80., 81., 82., 83.],
            [84., 85., 86., 87.]],
    
           [[80., 81., 82., 83.],
            [84., 85., 86., 87.],
            [88., 89., 90., 91.]],
    
           [[84., 85., 86., 87.],
            [88., 89., 90., 91.],
            [92., 93., 94., 95.]],
    
           [[88., 89., 90., 91.],
            [92., 93., 94., 95.],
            [96., 97., 98., 99.]]])




```python
dataX.shape
```




    (23, 3, 4)




```python
[np.random.choice(range(24), replace = True) for _ in range(10)]
```




    [2, 6, 23, 17, 22, 11, 1, 13, 22, 14]




```python
dataX.reshape(dataX.shape + (1,)).shape
```




    (23, 3, 4, 1)




```python
time = range(1,6)
disc = np.array([np.exp(-0.05*t) for t in time])
disc
```




    array([0.95122942, 0.90483742, 0.86070798, 0.81873075, 0.77880078])




```python
def A(lamb):
    return 0.016*(np.array([np.exp(-n*lamb) for n in time])*disc).sum()

def B(lamb):
    return 0.5*0.016*C(lamb)

def C(lamb):
    return (disc*np.array([np.exp(-(n-1)*lamb)-np.exp(-n*lamb) for n in time])).sum()
```


```python
lamb = 0.0154
A(lamb)+B(lamb), C(lamb)
```




    (0.06654174415562401, 0.06404516300888626)




```python

```

95a9424db04ec6a48897019e093793cdb1261f28
ipynb
Jupyter Notebook
simplyabcTask.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
simplyabcTask.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
simplyabcTask.ipynb
Theling/Delayed_MDP
db1a8808a05917a5915220947cf65268f0524fa9
MIT
Qwen/Qwen-72B
1. YES
2. YES