## This Notebook serves as a walthrough to irt models & demonstrate following objective:
  1. Obtain Item parameters of a 4 PL (parameter logistic) IRT model by training an equivalent Deep Neural Network.

  2. Realise smaller models such as 1PL via 4PL model trained using hyper parameter search viz. NAS functionality

### Item Response Theory: Brief context
It is also known as psychometric theory or latent response theory. It measures the ability or latent trait of a person by looking at the responses for items or questions asked using statistical models. It assumes as ability increases, probability of correct response also increases. This relationship can be depicted from Item Characteristic Curve (ICC) which is S-shaped or sigmoid curve. It calculates parameters like discrimination, difficulty and guessability for each item and ability for every user. 

### Model specification -- keras 4PL
    `4PL model` uses Four parameters difficulty, item discrimination, item guesssability and item slip to calculate the probability of correct response given the ability of person.
. . .

If $i$ be a $question$ and $j$ be a $user$. Then, Probability $P_{ij}$ of an $j^{th}$ $user$ responding correctly to $i^{th}$ $question$, where $a_i$ is the $discrimination$ parameter, $b_i$ is difficulty, $c_j$ is guessability and $s_j$ is probability of slip for all $questions$, is given by:

\begin{equation}\
P_{ij} =  c_i+ [(1-s_i) - c_i]Sigmoid(a_i(\theta_j-b_i))
\end{equation}
Where
\begin{equation}
\text{Sigmoid}(x) =  \frac{e^{x}}{1 + e^{x}}
\end{equation}

_________________________________________________

### 1. Load 1-Parameter logistic IRT dataset with values of item difficulties levels, user abilities & response of various users on various given items.


```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

def generate_train_test(train_data, size= 0.1):
    df_new = train_data[['question_code', 'user_id', 'correctness']]
    df_cols = df_new.columns
    xtrain, xtest, ytrain, ytest = train_test_split(df_new[df_cols[:-1]], df_new[df_cols[-1]], test_size=size)
    x_train_user = pd.get_dummies(xtrain['user_id']).values#contains the one-hot encoded user data
    x_train_questions = pd.get_dummies(xtrain['question_code']).values#contains the one-hot encoded questions

    y_train= ytrain.values
    print('shape of x_train_user:', x_train_user.shape,'\nshape of x_train_questions:', x_train_questions.shape,
          '\nshape of y_train:',y_train.shape)
    return train_data, xtrain, y_train, x_train_user, x_train_questions


path= "../datasets/irt_4pl_dataset.csv"
df= pd.read_csv(path)


train_data, xtrain, y_train, x_train_user, x_train_questions = generate_train_test(df, size= 0.05)
df.head(3)
```

    2021-08-05 12:17:12,406	INFO node.py:423 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2021-08-05_12-17-12_258/logs.
    2021-08-05 12:17:12,530	INFO services.py:363 -- Waiting for redis server at 127.0.0.1:64240 to respond...
    2021-08-05 12:17:12,663	INFO services.py:363 -- Waiting for redis server at 127.0.0.1:30595 to respond...
    2021-08-05 12:17:12,667	INFO services.py:760 -- Starting Redis shard with 20.0 GB max memory.
    2021-08-05 12:17:12,706	INFO services.py:1384 -- Starting the Plasma object store with 1.0 GB memory using /dev/shm.


    shape of x_train_user: (95000, 100) 
    shape of x_train_questions: (95000, 100) 
    shape of y_train: (95000,)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>question_code</th>
      <th>user_id</th>
      <th>difficulty</th>
      <th>ability</th>
      <th>response</th>
      <th>correctness</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>que_99</td>
      <td>user_99</td>
      <td>1.0</td>
      <td>0.559055</td>
      <td>0.391516</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>que_99</td>
      <td>user_99</td>
      <td>1.0</td>
      <td>0.559055</td>
      <td>0.391516</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>que_99</td>
      <td>user_99</td>
      <td>1.0</td>
      <td>0.559055</td>
      <td>0.391516</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>



_____________________________________

### 2. Instantiate a 4-Parameter Logistic IRT model, as primal model.


```python
from mlsquare.models import fourPL

primal = fourPL()
primal
```




    fourPL(IRT_param=None, constraint='NULL', control=[], data=None, max_guessing=1,
           na_action='NULL', start_val='NULL', type_='latent_trait')



___________________________________

### 3. Define initialization values for user & item parameters, boundary constraints or search space for hyper parameters. 
*Note:* Search space constraints are defined such that high regularization enforces 4PL model to imitates 1PL model, thus fitting the current 1PL dataset theoretically.

**4PL model as 1PL: Passing activity_regularizers l1 to hyperOpt search for layers (discrimination, guess, slip) corresponding to 1PL dataset, that is by pushing the guess_params & slip_params contribution in overall estimation to 0 and  disc_params's contribution to 1.**


```python
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.suggest.hyperopt import HyperOptSearch
from hyperopt import hp

src_space= {
    "slip_params.group_lasso.l1": hp.uniform("slip_params.group_lasso.l1", 0, 10),
    "guess_params.group_lasso.l1": hp.uniform("guess_params.group_lasso.l1", 0, 10),
    "disc_params.group_lasso.l1": hp.uniform("disc_params.group_lasso.l1", 0, 10),

}
nas_params= {'search_algo_name':'hyperOpt', 'search_space':src_space, 'union':False}

params_= {'disc_params':{'kernel_params':{'stddev':0.2}},
    'guess_params':{'kernel_params':{'stddev':0.2}},
          'slip_params':{'kernel_params':{'stddev':0.2}
                        }
         }
```

_________________________________________

### 4. "Dope" the primal model to obtain an equivalent 4PL Deep Neural Network on 1PL data, passing initializations & search space constraints.


```python
from mlsquare import dope

model_4t1 = dope(primal)
model_4t1.fit(x_user= x_train_user, x_questions= x_train_questions, y_vals= y_train, batch_size= 128, epochs=450,
        params=params_, nas_params=nas_params)

```

    Transpiling your model to it's Deep Neural Network equivalent...
    2021-08-05 12:20:05,090	ERROR worker.py:1412 -- Calling ray.init() again after it has already been called.


    
    Intitializing fit for fourPL model. . .
    Batch_size: 128; epochs: 450;
    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.4/13.6 GB
    


    2021-08-05 12:20:05,507	ERROR worker.py:1780 -- Warning: The actor WrappedFunc has size 19782331 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object.


    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 4/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.5/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'RUNNING': 1, 'PENDING': 3})
    PENDING trials:
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	PENDING
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	PENDING
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	PENDING
    RUNNING trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	RUNNING
    
    [2m[36m(pid=312)[0m 2021-08-05 12:20:06.470238: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    [2m[36m(pid=312)[0m 2021-08-05 12:20:07,839	ERROR worker.py:1412 -- Calling ray.init() again after it has already been called.
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m config val 1:
    [2m[36m(pid=312)[0m  key_list: ['disc_params', 'group_lasso'];
    [2m[36m(pid=312)[0m  values: {'l1': 2.7829298208870936, 'l2': 0}
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m config val 2:
    [2m[36m(pid=312)[0m  key_list: ['guess_params', 'group_lasso'];
    [2m[36m(pid=312)[0m  values: {'l1': 3.9434923495322796, 'l2': 0}
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m config val 3:
    [2m[36m(pid=312)[0m  key_list: ['slip_params', 'group_lasso'];
    [2m[36m(pid=312)[0m  values: {'l1': 5.964767729148698, 'l2': 0}
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m Intitializing fit for fourPL model. . .
    [2m[36m(pid=312)[0m Batch_size: 128; epochs: 450;
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.742924: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774150: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774212: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: d005995f49c5
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774229: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: d005995f49c5
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774332: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.32.3
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774364: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.32.3
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774373: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.32.3
    [2m[36m(pid=312)[0m 2021-08-05 12:20:08.774710: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    [2m[36m(pid=312)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    [2m[36m(pid=312)[0m 2021-08-05 12:20:09.082573: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    [2m[36m(pid=312)[0m 2021-08-05 12:20:09.082935: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m    1/2969 [..............................] - ETA: 37s - loss: 0.6022 - mae: 0.3906 - accuracy: 0.6875
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m   60/2969 [..............................] - ETA: 2s - loss: 0.5876 - mae: 0.3996 - accuracy: 0.6818 
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  115/2969 [>.............................] - ETA: 2s - loss: 0.5841 - mae: 0.3981 - accuracy: 0.6864
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  178/2969 [>.............................] - ETA: 2s - loss: 0.5764 - mae: 0.3944 - accuracy: 0.6935
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  242/2969 [=>............................] - ETA: 2s - loss: 0.5782 - mae: 0.3954 - accuracy: 0.6896
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  301/2969 [==>...........................] - ETA: 2s - loss: 0.5814 - mae: 0.3969 - accuracy: 0.6885
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  364/2969 [==>...........................] - ETA: 2s - loss: 0.5796 - mae: 0.3962 - accuracy: 0.6892
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  429/2969 [===>..........................] - ETA: 2s - loss: 0.5818 - mae: 0.3970 - accuracy: 0.6890
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  493/2969 [===>..........................] - ETA: 2s - loss: 0.5815 - mae: 0.3970 - accuracy: 0.6904
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  553/2969 [====>.........................] - ETA: 1s - loss: 0.5806 - mae: 0.3965 - accuracy: 0.6909
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  616/2969 [=====>........................] - ETA: 1s - loss: 0.5812 - mae: 0.3968 - accuracy: 0.6903
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  669/2969 [=====>........................] - ETA: 1s - loss: 0.5806 - mae: 0.3965 - accuracy: 0.6916
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  724/2969 [======>.......................] - ETA: 1s - loss: 0.5795 - mae: 0.3960 - accuracy: 0.6919
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  781/2969 [======>.......................] - ETA: 1s - loss: 0.5790 - mae: 0.3957 - accuracy: 0.6931
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  842/2969 [=======>......................] - ETA: 1s - loss: 0.5779 - mae: 0.3952 - accuracy: 0.6943
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  900/2969 [========>.....................] - ETA: 1s - loss: 0.5776 - mae: 0.3950 - accuracy: 0.6947
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m  963/2969 [========>.....................] - ETA: 1s - loss: 0.5773 - mae: 0.3949 - accuracy: 0.6947
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1017/2969 [=========>....................] - ETA: 1s - loss: 0.5777 - mae: 0.3951 - accuracy: 0.6942
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1077/2969 [=========>....................] - ETA: 1s - loss: 0.5780 - mae: 0.3953 - accuracy: 0.6939
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1140/2969 [==========>...................] - ETA: 1s - loss: 0.5784 - mae: 0.3955 - accuracy: 0.6939
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1192/2969 [===========>..................] - ETA: 1s - loss: 0.5785 - mae: 0.3955 - accuracy: 0.6937
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1248/2969 [===========>..................] - ETA: 1s - loss: 0.5793 - mae: 0.3959 - accuracy: 0.6927
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1304/2969 [============>.................] - ETA: 1s - loss: 0.5792 - mae: 0.3959 - accuracy: 0.6931
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1359/2969 [============>.................] - ETA: 1s - loss: 0.5792 - mae: 0.3959 - accuracy: 0.6928
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1418/2969 [=============>................] - ETA: 1s - loss: 0.5782 - mae: 0.3955 - accuracy: 0.6935
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1480/2969 [=============>................] - ETA: 1s - loss: 0.5784 - mae: 0.3956 - accuracy: 0.6934
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1540/2969 [==============>...............] - ETA: 1s - loss: 0.5785 - mae: 0.3956 - accuracy: 0.6934
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1604/2969 [===============>..............] - ETA: 1s - loss: 0.5795 - mae: 0.3962 - accuracy: 0.6927
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1667/2969 [===============>..............] - ETA: 1s - loss: 0.5796 - mae: 0.3961 - accuracy: 0.6928
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1729/2969 [================>.............] - ETA: 1s - loss: 0.5794 - mae: 0.3960 - accuracy: 0.6932
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1787/2969 [=================>............] - ETA: 1s - loss: 0.5799 - mae: 0.3961 - accuracy: 0.6929
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1840/2969 [=================>............] - ETA: 0s - loss: 0.5798 - mae: 0.3961 - accuracy: 0.6931
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1902/2969 [==================>...........] - ETA: 0s - loss: 0.5800 - mae: 0.3961 - accuracy: 0.6930
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 1966/2969 [==================>...........] - ETA: 0s - loss: 0.5796 - mae: 0.3960 - accuracy: 0.6934
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2026/2969 [===================>..........] - ETA: 0s - loss: 0.5797 - mae: 0.3959 - accuracy: 0.6935
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2091/2969 [====================>.........] - ETA: 0s - loss: 0.5797 - mae: 0.3959 - accuracy: 0.6937
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2157/2969 [====================>.........] - ETA: 0s - loss: 0.5795 - mae: 0.3958 - accuracy: 0.6945
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2222/2969 [=====================>........] - ETA: 0s - loss: 0.5795 - mae: 0.3959 - accuracy: 0.6944
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2281/2969 [======================>.......] - ETA: 0s - loss: 0.5796 - mae: 0.3959 - accuracy: 0.6941
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2344/2969 [======================>.......] - ETA: 0s - loss: 0.5794 - mae: 0.3958 - accuracy: 0.6943
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2402/2969 [=======================>......] - ETA: 0s - loss: 0.5797 - mae: 0.3959 - accuracy: 0.6940
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2462/2969 [=======================>......] - ETA: 0s - loss: 0.5797 - mae: 0.3958 - accuracy: 0.6943
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2516/2969 [========================>.....] - ETA: 0s - loss: 0.5798 - mae: 0.3959 - accuracy: 0.6940
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2576/2969 [=========================>....] - ETA: 0s - loss: 0.5798 - mae: 0.3959 - accuracy: 0.6942
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2639/2969 [=========================>....] - ETA: 0s - loss: 0.5800 - mae: 0.3960 - accuracy: 0.6941
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2699/2969 [==========================>...] - ETA: 0s - loss: 0.5802 - mae: 0.3961 - accuracy: 0.6940
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2759/2969 [==========================>...] - ETA: 0s - loss: 0.5805 - mae: 0.3963 - accuracy: 0.6936
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2822/2969 [===========================>..] - ETA: 0s - loss: 0.5805 - mae: 0.3963 - accuracy: 0.6936
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2881/2969 [============================>.] - ETA: 0s - loss: 0.5807 - mae: 0.3964 - accuracy: 0.6932
    [2m[36m(pid=312)[0m 
    [2m[36m(pid=312)[0m 2944/2969 [============================>.] - ETA: 0s - loss: 0.5805 - mae: 0.3963 - accuracy: 0.6934
    [2m[36m(pid=312)[0m 2969/2969 [==============================] - 3s 839us/step - loss: 0.5805 - mae: 0.3963 - accuracy: 0.6934
    Result for train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:
      checkpoint: weights_tune_[(0, 'disc_params.group_lasso.l1'), (5, 'guess_params.group_lasso.l1'),
        (8, 'slip_params.group_lasso.l1')].h5
      date: 2021-08-05_12-25-35
      done: true
      experiment_id: b2b59f5124084f44adf67b0312a02ab1
      hostname: d005995f49c5
      iterations_since_restore: 1
      mean_accuracy: 0.6933789253234863
      mean_error: 0.5804693102836609
      node_ip: 172.28.0.2
      pid: 312
      time_since_restore: 327.2989342212677
      time_this_iter_s: 327.2989342212677
      time_total_s: 327.2989342212677
      timestamp: 1628166335
      timesteps_since_restore: 0
      training_iteration: 1
      


    2021-08-05 12:25:36,059	ERROR worker.py:1780 -- Warning: The actor WrappedFunc has size 19782331 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object.


    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.7/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'TERMINATED': 1, 'PENDING': 3})
    PENDING trials:
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	PENDING
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	PENDING
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	PENDING
    TERMINATED trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	TERMINATED, [4 CPUs, 0 GPUs], [pid=312], 327 s, 1 iter, 0.693 acc
    
    [2m[36m(pid=313)[0m 2021-08-05 12:25:37.171612: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    [2m[36m(pid=313)[0m 2021-08-05 12:25:38,480	ERROR worker.py:1412 -- Calling ray.init() again after it has already been called.
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m config val 1:
    [2m[36m(pid=313)[0m  key_list: ['disc_params', 'group_lasso'];
    [2m[36m(pid=313)[0m  values: {'l1': 6.562664791374827, 'l2': 0}
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m config val 2:
    [2m[36m(pid=313)[0m  key_list: ['guess_params', 'group_lasso'];
    [2m[36m(pid=313)[0m  values: {'l1': 2.9186447579369856, 'l2': 0}
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m config val 3:
    [2m[36m(pid=313)[0m  key_list: ['slip_params', 'group_lasso'];
    [2m[36m(pid=313)[0m  values: {'l1': 1.3818327930197527, 'l2': 0}
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m Intitializing fit for fourPL model. . .
    [2m[36m(pid=313)[0m Batch_size: 128; epochs: 450;
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.332564: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.337904: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.337954: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: d005995f49c5
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.337964: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: d005995f49c5
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.338049: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.32.3
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.338076: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.32.3
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.338085: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.32.3
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.338282: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    [2m[36m(pid=313)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.625346: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    [2m[36m(pid=313)[0m 2021-08-05 12:25:39.625696: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m    1/2969 [..............................] - ETA: 38s - loss: 0.6050 - mae: 0.3900 - accuracy: 0.6875
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m   52/2969 [..............................] - ETA: 2s - loss: 0.5922 - mae: 0.4003 - accuracy: 0.6869 
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  108/2969 [>.............................] - ETA: 2s - loss: 0.5905 - mae: 0.3992 - accuracy: 0.6843
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  162/2969 [>.............................] - ETA: 2s - loss: 0.5763 - mae: 0.3926 - accuracy: 0.6966
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  217/2969 [=>............................] - ETA: 2s - loss: 0.5809 - mae: 0.3948 - accuracy: 0.6910
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  262/2969 [=>............................] - ETA: 2s - loss: 0.5846 - mae: 0.3967 - accuracy: 0.6880
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  317/2969 [==>...........................] - ETA: 2s - loss: 0.5839 - mae: 0.3961 - accuracy: 0.6901
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  368/2969 [==>...........................] - ETA: 2s - loss: 0.5837 - mae: 0.3963 - accuracy: 0.6892
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  422/2969 [===>..........................] - ETA: 2s - loss: 0.5850 - mae: 0.3968 - accuracy: 0.6894
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  473/2969 [===>..........................] - ETA: 2s - loss: 0.5848 - mae: 0.3967 - accuracy: 0.6905
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  527/2969 [====>.........................] - ETA: 2s - loss: 0.5849 - mae: 0.3969 - accuracy: 0.6902
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  579/2969 [====>.........................] - ETA: 2s - loss: 0.5846 - mae: 0.3966 - accuracy: 0.6906
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  622/2969 [=====>........................] - ETA: 2s - loss: 0.5848 - mae: 0.3967 - accuracy: 0.6904
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  681/2969 [=====>........................] - ETA: 2s - loss: 0.5834 - mae: 0.3961 - accuracy: 0.6919
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  724/2969 [======>.......................] - ETA: 2s - loss: 0.5831 - mae: 0.3960 - accuracy: 0.6917
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  782/2969 [======>.......................] - ETA: 2s - loss: 0.5827 - mae: 0.3957 - accuracy: 0.6929
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  834/2969 [=======>......................] - ETA: 2s - loss: 0.5819 - mae: 0.3953 - accuracy: 0.6939
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  893/2969 [========>.....................] - ETA: 1s - loss: 0.5810 - mae: 0.3949 - accuracy: 0.6949
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  943/2969 [========>.....................] - ETA: 1s - loss: 0.5810 - mae: 0.3950 - accuracy: 0.6946
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m  996/2969 [=========>....................] - ETA: 1s - loss: 0.5813 - mae: 0.3950 - accuracy: 0.6944
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1050/2969 [=========>....................] - ETA: 1s - loss: 0.5817 - mae: 0.3953 - accuracy: 0.6938
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1106/2969 [==========>...................] - ETA: 1s - loss: 0.5824 - mae: 0.3956 - accuracy: 0.6936
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1160/2969 [==========>...................] - ETA: 1s - loss: 0.5821 - mae: 0.3954 - accuracy: 0.6939
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1218/2969 [===========>..................] - ETA: 1s - loss: 0.5826 - mae: 0.3957 - accuracy: 0.6932
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1276/2969 [===========>..................] - ETA: 1s - loss: 0.5827 - mae: 0.3958 - accuracy: 0.6932
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1331/2969 [============>.................] - ETA: 1s - loss: 0.5826 - mae: 0.3957 - accuracy: 0.6932
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1381/2969 [============>.................] - ETA: 1s - loss: 0.5822 - mae: 0.3956 - accuracy: 0.6934
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1432/2969 [=============>................] - ETA: 1s - loss: 0.5820 - mae: 0.3956 - accuracy: 0.6933
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1491/2969 [==============>...............] - ETA: 1s - loss: 0.5821 - mae: 0.3956 - accuracy: 0.6934
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1542/2969 [==============>...............] - ETA: 1s - loss: 0.5822 - mae: 0.3956 - accuracy: 0.6934
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1597/2969 [===============>..............] - ETA: 1s - loss: 0.5831 - mae: 0.3961 - accuracy: 0.6928
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1642/2969 [===============>..............] - ETA: 1s - loss: 0.5833 - mae: 0.3961 - accuracy: 0.6931
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1698/2969 [================>.............] - ETA: 1s - loss: 0.5832 - mae: 0.3960 - accuracy: 0.6932
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1748/2969 [================>.............] - ETA: 1s - loss: 0.5834 - mae: 0.3961 - accuracy: 0.6931
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1791/2969 [=================>............] - ETA: 1s - loss: 0.5835 - mae: 0.3961 - accuracy: 0.6930
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1843/2969 [=================>............] - ETA: 1s - loss: 0.5835 - mae: 0.3960 - accuracy: 0.6931
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1896/2969 [==================>...........] - ETA: 1s - loss: 0.5835 - mae: 0.3960 - accuracy: 0.6933
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 1949/2969 [==================>...........] - ETA: 0s - loss: 0.5836 - mae: 0.3961 - accuracy: 0.6932
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2006/2969 [===================>..........] - ETA: 0s - loss: 0.5834 - mae: 0.3959 - accuracy: 0.6934
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2056/2969 [===================>..........] - ETA: 0s - loss: 0.5832 - mae: 0.3958 - accuracy: 0.6940
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2107/2969 [====================>.........] - ETA: 0s - loss: 0.5835 - mae: 0.3960 - accuracy: 0.6937
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2162/2969 [====================>.........] - ETA: 0s - loss: 0.5832 - mae: 0.3958 - accuracy: 0.6945
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2217/2969 [=====================>........] - ETA: 0s - loss: 0.5832 - mae: 0.3958 - accuracy: 0.6944
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2270/2969 [=====================>........] - ETA: 0s - loss: 0.5833 - mae: 0.3959 - accuracy: 0.6942
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2323/2969 [======================>.......] - ETA: 0s - loss: 0.5831 - mae: 0.3957 - accuracy: 0.6943
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2379/2969 [=======================>......] - ETA: 0s - loss: 0.5834 - mae: 0.3959 - accuracy: 0.6940
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2432/2969 [=======================>......] - ETA: 0s - loss: 0.5834 - mae: 0.3959 - accuracy: 0.6941
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2479/2969 [========================>.....] - ETA: 0s - loss: 0.5834 - mae: 0.3959 - accuracy: 0.6943
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2527/2969 [========================>.....] - ETA: 0s - loss: 0.5836 - mae: 0.3960 - accuracy: 0.6940
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2585/2969 [=========================>....] - ETA: 0s - loss: 0.5836 - mae: 0.3959 - accuracy: 0.6943
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2637/2969 [=========================>....] - ETA: 0s - loss: 0.5837 - mae: 0.3960 - accuracy: 0.6941
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2695/2969 [==========================>...] - ETA: 0s - loss: 0.5838 - mae: 0.3961 - accuracy: 0.6941
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2741/2969 [==========================>...] - ETA: 0s - loss: 0.5841 - mae: 0.3962 - accuracy: 0.6937
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2799/2969 [===========================>..] - ETA: 0s - loss: 0.5840 - mae: 0.3962 - accuracy: 0.6936
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2845/2969 [===========================>..] - ETA: 0s - loss: 0.5841 - mae: 0.3962 - accuracy: 0.6936


    2021-08-05 12:31:06,491	ERROR worker.py:1780 -- Warning: The actor WrappedFunc has size 19782331 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object.


    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2900/2969 [============================>.] - ETA: 0s - loss: 0.5842 - mae: 0.3963 - accuracy: 0.6933
    Result for train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:
      checkpoint: weights_tune_[(2, 'disc_params.group_lasso.l1'), (9, 'guess_params.group_lasso.l1'),
        (5, 'slip_params.group_lasso.l1')].h5
      date: 2021-08-05_12-31-06
      done: true
      experiment_id: acc4cfab0ec841ce9a8abf40a281c9f7
      hostname: d005995f49c5
      iterations_since_restore: 1
      mean_accuracy: 0.6934000253677368
      mean_error: 0.5841391086578369
      node_ip: 172.28.0.2
      pid: 313
      time_since_restore: 327.1404161453247
      time_this_iter_s: 327.1404161453247
      time_total_s: 327.1404161453247
      timestamp: 1628166666
      timesteps_since_restore: 0
      training_iteration: 1
      
    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.7/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'TERMINATED': 2, 'PENDING': 2})
    PENDING trials:
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	PENDING
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	PENDING
    TERMINATED trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	TERMINATED, [4 CPUs, 0 GPUs], [pid=312], 327 s, 1 iter, 0.693 acc
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	TERMINATED, [4 CPUs, 0 GPUs], [pid=313], 327 s, 1 iter, 0.693 acc
    
    [2m[36m(pid=313)[0m 
    [2m[36m(pid=313)[0m 2950/2969 [============================>.] - ETA: 0s - loss: 0.5841 - mae: 0.3963 - accuracy: 0.6934
    [2m[36m(pid=313)[0m 2969/2969 [==============================] - 3s 958us/step - loss: 0.5841 - mae: 0.3963 - accuracy: 0.6934
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:08.370318: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:09,732	ERROR worker.py:1412 -- Calling ray.init() again after it has already been called.
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m config val 1:
    [2m[36m(pid=3225)[0m  key_list: ['disc_params', 'group_lasso'];
    [2m[36m(pid=3225)[0m  values: {'l1': 6.634291754359234, 'l2': 0}
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m config val 2:
    [2m[36m(pid=3225)[0m  key_list: ['guess_params', 'group_lasso'];
    [2m[36m(pid=3225)[0m  values: {'l1': 7.145003759304225, 'l2': 0}
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m config val 3:
    [2m[36m(pid=3225)[0m  key_list: ['slip_params', 'group_lasso'];
    [2m[36m(pid=3225)[0m  values: {'l1': 5.195499108646523, 'l2': 0}
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m Intitializing fit for fourPL model. . .
    [2m[36m(pid=3225)[0m Batch_size: 128; epochs: 450;
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.623240: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628720: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628768: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: d005995f49c5
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628778: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: d005995f49c5
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628868: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.32.3
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628903: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.32.3
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.628912: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.32.3
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.629113: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    [2m[36m(pid=3225)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.938766: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    [2m[36m(pid=3225)[0m 2021-08-05 12:31:10.939086: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m    1/2969 [..............................] - ETA: 40s - loss: 0.6049 - mae: 0.3903 - accuracy: 0.6875
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m   54/2969 [..............................] - ETA: 2s - loss: 0.5904 - mae: 0.3998 - accuracy: 0.6863 
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  114/2969 [>.............................] - ETA: 2s - loss: 0.5877 - mae: 0.3983 - accuracy: 0.6856
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  175/2969 [>.............................] - ETA: 2s - loss: 0.5794 - mae: 0.3942 - accuracy: 0.6932
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  237/2969 [=>............................] - ETA: 2s - loss: 0.5811 - mae: 0.3953 - accuracy: 0.6897
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  291/2969 [=>............................] - ETA: 2s - loss: 0.5833 - mae: 0.3965 - accuracy: 0.6891
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  352/2969 [==>...........................] - ETA: 2s - loss: 0.5829 - mae: 0.3963 - accuracy: 0.6887
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  412/2969 [===>..........................] - ETA: 2s - loss: 0.5846 - mae: 0.3970 - accuracy: 0.6885
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  473/2969 [===>..........................] - ETA: 2s - loss: 0.5839 - mae: 0.3968 - accuracy: 0.6900
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  530/2969 [====>.........................] - ETA: 2s - loss: 0.5842 - mae: 0.3970 - accuracy: 0.6893
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  591/2969 [====>.........................] - ETA: 2s - loss: 0.5836 - mae: 0.3967 - accuracy: 0.6904
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  652/2969 [=====>........................] - ETA: 1s - loss: 0.5840 - mae: 0.3968 - accuracy: 0.6903
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  705/2969 [======>.......................] - ETA: 1s - loss: 0.5822 - mae: 0.3960 - accuracy: 0.6919
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  765/2969 [======>.......................] - ETA: 1s - loss: 0.5817 - mae: 0.3957 - accuracy: 0.6921
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  822/2969 [=======>......................] - ETA: 1s - loss: 0.5814 - mae: 0.3955 - accuracy: 0.6932
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  885/2969 [=======>......................] - ETA: 1s - loss: 0.5800 - mae: 0.3948 - accuracy: 0.6945
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m  946/2969 [========>.....................] - ETA: 1s - loss: 0.5801 - mae: 0.3949 - accuracy: 0.6944
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1009/2969 [=========>....................] - ETA: 1s - loss: 0.5805 - mae: 0.3951 - accuracy: 0.6938
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1066/2969 [=========>....................] - ETA: 1s - loss: 0.5810 - mae: 0.3954 - accuracy: 0.6935
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1129/2969 [==========>...................] - ETA: 1s - loss: 0.5814 - mae: 0.3955 - accuracy: 0.6935
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1192/2969 [===========>..................] - ETA: 1s - loss: 0.5814 - mae: 0.3955 - accuracy: 0.6934
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1249/2969 [===========>..................] - ETA: 1s - loss: 0.5821 - mae: 0.3959 - accuracy: 0.6925
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1309/2969 [============>.................] - ETA: 1s - loss: 0.5819 - mae: 0.3959 - accuracy: 0.6929
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1367/2969 [============>.................] - ETA: 1s - loss: 0.5817 - mae: 0.3957 - accuracy: 0.6928
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1429/2969 [=============>................] - ETA: 1s - loss: 0.5812 - mae: 0.3956 - accuracy: 0.6931
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1492/2969 [==============>...............] - ETA: 1s - loss: 0.5813 - mae: 0.3956 - accuracy: 0.6932
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1553/2969 [==============>...............] - ETA: 1s - loss: 0.5816 - mae: 0.3957 - accuracy: 0.6929
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1616/2969 [===============>..............] - ETA: 1s - loss: 0.5825 - mae: 0.3962 - accuracy: 0.6925
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1665/2969 [===============>..............] - ETA: 1s - loss: 0.5826 - mae: 0.3962 - accuracy: 0.6925
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1727/2969 [================>.............] - ETA: 1s - loss: 0.5822 - mae: 0.3960 - accuracy: 0.6931
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1788/2969 [=================>............] - ETA: 0s - loss: 0.5827 - mae: 0.3961 - accuracy: 0.6926
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1847/2969 [=================>............] - ETA: 0s - loss: 0.5827 - mae: 0.3961 - accuracy: 0.6929
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1894/2969 [==================>...........] - ETA: 0s - loss: 0.5827 - mae: 0.3961 - accuracy: 0.6930
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 1955/2969 [==================>...........] - ETA: 0s - loss: 0.5826 - mae: 0.3960 - accuracy: 0.6932
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2017/2969 [===================>..........] - ETA: 0s - loss: 0.5826 - mae: 0.3959 - accuracy: 0.6932
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2081/2969 [====================>.........] - ETA: 0s - loss: 0.5824 - mae: 0.3959 - accuracy: 0.6938
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2135/2969 [====================>.........] - ETA: 0s - loss: 0.5826 - mae: 0.3960 - accuracy: 0.6939
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2196/2969 [=====================>........] - ETA: 0s - loss: 0.5827 - mae: 0.3960 - accuracy: 0.6940
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2257/2969 [=====================>........] - ETA: 0s - loss: 0.5823 - mae: 0.3959 - accuracy: 0.6942
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2320/2969 [======================>.......] - ETA: 0s - loss: 0.5825 - mae: 0.3959 - accuracy: 0.6940
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2380/2969 [=======================>......] - ETA: 0s - loss: 0.5827 - mae: 0.3960 - accuracy: 0.6938
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2439/2969 [=======================>......] - ETA: 0s - loss: 0.5826 - mae: 0.3959 - accuracy: 0.6939
    [2m[36m(pid=3225)[0m 2500/2969 [========================>.....] - ETA: 0s - loss: 0.5828 - mae: 0.3960 - accuracy: 0.6938
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2560/2969 [========================>.....] - ETA: 0s - loss: 0.5826 - mae: 0.3959 - accuracy: 0.6941
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2616/2969 [=========================>....] - ETA: 0s - loss: 0.5829 - mae: 0.3960 - accuracy: 0.6940
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2671/2969 [=========================>....] - ETA: 0s - loss: 0.5829 - mae: 0.3961 - accuracy: 0.6939
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2729/2969 [==========================>...] - ETA: 0s - loss: 0.5832 - mae: 0.3962 - accuracy: 0.6936
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2789/2969 [===========================>..] - ETA: 0s - loss: 0.5831 - mae: 0.3962 - accuracy: 0.6935
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2842/2969 [===========================>..] - ETA: 0s - loss: 0.5833 - mae: 0.3963 - accuracy: 0.6933
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2893/2969 [============================>.] - ETA: 0s - loss: 0.5835 - mae: 0.3964 - accuracy: 0.6931
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2948/2969 [============================>.] - ETA: 0s - loss: 0.5834 - mae: 0.3963 - accuracy: 0.6932
    Result for train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:
      checkpoint: weights_tune_[(6, 'disc_params.group_lasso.l1'), (1, 'guess_params.group_lasso.l1'),
        (8, 'slip_params.group_lasso.l1')].h5
      date: 2021-08-05_12-36-48
      done: true
      experiment_id: b87ca9906d2e4494930ffcd41699b968
      hostname: d005995f49c5
      iterations_since_restore: 1
      mean_accuracy: 0.6931789517402649
      mean_error: 0.5833627581596375
      node_ip: 172.28.0.2
      pid: 3225
      time_since_restore: 337.9791102409363
      time_this_iter_s: 337.9791102409363
      time_total_s: 337.9791102409363
      timestamp: 1628167008
      timesteps_since_restore: 0
      training_iteration: 1
      
    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.7/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'TERMINATED': 3, 'PENDING': 1})
    PENDING trials:
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	PENDING
    TERMINATED trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	TERMINATED, [4 CPUs, 0 GPUs], [pid=312], 327 s, 1 iter, 0.693 acc
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	TERMINATED, [4 CPUs, 0 GPUs], [pid=313], 327 s, 1 iter, 0.693 acc
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	TERMINATED, [4 CPUs, 0 GPUs], [pid=3225], 337 s, 1 iter, 0.693 acc
    
    [2m[36m(pid=3225)[0m 
    [2m[36m(pid=3225)[0m 2969/2969 [==============================] - 3s 856us/step - loss: 0.5834 - mae: 0.3963 - accuracy: 0.6932


    2021-08-05 12:36:48,645	ERROR worker.py:1780 -- Warning: The actor WrappedFunc has size 19782331 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object.


    [2m[36m(pid=3230)[0m 2021-08-05 12:36:49.768151: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51,048	ERROR worker.py:1412 -- Calling ray.init() again after it has already been called.
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m config val 1:
    [2m[36m(pid=3230)[0m  key_list: ['disc_params', 'group_lasso'];
    [2m[36m(pid=3230)[0m  values: {'l1': 0.8418982931597918, 'l2': 0}
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m config val 2:
    [2m[36m(pid=3230)[0m  key_list: ['guess_params', 'group_lasso'];
    [2m[36m(pid=3230)[0m  values: {'l1': 9.54924129569551, 'l2': 0}
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m config val 3:
    [2m[36m(pid=3230)[0m  key_list: ['slip_params', 'group_lasso'];
    [2m[36m(pid=3230)[0m  values: {'l1': 4.4228552146004905, 'l2': 0}
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m Intitializing fit for fourPL model. . .
    [2m[36m(pid=3230)[0m Batch_size: 128; epochs: 450;
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.897031: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902552: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902603: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: d005995f49c5
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902613: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: d005995f49c5
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902703: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.32.3
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902731: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.32.3
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902739: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.32.3
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:51.902960: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    [2m[36m(pid=3230)[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:52.186884: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    [2m[36m(pid=3230)[0m 2021-08-05 12:36:52.187210: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m    1/2969 [..............................] - ETA: 40s - loss: 0.6019 - mae: 0.3906 - accuracy: 0.6875
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m   59/2969 [..............................] - ETA: 2s - loss: 0.5886 - mae: 0.4008 - accuracy: 0.6801 
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  118/2969 [>.............................] - ETA: 2s - loss: 0.5812 - mae: 0.3972 - accuracy: 0.6864
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  182/2969 [>.............................] - ETA: 2s - loss: 0.5754 - mae: 0.3943 - accuracy: 0.6932
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  244/2969 [=>............................] - ETA: 2s - loss: 0.5774 - mae: 0.3955 - accuracy: 0.6892
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  303/2969 [==>...........................] - ETA: 2s - loss: 0.5803 - mae: 0.3968 - accuracy: 0.6886
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  361/2969 [==>...........................] - ETA: 2s - loss: 0.5793 - mae: 0.3964 - accuracy: 0.6882
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  422/2969 [===>..........................] - ETA: 2s - loss: 0.5806 - mae: 0.3969 - accuracy: 0.6888
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  484/2969 [===>..........................] - ETA: 2s - loss: 0.5813 - mae: 0.3973 - accuracy: 0.6894
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  548/2969 [====>.........................] - ETA: 2s - loss: 0.5803 - mae: 0.3968 - accuracy: 0.6895
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  603/2969 [=====>........................] - ETA: 1s - loss: 0.5805 - mae: 0.3969 - accuracy: 0.6902
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  660/2969 [=====>........................] - ETA: 1s - loss: 0.5799 - mae: 0.3966 - accuracy: 0.6905
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  718/2969 [======>.......................] - ETA: 1s - loss: 0.5787 - mae: 0.3961 - accuracy: 0.6913
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  776/2969 [======>.......................] - ETA: 1s - loss: 0.5783 - mae: 0.3958 - accuracy: 0.6921
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  823/2969 [=======>......................] - ETA: 1s - loss: 0.5778 - mae: 0.3956 - accuracy: 0.6930
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  886/2969 [=======>......................] - ETA: 1s - loss: 0.5764 - mae: 0.3949 - accuracy: 0.6943
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m  949/2969 [========>.....................] - ETA: 1s - loss: 0.5766 - mae: 0.3950 - accuracy: 0.6940
    [2m[36m(pid=3230)[0m 1011/2969 [=========>....................] - ETA: 1s - loss: 0.5768 - mae: 0.3951 - accuracy: 0.6937
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1069/2969 [=========>....................] - ETA: 1s - loss: 0.5775 - mae: 0.3955 - accuracy: 0.6932
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1122/2969 [==========>...................] - ETA: 1s - loss: 0.5778 - mae: 0.3956 - accuracy: 0.6932
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1182/2969 [==========>...................] - ETA: 1s - loss: 0.5779 - mae: 0.3957 - accuracy: 0.6933
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1241/2969 [===========>..................] - ETA: 1s - loss: 0.5784 - mae: 0.3959 - accuracy: 0.6924
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1296/2969 [============>.................] - ETA: 1s - loss: 0.5784 - mae: 0.3960 - accuracy: 0.6926
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1359/2969 [============>.................] - ETA: 1s - loss: 0.5784 - mae: 0.3960 - accuracy: 0.6924
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1420/2969 [=============>................] - ETA: 1s - loss: 0.5774 - mae: 0.3956 - accuracy: 0.6930
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1480/2969 [=============>................] - ETA: 1s - loss: 0.5776 - mae: 0.3957 - accuracy: 0.6930
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1540/2969 [==============>...............] - ETA: 1s - loss: 0.5777 - mae: 0.3957 - accuracy: 0.6930
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1597/2969 [===============>..............] - ETA: 1s - loss: 0.5786 - mae: 0.3962 - accuracy: 0.6924
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1657/2969 [===============>..............] - ETA: 1s - loss: 0.5790 - mae: 0.3963 - accuracy: 0.6924
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1720/2969 [================>.............] - ETA: 1s - loss: 0.5786 - mae: 0.3961 - accuracy: 0.6928
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1783/2969 [=================>............] - ETA: 1s - loss: 0.5790 - mae: 0.3962 - accuracy: 0.6925
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1835/2969 [=================>............] - ETA: 0s - loss: 0.5791 - mae: 0.3962 - accuracy: 0.6925
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1896/2969 [==================>...........] - ETA: 0s - loss: 0.5790 - mae: 0.3961 - accuracy: 0.6929
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 1959/2969 [==================>...........] - ETA: 0s - loss: 0.5788 - mae: 0.3960 - accuracy: 0.6931
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2010/2969 [===================>..........] - ETA: 0s - loss: 0.5789 - mae: 0.3960 - accuracy: 0.6931
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2063/2969 [===================>..........] - ETA: 0s - loss: 0.5787 - mae: 0.3959 - accuracy: 0.6937
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2123/2969 [====================>.........] - ETA: 0s - loss: 0.5790 - mae: 0.3961 - accuracy: 0.6937
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2184/2969 [=====================>........] - ETA: 0s - loss: 0.5788 - mae: 0.3960 - accuracy: 0.6941
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2246/2969 [=====================>........] - ETA: 0s - loss: 0.5786 - mae: 0.3959 - accuracy: 0.6942
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2304/2969 [======================>.......] - ETA: 0s - loss: 0.5790 - mae: 0.3960 - accuracy: 0.6937
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2356/2969 [======================>.......] - ETA: 0s - loss: 0.5787 - mae: 0.3959 - accuracy: 0.6940
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2414/2969 [=======================>......] - ETA: 0s - loss: 0.5790 - mae: 0.3960 - accuracy: 0.6937
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2473/2969 [=======================>......] - ETA: 0s - loss: 0.5789 - mae: 0.3960 - accuracy: 0.6940
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2531/2969 [========================>.....] - ETA: 0s - loss: 0.5790 - mae: 0.3960 - accuracy: 0.6939
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2580/2969 [=========================>....] - ETA: 0s - loss: 0.5790 - mae: 0.3960 - accuracy: 0.6940
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2636/2969 [=========================>....] - ETA: 0s - loss: 0.5792 - mae: 0.3961 - accuracy: 0.6939
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2690/2969 [==========================>...] - ETA: 0s - loss: 0.5793 - mae: 0.3962 - accuracy: 0.6938
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2747/2969 [==========================>...] - ETA: 0s - loss: 0.5796 - mae: 0.3963 - accuracy: 0.6934
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2796/2969 [===========================>..] - ETA: 0s - loss: 0.5795 - mae: 0.3963 - accuracy: 0.6934
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2847/2969 [===========================>..] - ETA: 0s - loss: 0.5796 - mae: 0.3963 - accuracy: 0.6933
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2898/2969 [============================>.] - ETA: 0s - loss: 0.5798 - mae: 0.3964 - accuracy: 0.6931
    Result for train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:
      checkpoint: weights_tune_[(1, 'disc_params.group_lasso.l1'), (6, 'guess_params.group_lasso.l1'),
        (5, 'slip_params.group_lasso.l1')].h5
      date: 2021-08-05_12-42-24
      done: true
      experiment_id: 8877a1b470414e1680751ff7d7118d16
      hostname: d005995f49c5
      iterations_since_restore: 1
      mean_accuracy: 0.6931789517402649
      mean_error: 0.5796586275100708
      node_ip: 172.28.0.2
      pid: 3230
      time_since_restore: 333.13375973701477
      time_this_iter_s: 333.13375973701477
      time_total_s: 333.13375973701477
      timestamp: 1628167344
      timesteps_since_restore: 0
      training_iteration: 1
      
    [2m[36m(pid=3230)[0m 
    [2m[36m(pid=3230)[0m 2951/2969 [============================>.] - ETA: 0s - loss: 0.5796 - mae: 0.3963 - accuracy: 0.6932
    [2m[36m(pid=3230)[0m 2969/2969 [==============================] - 3s 873us/step - loss: 0.5797 - mae: 0.3963 - accuracy: 0.6932
    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.7/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'TERMINATED': 4})
    TERMINATED trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	TERMINATED, [4 CPUs, 0 GPUs], [pid=312], 327 s, 1 iter, 0.693 acc
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	TERMINATED, [4 CPUs, 0 GPUs], [pid=313], 327 s, 1 iter, 0.693 acc
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	TERMINATED, [4 CPUs, 0 GPUs], [pid=3225], 337 s, 1 iter, 0.693 acc
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	TERMINATED, [4 CPUs, 0 GPUs], [pid=3230], 333 s, 1 iter, 0.693 acc
    
    == Status ==
    Using AsyncHyperBand: num_stopped=0
    Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
    Bracket: Iter 90.000: None | Iter 30.000: None
    Bracket: Iter 90.000: None
    Resources requested: 0/4 CPUs, 0/1 GPUs
    Memory usage on this node: 1.7/13.6 GB
    Result logdir: /root/ray_results/hyperOpt_optimization
    Number of trials: 4 ({'TERMINATED': 4})
    TERMINATED trials:
     - train_model_1_disc_params.group_lasso.l1=2.7829,guess_params.group_lasso.l1=3.9435,slip_params.group_lasso.l1=5.9648:	TERMINATED, [4 CPUs, 0 GPUs], [pid=312], 327 s, 1 iter, 0.693 acc
     - train_model_2_disc_params.group_lasso.l1=6.5627,guess_params.group_lasso.l1=2.9186,slip_params.group_lasso.l1=1.3818:	TERMINATED, [4 CPUs, 0 GPUs], [pid=313], 327 s, 1 iter, 0.693 acc
     - train_model_3_disc_params.group_lasso.l1=6.6343,guess_params.group_lasso.l1=7.145,slip_params.group_lasso.l1=5.1955:	TERMINATED, [4 CPUs, 0 GPUs], [pid=3225], 337 s, 1 iter, 0.693 acc
     - train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229:	TERMINATED, [4 CPUs, 0 GPUs], [pid=3230], 333 s, 1 iter, 0.693 acc
    
    Creating model...
    
    config val 1:
     key_list: ['disc_params', 'group_lasso'];
     values: {'l1': 0.8418982931597918, 'l2': 0}
    
    
    
    config val 2:
     key_list: ['guess_params', 'group_lasso'];
     values: {'l1': 9.54924129569551, 'l2': 0}
    
    
    
    config val 3:
     key_list: ['slip_params', 'group_lasso'];
     values: {'l1': 4.4228552146004905, 'l2': 0}
    
    
    Loading from /root/ray_results/hyperOpt_optimization/train_model_4_disc_params.group_lasso.l1=0.8419,guess_params.group_lasso.l1=9.5492,slip_params.group_lasso.l1=4.4229_2021-08-05_12-36-48_oe4qj2r/weights_tune_[(1, 'disc_params.group_lasso.l1'), (6, 'guess_params.group_lasso.l1'), (5, 'slip_params.group_lasso.l1')].h5
    2969/2969 [==============================] - 16s 976us/step - loss: 0.5790 - mae: 0.3961 - accuracy: 0.6922
    
    Training on : 95000 samples for : 450 epochs has completed in : 1340.374 seconds.
    
    AIC value: 1005.091 and AICc value: 1010.435
    
    Use `object.plot()` to view train/validation loss curves;
    Use `object.history` to obtain train/validation loss across all the epochs.
    Use `object.coefficients()` to obtain model parameters--Question difficulty, discrimination, guessing & slip
    Use `object.AIC` & `object.AIC` to obtain Akaike Information Criterion(AIC & AICc) values.





    <mlsquare.adapters.sklearn.IrtKerasRegressor at 0x7fc100c44c10>



**Following lists the layer names in order from model archiecture of doped model.**

Note:
* trained weights of layer 8 named `slip_param` corresponds to  slip parameter values.
* trained weights of layer 9 named `guessing_param` corresponds to guess parameter values.
* trained weights of layer 1 named `disc_param` corresponds to discrimination parameter values.
* trained weights of layer 4 named `latent_trait/ability` corresponds to user ability values.


```python
for idx, lay in enumerate(model_4t1.model.layers):
    print('{} and layer name: {}'.format(idx, lay.name))
```

    0 and layer name: questions/items
    1 and layer name: disc_param
    2 and layer name: user_id
    3 and layer name: disc_activation
    4 and layer name: latent_trait/ability
    5 and layer name: difficulty_level
    6 and layer name: lambda_latent_inter.
    7 and layer name: alpha_param.
    8 and layer name: slip_param
    9 and layer name: guessing_param
    10 and layer name: alpha_lambda_add
    11 and layer name: slip_param_inter.
    12 and layer name: Sigmoid_func
    13 and layer name: slip/guess_interaction
    14 and layer name: disc/guess_param_inter.
    15 and layer name: guess_param_inter/add
    16 and layer name: prediction_layer


### Obtain User & Item parameters from trained 4PL IRT model
**User parameters (per user):** Ability values.

**Item parameters (per item):** Item difficulty, Item Discrimination, Item guessability, Item Slip




```python
import matplotlib.pyplot as plt
def get_keras_parameters(model, xtrain):# model is a dope object
    ability = model.coefficients()['latent_trait/ability']
    ability = [x[0] for x in ability]

    diff= model.coefficients()['difficulty_level']
    diff = [x[0] for x in diff]
    
    disc= model.coefficients()['disc_param']
    disc = [x[0] for x in disc]
    
    guess= model.coefficients()['guessing_param']
    guess = [x[0] for x in guess]
    
    slip= model.coefficients()['slip_param']
    slip = [x[0] for x in slip]

    question_code = pd.get_dummies(xtrain['question_code']).columns
    df_quest_related_params = pd.DataFrame({'slip_keras':slip, 'guessing_keras':guess, 'discrimination_keras': disc,'difficulty_keras': diff, 'question_code':question_code})

    user_id = pd.get_dummies(xtrain['user_id']).columns
    df_ability = pd.DataFrame({'user_id':user_id, 'ability_keras':ability})
    return df_quest_related_params, df_ability

def scatter_plot(x, y, x_label, y_label):
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    min_x = min(list(x)+list(y)) - 0.5
    max_x = max(list(x)+list(y)) + 0.5
    ax.plot([min_x, max_x], [min_x, max_x])
    
    ax.set_xlim([min_x, max_x])
    ax.set_ylim([min_x, max_x])
    
    ax.set_title('{} vs. {} correlation scatterplot'.format(x_label, y_label))
    ax.scatter(x, y, c="red")
    plt.ylabel(y_label)
    plt.xlabel(x_label)
    plt.show()
    
def hexbin_plot(x, y, x_label, y_label):
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    min_x = min(list(x)+list(y)) - 0.1
    max_x = max(list(x)+list(y)) + 0.1
    ax.plot([min_x, max_x], [min_x, max_x])
    
    ax.set_xlim([min_x, max_x])
    ax.set_ylim([min_x, max_x])
    
    ax.set_title('{} vs. {} correlation scatterplot'.format(x_label, y_label))
    hbin= ax.hexbin(x, y, gridsize=25, mincnt=2)
    cb = fig.colorbar(hbin, ax=ax)
    cb.set_label('occurence_density')
    plt.ylabel(y_label)
    plt.xlabel(x_label)
    plt.show()
```


```python
question_code = pd.get_dummies(xtrain['question_code']).columns
user_id = pd.get_dummies(xtrain['user_id']).columns

quest_related_params_df, ability_df = get_keras_parameters(model_4t1, xtrain)

true_difficulty = {_.question_code: _.difficulty for _ in train_data.itertuples()}
quest_related_params_df['difficulty_true'] = quest_related_params_df['question_code'].apply(lambda x: true_difficulty.get(x))

quest_related_params_df['discrimination_true']= 1#Disc 1; Since Input data was for 1 PL model

quest_related_params_df['guessing_true'] = 0 #Guess 0; Since Input data was for 1 PL model

quest_related_params_df['slip_true']=0 #Slip 0; Since Input data was for 1 PL model

quest_related_params_df.head(3)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>slip_keras</th>
      <th>guessing_keras</th>
      <th>discrimination_keras</th>
      <th>difficulty_keras</th>
      <th>question_code</th>
      <th>difficulty_true</th>
      <th>discrimination_true</th>
      <th>guessing_true</th>
      <th>slip_true</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.000089</td>
      <td>0.000052</td>
      <td>0.999977</td>
      <td>-1.041043</td>
      <td>que_0</td>
      <td>-1.000000</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.000088</td>
      <td>0.000049</td>
      <td>0.999977</td>
      <td>-1.027209</td>
      <td>que_1</td>
      <td>-0.979798</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.000079</td>
      <td>0.000043</td>
      <td>1.000103</td>
      <td>-0.749467</td>
      <td>que_10</td>
      <td>-0.797980</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>




```python
true_ability = {_.user_id: _.ability for _ in train_data.itertuples()}
ability_df['ability_true'] = ability_df['user_id'].apply(lambda x: true_ability.get(x))
ability_df.head(3)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user_id</th>
      <th>ability_keras</th>
      <th>ability_true</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>user_0</td>
      <td>-0.484500</td>
      <td>-0.501354</td>
    </tr>
    <tr>
      <th>1</th>
      <td>user_1</td>
      <td>0.336772</td>
      <td>0.330120</td>
    </tr>
    <tr>
      <th>2</th>
      <td>user_10</td>
      <td>2.554405</td>
      <td>2.763429</td>
    </tr>
  </tbody>
</table>
</div>



_________________________________________
### Plots: Observed parameters vs. True parameters correlation

#### 1. Observed Slip vs. True Slip


```python
scatter_plot(quest_related_params_df['slip_true'], quest_related_params_df['slip_keras'], "True slip", "Observed slip")
```

#### 2. Observed Guess vs. True Guess


```python
scatter_plot(quest_related_params_df['guessing_true'], quest_related_params_df['guessing_keras'], "True guessing", "Observed guessing")
```

#### 3. Observed Discrimination vs. True Discrimination


```python
scatter_plot(quest_related_params_df['discrimination_true'], quest_related_params_df["discrimination_keras"], "True discrimination", "Observed discrimination")

```

#### 4. Observed Difficulty vs. True Difficulty


```python
scatter_plot(quest_related_params_df['difficulty_true'], quest_related_params_df['difficulty_keras'], "True Difficulty", "Observed difficulty")

```

#### 5. Observed Abilities vs. True Abilities


```python
scatter_plot(ability_df['ability_true'], ability_df['ability_keras'], "True Ability", "Observed Ability")

```

#### 6. Observed Response vs. True Response


```python
import math
def fourPL_func(ability, diff, disc, guess, slip):
    res= guess + ((1-slip) + guess)*(1 / (1 + math.exp(-disc*(ability-diff))))
    return res
def get_true_values(quest_related_params_df, ability_df, train_data):
    difficulty_keras_map = {_.question_code: _.difficulty_keras for _ in quest_related_params_df.itertuples()}
    ability_keras_map = {_.user_id: _.ability_keras for _ in ability_df.itertuples()}
    guessing_keras_map = {_.question_code: _.guessing_keras for _ in quest_related_params_df.itertuples()}
    slip_keras_map = {_.question_code: _.slip_keras for _ in quest_related_params_df.itertuples()}
    discrimination_keras_map = {_.question_code: _.discrimination_keras for _ in quest_related_params_df.itertuples()}

    train_data = train_data.drop_duplicates(['question_code', 'user_id'])
    train_data['difficulty_keras'] = train_data['question_code'].apply(lambda x: difficulty_keras_map.get(x))
    train_data['ability_keras'] = train_data['user_id'].apply(lambda x: ability_keras_map.get(x))
    train_data['guessing_keras'] = train_data['question_code'].apply(lambda x: guessing_keras_map.get(x))
    train_data['slip_keras']= train_data['question_code'].apply(lambda x: slip_keras_map.get(x))
    train_data['discrimination_keras'] = train_data['question_code'].apply(lambda x: discrimination_keras_map.get(x))
    train_data['response_keras'] = train_data.apply(lambda x: fourPL_func(x['ability_keras'], x['difficulty_keras'], x['discrimination_keras'], x['guessing_keras'], x['slip_keras']), axis=1)
    return train_data


train_data = get_true_values(quest_related_params_df, ability_df, train_data)
train_data.head(5)

```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>question_code</th>
      <th>user_id</th>
      <th>difficulty</th>
      <th>ability</th>
      <th>response</th>
      <th>correctness</th>
      <th>difficulty_keras</th>
      <th>ability_keras</th>
      <th>guessing_keras</th>
      <th>slip_keras</th>
      <th>discrimination_keras</th>
      <th>response_keras</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>que_99</td>
      <td>user_99</td>
      <td>1.0</td>
      <td>0.559055</td>
      <td>0.391516</td>
      <td>0</td>
      <td>1.011985</td>
      <td>0.401654</td>
      <td>0.000047</td>
      <td>0.000125</td>
      <td>1.000112</td>
      <td>0.351987</td>
    </tr>
    <tr>
      <th>10</th>
      <td>que_99</td>
      <td>user_98</td>
      <td>1.0</td>
      <td>0.164593</td>
      <td>0.302503</td>
      <td>0</td>
      <td>1.011985</td>
      <td>0.181314</td>
      <td>0.000047</td>
      <td>0.000125</td>
      <td>1.000112</td>
      <td>0.303506</td>
    </tr>
    <tr>
      <th>20</th>
      <td>que_99</td>
      <td>user_97</td>
      <td>1.0</td>
      <td>0.566402</td>
      <td>0.393268</td>
      <td>1</td>
      <td>1.011985</td>
      <td>0.517328</td>
      <td>0.000047</td>
      <td>0.000125</td>
      <td>1.000112</td>
      <td>0.378801</td>
    </tr>
    <tr>
      <th>30</th>
      <td>que_99</td>
      <td>user_96</td>
      <td>1.0</td>
      <td>1.186521</td>
      <td>0.546496</td>
      <td>0</td>
      <td>1.011985</td>
      <td>1.059723</td>
      <td>0.000047</td>
      <td>0.000125</td>
      <td>1.000112</td>
      <td>0.511940</td>
    </tr>
    <tr>
      <th>40</th>
      <td>que_99</td>
      <td>user_95</td>
      <td>1.0</td>
      <td>-0.039525</td>
      <td>0.261242</td>
      <td>0</td>
      <td>1.011985</td>
      <td>-0.229570</td>
      <td>0.000047</td>
      <td>0.000125</td>
      <td>1.000112</td>
      <td>0.224170</td>
    </tr>
  </tbody>
</table>
</div>




```python

hexbin_plot(train_data['response'], train_data['response_keras'], "True Response", "Observed Response")

```

__________________________________

94bd58603a7843ab2e425b96fd570870df30b9ca
ipynb
Jupyter Notebook
examples/IRT_model_and_NAS.ipynb
mlsquare/mlsquare
187c3b36a12b564b9dc8bd1320055272844793eb
MIT
2020-04-02T04:25:49.000Z
2020-04-02T04:25:49.000Z
examples/IRT_model_and_NAS.ipynb
mlsquare/mlsquare
187c3b36a12b564b9dc8bd1320055272844793eb
MIT
2019-09-24T01:55:02.000Z
2019-12-02T17:52:17.000Z
examples/IRT_model_and_NAS.ipynb
mlsquare/mlsquare
187c3b36a12b564b9dc8bd1320055272844793eb
MIT
2019-08-01T10:54:00.000Z
2019-11-12T12:09:39.000Z
Qwen/Qwen-72B
1. YES
2. YES