# Main KernelPyLogit Example

The purpose of this Jupyter notebook is to demonstrate the key functionalities of PyKernelLogit by means of a real case study. Take into account that some of the functionalities described here are inherited from the original PyLogit package, especially those related with the estimation of conditional logit models. This example is divided in several parts.
1. Loading and preprocessing the data. This involves converting a dataframe between 'wide' and 'long' format.
2. Estimating a linear Random Utility model using a conditional logit model.
    1. Definition of a conditional logit model.
    2. Estimation of the previous model.
    3. Evaluation of the estimated model.
    4. Prediction using the estimated model.
    5. Computation of the Willingness to Pay (WTP) indicator using the estimated model.
3. Estimating a non-parametric model using Kernel Logistic Regression (KLR).
    1. Definition of a KLR model.
    2. Estimation of the previous KLR model.
    3. Evaluation of the estimated model.
    4. Prediction using the estimated model.
    5. Computation of the Willingness to Pay (WTP) indicator using estimated KLR model.


The dataset selected for this example is the 'Mode' dataset from the 'mlogit' package for R programming language (https://cran.r-project.org/web/packages/mlogit/mlogit.pdf). This dataset was elaborated by a project team in 'Discrete choice methods with simulation' (Train, 2003) and contains data about the mode choice of $453$ commuters. Each commuter has four available travel modes for their trips to work in a choice set $C=\{ \hbox{}bus, car, carpool, rail  \}$. The time and cost of travel for each mode were determined for each commuter. The attributes of the alternative $i \in C$ for the commuter $n$  are $x_{in}=(x_{in,cost}, x_{in,time})$ .


The first step is to load the different packages that will be used in this example.


```python
from collections import OrderedDict    # For recording the model specification 

import pandas as pd                    # For file input/output
import numpy as np                     # For vectorized math operations

import PyKernelLogit as pkl            # For the estimation of the MNL and KLR models.
```

## 1. Loading and preprocessing the data

### 1.1. Load the data
The first step to estimate a model using PyKernelLogit is loading the dataset using a pandas dataframe.


```python
# Load the Mode dataset from a .csv file. Nota that all the elements are delimited by a ',' character.
mode_choice = pd.read_csv("../data/ModeChoice.csv", sep=',')

# Show the first 5 rows of the data
mode_choice.head().T
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>choice</th>
      <td>car</td>
      <td>rail</td>
      <td>car</td>
      <td>car</td>
      <td>car</td>
    </tr>
    <tr>
      <th>cost.car</th>
      <td>1.50701</td>
      <td>6.057</td>
      <td>5.79468</td>
      <td>1.86914</td>
      <td>2.49895</td>
    </tr>
    <tr>
      <th>cost.carpool</th>
      <td>2.33561</td>
      <td>2.89692</td>
      <td>2.13745</td>
      <td>2.57243</td>
      <td>1.72201</td>
    </tr>
    <tr>
      <th>cost.bus</th>
      <td>1.80051</td>
      <td>2.23713</td>
      <td>2.57638</td>
      <td>1.90352</td>
      <td>2.686</td>
    </tr>
    <tr>
      <th>cost.rail</th>
      <td>2.35892</td>
      <td>1.85545</td>
      <td>2.74748</td>
      <td>2.26828</td>
      <td>2.97387</td>
    </tr>
    <tr>
      <th>time.car</th>
      <td>18.5032</td>
      <td>31.3111</td>
      <td>22.5474</td>
      <td>26.0903</td>
      <td>4.69914</td>
    </tr>
    <tr>
      <th>time.carpool</th>
      <td>26.3382</td>
      <td>34.257</td>
      <td>23.2552</td>
      <td>29.896</td>
      <td>12.4141</td>
    </tr>
    <tr>
      <th>time.bus</th>
      <td>20.8678</td>
      <td>67.1819</td>
      <td>63.3091</td>
      <td>19.7527</td>
      <td>43.092</td>
    </tr>
    <tr>
      <th>time.rail</th>
      <td>30.0335</td>
      <td>60.2931</td>
      <td>49.1716</td>
      <td>13.4727</td>
      <td>39.7433</td>
    </tr>
  </tbody>
</table>
</div>



### 1.2. Convert the dataset to long format

In order to estimate models using the PyKernelLogit library it is necessary to convert the data to "long" format.

Long format has 1 row per individual per available alternative, and wide format has 1 row per individual or observation. Long format is useful because it permits one to directly use matrix dot products to calculate the index, $V_{ij} = x_{ij} \beta$, for each individual $\left(i \right)$ for each alternative $\left(j \right)$. In situations where a dataset is provided in wide format (as in the case of the Mode dataset), it will be necessary to convert the data from wide format to long format.

To convert a dataset to long format, it is needed to specify:
<ol>
    <li>The variables or columns that are specific to a given individual, regardless of what alternative is being considered.</li>
    <li>The variables that vary across some or all alternatives, for a given individual (e.g. travel time).</li>
    <li>The availability variables.</li>
    <li>The <u>unique</u> observation id column.</li>
    <li>The choice column.</li>
</ol>

The following cells of code identify perform the necessary conversions from the wide to the long dataframe.



```python
# Look at the columns of the Mode dataset
mode_choice.columns
```




    Index(['choice', 'cost.car', 'cost.carpool', 'cost.bus', 'cost.rail',
           'time.car', 'time.carpool', 'time.bus', 'time.rail'],
          dtype='object')




```python
# Convert choice from categorical variable to numeric one
mode_choice['choice'] = mode_choice['choice'].map({'car' : 1, 'carpool' : 2 , 'bus' : 3, 'rail' : 4})

# Create the list of individual specific variables
ind_variables = [] # There are no indivicual specific variables in this dataset

# Variables that vary across some or all alternatives, for a given individual
alt_varying_variables = {u'cost': dict([(1, 'cost.car'),
                                        (2, 'cost.carpool'),
                                        (3, 'cost.bus'),
                                        (4, 'cost.rail')]),
                         u'time': dict([(1, 'time.car'),
                                        (2, 'time.carpool'),
                                        (3, 'time.bus'),
                                        (4, 'time.rail')])
                        }

### Specify the availability variables
# Note that the keys of the dictionary are the alternative id's.
# The values are the columns denoting the availability for the
# given mode in the dataset. If the dataset does not contains any
# availability columns (such as in the Mode dataset), then the 
# keyword 'available_for_all' can be used to specify that an alternative
# is availables for all individuals.
availability_variables = {1: 'available_for_all',
                          2: 'available_for_all', 
                          3: 'available_for_all',
                          4: 'available_for_all'}

### Determine the columns for: alternative ids, the observation ids and the choice

# The 'custom_alt_id' is the name of a column to be created in the long format data
# It will identify the alternative associated with each row.
custom_alt_id = "mode_id"

# Create a custom id column. Note the +1 ensures the id's start at one.
obs_id_column = "custom_id"
mode_choice[obs_id_column] = np.arange(mode_choice.shape[0], dtype=int) + 1

# Create a variable recording the choice column
choice_column = "choice"
```


```python
# Perform the conversion to long format
long_mode_choice = pkl.convert_wide_to_long(mode_choice, 
                                           ind_variables, 
                                           alt_varying_variables, 
                                           availability_variables, 
                                           obs_id_column, 
                                           choice_column,
                                           new_alt_id_name=custom_alt_id)
```


```python
# Look at the resulting long format dataframe
long_mode_choice.head(10).T
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
      <th>6</th>
      <th>7</th>
      <th>8</th>
      <th>9</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>custom_id</th>
      <td>1.00000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>2.000000</td>
      <td>2.000000</td>
      <td>2.000000</td>
      <td>2.000000</td>
      <td>3.000000</td>
      <td>3.000000</td>
    </tr>
    <tr>
      <th>mode_id</th>
      <td>1.00000</td>
      <td>2.000000</td>
      <td>3.000000</td>
      <td>4.000000</td>
      <td>1.000000</td>
      <td>2.000000</td>
      <td>3.000000</td>
      <td>4.000000</td>
      <td>1.000000</td>
      <td>2.000000</td>
    </tr>
    <tr>
      <th>choice</th>
      <td>1.00000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>cost</th>
      <td>1.50701</td>
      <td>2.335612</td>
      <td>1.800512</td>
      <td>2.358920</td>
      <td>6.056998</td>
      <td>2.896919</td>
      <td>2.237128</td>
      <td>1.855451</td>
      <td>5.794677</td>
      <td>2.137454</td>
    </tr>
    <tr>
      <th>time</th>
      <td>18.50320</td>
      <td>26.338233</td>
      <td>20.867794</td>
      <td>30.033469</td>
      <td>31.311107</td>
      <td>34.256956</td>
      <td>67.181889</td>
      <td>60.293126</td>
      <td>22.547429</td>
      <td>23.255171</td>
    </tr>
  </tbody>
</table>
</div>



### 1.3. Divide the final dataset into a train and test subsets

In machine learning it is a typical practise to divide the data into two sets: the training set and the test set. As these names imply, the model is trained using the training set, and then it is tested using the test set. This is commonly used to obtain the accuracy of the classifier on the new cases (the observations in the test set), which is called generalization error. This value indicates how well the model performs on instances it has never seen before. PyKernelLogit allows to divide a long format dataframe into a train and test subsets. 


```python
# Define the percentaje of total observations which are used for training. 
train_size = 0.7

(train_set, test_set) = pkl.divide_long_train_test(long_mode_choice, train_size, obs_id_column)
```

## 2. Estimating a linear Random Utility model using a conditional logit model

Firstly, a linear Random Utility model using a conditional logit model will be estimated using the Mode dataset.

### 2.1. Definition of a conditional logit model

Before estimating a model, it is necessary to pre-compute all of the variables that are going to be used in the model. This is different from the functionalities of other package such as mlogit or statsmodels that use formula strings to create new variables. This is also somewhat different from Python Biogeme where new variables can be defined in the script but not actually created by the user before model estimation. PyKernelLogit (as it contains the same internal structure as PyLogit) does not perform variable creation. It only estimates models using variables that already exist.

For the Mode dataset no variable preprocessing was necessary. Moreover, it was decided not to scale any of the variables. No new variables have been defined either.

The first step is to create the model specification being used in this example:

$$
\begin{aligned}
V_{\textrm{Car}, n} &= \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Car}, n} + \\ 
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Car}, n} \\
        V_{\textrm{Carpool}, n} &= \textrm{ASC Carpool} + \\ 
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Carpool}, n} + \\ 
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Carpool}, n} \\
        V_{\textrm{Bus}, n} &= \textrm{ASC Bus} + \\ 
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Bus}, n} + \\ 
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Bus}, n} \\
        V_{\textrm{Rail}, n} &= \textrm{ASC Rail} + \\ 
        & \quad \beta_{\textrm{travel cost}} \cdot \textrm{Cost}_{\textrm{Rail}, n} + \\ 
        & \quad \beta_{\textrm{travel time}} \cdot \textrm{Time}_{\textrm{Rail}, n}
\end{aligned}
$$

Note that packages such as mlogit and statsmodels do not, by default, handle coefficients that vary over some alternatives but not all. PyKernelLogit does have this capability.

In order to specify a linear model in PyKernelLogit it is necessary to create two ordered dictionaries:

* Specification ordered dictionary. It contains the desired PyKernelLogit model specification. The dictionary keys should be variables within the long format dataframe (i.e. the column names from the dataframe), whereas, their associate value should be a list of integers, where the integers are the alternative ID's of the alternative whose utility specification the explanatory variable is entering. In other words, for each integer $i$ in the list, a $\beta_{ik}$ parameter is created, which multiplies the attribute $x_{ink}$ selected on the ordered dictionary key. It is also possible to use the same $\beta_{k}$ parameter for more than one alternative. In this case, a nested list should be added containing the alternative ID's of the alternatives that will share the common parameter.
    
* Variable names ordered dictionary. This order dictionary should contain the same keys that the Specification ordered dictionary. The value should be a list with the names for the $\beta_{k}$ parameters that will be estimated for each of the elements within the outer-most list on the Specification ordered dictionary.


The keys of both ordered dictionaries should be variables from the long format dataframe with the sole exception of the intercept key. The value associated with this key is a list with the alternative ID's for which an intercept will be added in the model specification.


```python
# Create an empty basic_specification and basic_names ordered dictionary
basic_specification = OrderedDict()
basic_names = OrderedDict()

# Model specification
basic_specification["intercept"] = [2, 3, 4]
basic_names["intercept"] = ['ASC CarPool',
                            'ASC Bus',
                            'ASC Rail']

basic_specification["cost"] = [[1, 2, 3, 4]]
basic_names["cost"] = ['Travel cost (dollars)']

basic_specification["time"] = [[1, 2, 3, 4]]
basic_names["time"] = ['Travel time (minutes)']
```

### 2.2. Estimation of the previous model

Once the model has been specified, it is necessary to estimate its parameters. PyKernelLogit allows to estimate the previous model using Maximum Likelihood Estimation (MLE) or Penalized Maximum Likelihood Estimation (PMLE). In this example, as the model is not complex, MLE is used.


```python
# Estimate the multinomial logit model (MNL)
linear_model = pkl.create_choice_model(data=train_set,
                                           alt_id_col=custom_alt_id,
                                           obs_id_col=obs_id_column,
                                           choice_col=choice_column,
                                           specification=basic_specification,
                                           model_type="MNL",
                                           names=basic_names)
```

The previous model can be estimated using the fit_mle() method. This method is designed to estimate Multinomial Logit Models and receives several parameters: 
* init_vals. The initial values of $\boldsymbol{\beta}$ (RUM) or $\boldsymbol{\alpha}$ (KLR) to start the optimization process with. It should be a vector with one value for each utility coefficient being estimated.

* method. It should be a valid string that can be passed to scipy.optimize.minimize. Determines the optimization algorithm that is used for this problem. The default newton-cg which uses the Newton-CG algorithm.

* PMLE. It determines if a Penalized Maximum Likelihood Estimation should be executed. The value of the parameter determines the type of PMLE and the possible values are LASSO and RIDGE.

* PMLE_lambda. Lambda parameter for LASSO or ridge regression. It should be an int, float or long and determines the penalty parameter for the PMLE.




```python
# Fit the linear model using MLE
linear_model.fit_mle(np.zeros(5), method='BFGS')
```

    Log-likelihood at zero: -439.4553
    Initial Log-likelihood: -439.4553
    Estimation Time for Point Estimation: 0.01 seconds.
    Final log-likelihood: -249.6988


### 2.3. Evaluation of the estimated model

Once the model has been estimated, the next step is to asses the performance of the model.

In machine learning, there are several alternatives for assessing whether a model is good enough for the specific application for which the model is going to be used. The goodness of fit of a model is a statistic which descries how well a model, with its estimated parameters, performs compared with a model in which all its parameters are zero (which is usually equivalent to having no model at all, i.e. in the discrete choice context is equivalent to a model for which the probability of choosing an alternative is the same for all the alternatives). This comparison is made using the log-likelihood function, which is evaluated at both models, one with estimated parameters and other with all its parameters equal zero. The likelihood ratio index (also called McFadden R square) is defined as:
\begin{equation}
    \label{eq:ro-square}
    \rho^2 = 1-\frac{LL(\hat{\boldsymbol{\beta}})}{LL(\boldsymbol{0})}.
\end{equation}

This measure has been derived to mimic the $R^2$ used in linear regression, but in this case its interpretation is not similar at all. If the estimated model has the same log-likelihood as an equal probability model, then $\rho^2 = 0$. On the contrary, if the estimated model perfectly fits the data ($LL(\hat{\boldsymbol{\beta}}) = 0$), then $\rho^2 = 1$. This value, unlike $R^2$, cannot be interpreted, and it must only be used to compare between two models estimated using the same sample. For this reason, two models estimated on different samples cannot be compared via their likelihood ratio index values.

It presents another disadvantage because this measure is monotonic in the number of parameters of the model, which means that this index increase each time an additional variable is added to the model, even if it does not explain anything. There is a variant of the $\rho^2$ index which offer a corrected measure:
\begin{equation}
    \label{eq:pseudo-ro-bar-square}
    \overline{\rho}^{2} = 1-\frac{LL(\hat{\boldsymbol{\beta}})-\nu}{LL(0)},
\end{equation}
where $\nu$ is the number of parameters in the model.

When a model is estimated, PyKernelLogit allows to obtain a summary of the results by using the function get_statsmodels_summary. This function integrates both, $\rho^2$ and $\overline{\rho}^{2}$ goodness of fit measures, which are called here as Pseudo R-squ. and Pseudo R-bar-squ., respectively.


```python
# Obtain a summary of the results for the linear model
linear_model.get_statsmodels_summary()
```




<table class="simpletable">
<caption>Multinomial Logit Model Regression Results</caption>
<tr>
  <th>Dep. Variable:</th>         <td>choice</td>          <th>  No. Observations:  </th>    <td>317</td>  
</tr>
<tr>
  <th>Model:</th>         <td>Multinomial Logit Model</td> <th>  Df Residuals:      </th>    <td>312</td>  
</tr>
<tr>
  <th>Method:</th>                  <td>MLE</td>           <th>  Df Model:          </th>     <td>5</td>   
</tr>
<tr>
  <th>Date:</th>             <td>Fri, 28 Feb 2020</td>     <th>  Pseudo R-squ.:     </th>   <td>0.432</td> 
</tr>
<tr>
  <th>Time:</th>                 <td>14:15:35</td>         <th>  Pseudo R-bar-squ.: </th>   <td>0.420</td> 
</tr>
<tr>
  <th>AIC:</th>                   <td>509.398</td>         <th>  Log-Likelihood:    </th> <td>-249.699</td>
</tr>
<tr>
  <th>BIC:</th>                   <td>528.192</td>         <th>  LL-Null:           </th> <td>-439.455</td>
</tr>
</table>
<table class="simpletable">
<tr>
            <td></td>               <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  
</tr>
<tr>
  <th>ASC CarPool</th>           <td>   -4.1890</td> <td>    0.470</td> <td>   -8.915</td> <td> 0.000</td> <td>   -5.110</td> <td>   -3.268</td>
</tr>
<tr>
  <th>ASC Bus</th>               <td>   -3.4403</td> <td>    0.390</td> <td>   -8.819</td> <td> 0.000</td> <td>   -4.205</td> <td>   -2.676</td>
</tr>
<tr>
  <th>ASC Rail</th>              <td>   -2.7398</td> <td>    0.354</td> <td>   -7.745</td> <td> 0.000</td> <td>   -3.433</td> <td>   -2.046</td>
</tr>
<tr>
  <th>Travel cost (dollars)</th> <td>   -0.7903</td> <td>    0.112</td> <td>   -7.077</td> <td> 0.000</td> <td>   -1.009</td> <td>   -0.571</td>
</tr>
<tr>
  <th>Travel time (minutes)</th> <td>   -0.0829</td> <td>    0.009</td> <td>   -9.051</td> <td> 0.000</td> <td>   -0.101</td> <td>   -0.065</td>
</tr>
</table>



These goodness of fit measures cannot measure how the model predict on new data points, for which it has not yet seen the true value of the dependent variable (the alternative chosen). Therefore, some other model validation techniques are available on the PyKernelLogit package.

A much better way to evaluate the performance of a classifier is to look at the so-called confusion matrix. The idea is to count the number of instances of class A which are classified as class B. The confusion matrix can be computed as follows.


```python
# Compute the confunsion matrix using the test set
confusion_matrix = linear_model.confusion_matrix(test_set)

# The generated confusion matrix can be shown directly as a pandas dataframe
confusion_matrix
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>56</td>
      <td>3</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>0</td>
      <td>0</td>
      <td>4</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>0</td>
      <td>9</td>
      <td>9</td>
    </tr>
    <tr>
      <th>4</th>
      <td>11</td>
      <td>0</td>
      <td>5</td>
      <td>23</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Another option is to use the heatmap method of the seaborn package to 
# visualize the confusion matrix. This generates more elegant ilustrations.

%matplotlib inline

# Import seaborn and matplotlib
import seaborn as sns
import matplotlib.pyplot as plt     

# Generate a new figure and set the figure size.
plt.figure(figsize=(10,5))

# Generate a heatmap for the confusion matrix using seaborn
ax = sns.heatmap(confusion_matrix, annot=True, linewidth=0.5, cmap='PuBu')

# labels, title and ticks
ax.set_xlabel('Predicted alternative')
ax.set_ylabel('True alternative')
ax.set_title('Confusion Matrix')

# Figures can be stored as images by means of ax.figure.savefig() method.
```

As it is shown, the confusion matrix gives lot of useful information, but sometimes it is preferred a more concise metric. An other interesting way of assessing the performance of a model is to look the accuracy of the positive predictions, which is called the precision. 

This metric has an important drawback because if the model makes a single positive prediction and it is correct, then the precision will be 1/1 = 100\%. This results would not be useful since the classifier would ignore all the predictions except for the only one positive instance. For that reason, the precision metric is commonly used along with another metric called recall, or sensitivity. The recall is the ratio of positive instances that are correctly detected by the classifier. 

Finally, there is a third metric called F-score, which combines the precision and the recall results to compute the score. The F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0. The $\beta$ parameter determines the weight of recall in the combined score. When $\beta < 1$, the F-score lends more weight to precision, while $\beta > 1$ favors recall. Moreover, $\beta \rightarrow 0$ considers only the precision and $\beta \rightarrow \infty$ only the recall.

This metrics can be applied directly to problems where only two alternatives are considered. For problems with more than two alternatives an averaging technique should be applied. PyKernelLogit offers two averaging techniques:
* Macro-averaging.
* Micro-averaging. 

The precision, recall and F-score metrics can be computed for the previous model using the following command.


```python
# Compute the precision, recall and F1-score for the linear model using a
# Micro-averaging technique. The F1-score is the F-score for beta = 1.
linear_model.precision_recall_fscore(test_set, "micro", beta = 1)
```




    (0.6470588235294118, 0.6470588235294118, 0.6470588235294118)



### 2.4. Prediction using the estimated model

Once the model has been estimated, new instances can be predicted using the model. The next block of code shows how to create a pandas dataframe which contains a matrix with the probabilities predicted for each alternative for a given observation.


```python
# The different observations from the test_set are represented on each row.
# The columns represent each of the alternatives. The value from each cell
# is the probability that the alternative on the corresponding column was
# selected on that observation.
pd.DataFrame(data=linear_model.predict(test_set).reshape(-1, 4))
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.963694</td>
      <td>0.003965</td>
      <td>0.020135</td>
      <td>0.012206</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.838091</td>
      <td>0.005317</td>
      <td>0.044212</td>
      <td>0.112380</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.981976</td>
      <td>0.014513</td>
      <td>0.001126</td>
      <td>0.002385</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.590096</td>
      <td>0.086270</td>
      <td>0.042587</td>
      <td>0.281048</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.148308</td>
      <td>0.019157</td>
      <td>0.082536</td>
      <td>0.749999</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>131</th>
      <td>0.216609</td>
      <td>0.103513</td>
      <td>0.087728</td>
      <td>0.592150</td>
    </tr>
    <tr>
      <th>132</th>
      <td>0.777684</td>
      <td>0.017586</td>
      <td>0.056032</td>
      <td>0.148698</td>
    </tr>
    <tr>
      <th>133</th>
      <td>0.781351</td>
      <td>0.117042</td>
      <td>0.066393</td>
      <td>0.035214</td>
    </tr>
    <tr>
      <th>134</th>
      <td>0.767810</td>
      <td>0.031292</td>
      <td>0.009875</td>
      <td>0.191023</td>
    </tr>
    <tr>
      <th>135</th>
      <td>0.880017</td>
      <td>0.053360</td>
      <td>0.050753</td>
      <td>0.015870</td>
    </tr>
  </tbody>
</table>
<p>136 rows × 4 columns</p>
</div>



### 2.5. Computation of the Willingness to Pay (WTP) indicator using the estimated model

There are many indicators that are particularly relevant in the context of discrete choice models, such as, Willingness to Pay (WTP) and the Value of Time (VOT). If the estimated model contains a cost or price variable, which indicates the total cost of the alternative for a given decision-maker, it is possible to analyze the trade-off between any other variable form the model and money. This trade-off reflects the **willingness of the decision-maker to pay** for a modification of one unit of another variable of the model.

Lets consider that $c_{in}$ is the cost of the alternative $i$ for a decision-maker $n$, and $x_{in}$ is the value of another variable. Now, let $V_{in}(c_{in}, x_{in})$ be the value of the utility function that is associated to an alternative $i$ for a decision-maker $n$. It can be considered a scenario where the variable under interest takes the value $x_{in} + \delta^{x}_{in}$, where $\delta^{x}_{in}$ is an increment over the value of the $x$ variable. It is denoted by $\delta^{c}_{in}$ the additional cost increment that would be necessary to achieve the same utility as if no increment were made over $x$ variable, that is
\begin{equation}
    \label{eq:WTP-same-utility}
    V_{i n}\left(c_{i n}+\delta_{i n}^{c}, x_{i n}+\delta_{i n}^{x}\right)=V_{i n}\left(c_{i n}, x_{i n}\right).
\end{equation}

If $x_{in}$ and $c_{in}$ appear linearly in the utility function, that is if 
$V_{i n}\left(c_{i n}, x_{i n}\right)=\beta_{c} c_{i n}+\beta_{x} x_{i n}+\cdots$
Then, the WTP can be calculated directly using the coefficients ratio as:
\begin{equation}
    WTP = \frac{\delta_{i n}^{c}}{\delta_{i n}^{x}}=-\frac{\left(\partial V_{i n} / \partial x_{i n}\right)\left(c_{i n}, x_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, x_{i n}\right)}=-\frac{\beta_{x}}{\beta_{c}}
\end{equation}
where $\beta_{c}$ is the calculated coefficient for the cost.

The above equation is the willingness to pay for an increase of the value of
the variable $x_{in}$. Therefore, if this increase improves the utility of the alternative, then $\beta_{x}>0$.


Another concept is the **Value of Time (VOT)**, which is closely related with the previous one, as it reflects the price that a traveler is willing to pay to decrease the travel time in one unit. Hence, the formula for the VOT is:
\begin{equation}
    \mathrm{VOT}_{i n}=\delta_{i n}^{c} /\left(-\delta_{i n}^{t}\right)=\frac{\left(\partial V_{i n} / \partial t_{i n}\right)\left(c_{i n}, t_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, t_{i n}\right)}
\end{equation}

As $V$ is linear in the cost and time parameters, we have
\begin{equation}
    \mathrm{VOT}_{i n}=\delta_{i n}^{c} /\left(-\delta_{i n}^{t}\right)=\frac{\beta_{t}}{\beta_{c}}
\end{equation}


In this example, it is interesting to compute the **VOT** as the model contains the cost and time variables.


```python
# To compute the VOT for the linear model, we need to apply the previous equation.
# Take into account that travel time is in minutes, therefore beta_t = 1/minutes,
# which is equal as beta_t = 60 * 1/hours. Now, VOT is measured in dollars/hours.

(linear_model.coefs['Travel time (minutes)']*60) / linear_model.coefs['Travel cost (dollars)']
```




    6.293743644982097



## 3. Estimating a non-parametric model using Kernel Logistic Regression (KLR)

Now, a non-parametric model is estimated using KLR capabilities from PyKernelLogit.

### 3.1. Definition of a KLR model

The first step is to consider if the data should be preprocessed. It is a good practice to scale the data before a KLR model is applied. Skcikit-learn Python package can be used to scale the data.


```python
# Scale the data
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

scaler.fit(train_set[['cost', 'time']])

train_set_scaled = train_set.copy()
train_set_scaled[['cost', 'time']] = scaler.transform(train_set[['cost', 'time']])

test_set_scaled = test_set.copy()
test_set_scaled[['cost', 'time']] = scaler.transform(test_set[['cost', 'time']])
```

The following step to define a KLR model is to compute the kernel matrix, or Gram matrix, using the input dataframe. The kernel_matrix function returns a list composed of one kernel matrix $\mathbf{K}^{(i)}$ per alternative $i$. This function takes the following input values:
* X. The dataset in long format for which the kernel matrix is computed, i.e. $\mathbf{X}_{i}$. 

* alt_id_col. The name for the alternative ID column in the long dataframe.

* obs_id_col. The column from the wide dataset that contains the ID of each observation.

* variables. A Python dictionary where each key represents each of the alternatives IDs and its associated value contains a list of characteristics from the long dataframe X to be used on that alternative.

* kernel_type. The type of kernel function to be used. The kernel_matrix function relies on the scikit-learn Python package for the kernel functions. The value passed on the kernel_type parameter specifies the scikit-learn kernel function which is going to be used to compute the kernel matrix.

* Z. The kernel_matrix function also allows the user to specify a reference matrix $\mathbf{Z}_{i}$ to be used when computing the kernel matrix. If no Z dataframe is specified, then the kernel matrix is computed taking as reference matrix the input matrix X.

* \*\*kernel\_kwargs. Keyworded variable length of arguments to be provided to the invoked scikit-learn kernel function.


```python
variables = {1: ['cost', 'time'],
             2: ['cost', 'time'],
             3: ['cost', 'time'],
             4: ['cost', 'time']}



K_long_format_train = pkl.long_format_with_kernel_matrix(train_set_scaled, 
                                                         custom_alt_id, 
                                                         obs_id_column, 
                                                         choice_column, 
                                                         variables, 
                                                         kernel_type = "RBF")
```

The next step is to define the KLR model specification. Similarly to the linear model specification, it is necessary to create two ordered dictionaries:
* Specification ordered dictionary.
* Variable names ordered dictionary.

The PyKernelLogit function define_kernel_specification generates automatically the specification and variable names ordered dictionaries for the desired kernel model using the specification attributes provided through its input parameters:
* K_long_format. A long format dataframe that contains the kernel matrix for all the alternatives in the dataset. This dataframe should be obtained using the long_format_with_kernel_matrix function.

* alt_id_col. The name for the alternative ID column in the long dataframe.

* obs_id_col. The column from the wide dataset that contains the ID of each observation.

* specification. A previously defined specification ordered dictionary where the kernel logistic model specification will be inserted. It is recommended to pass an empty ordered dictionary.

* names. A previously defined variable names ordered dictionary where it will be inserted the names for the $\boldsymbol{\alpha}_i$ parameters to be estimated. It is recommended to pass an empty ordered dictionary.

* alpha_per_alternative. A boolean variable. If this this variable is True then one vector of parameters to estimate $\boldsymbol{\alpha}_i$ is used per alternative $i$. Otherwise, the same vector of parameters $\boldsymbol{\alpha}$ is used for all the alternatives. This generates a more restricted model, but this model will have much less parameters to be estimated. The default value is False.

* intercept. If this parameter is None (the default value), then no intercept parameter is added to the model. If this parameter contains an integer value, then an intercept parameter is added to the model. The intercept parameter must be the alternative ID of the alternative selected to be the reference alternative. No intercept is generated for the reference alternative because it is considered to be zero.

This method returns a tuple with 3 elements: the basic_specification and basic_names ordered dictionaries and the total number of variables to be estimated in the model.

The following code fragment is used to define the basic_specification and basic_names ordered dictionaries for a KLR model with the same $\boldsymbol{\alpha}$ vector of parameters for all the alternatives and an intercept parameter.



```python
[basic_specification, basic_names, total_vars] = pkl.define_kernel_specification(K_long_format_train, 
                                                                                 custom_alt_id,
                                                                                 obs_id_column,
                                                                                 specification=OrderedDict(),
                                                                                 names=OrderedDict(),
                                                                                 alpha_per_alternative=False,
                                                                                 intercept=1)
```

### 3.2. Estimation of the previous KLR model

Once the KLR model has been specified, it is necessary to estimate its parameters in a similar way that the previous linear model.


```python
# Estimate the multinomial logit model (MNL)
kernel_model = pkl.create_choice_model(data=K_long_format_train,
                                       alt_id_col=custom_alt_id,
                                       obs_id_col=obs_id_column,
                                       choice_col=choice_column,
                                       specification=basic_specification,
                                       model_type="MNL",
                                       names=basic_names)
```

The previous model can be also estimated using the fit_mle() method. Now, instead of using Maximum Likelihood Estimation (MLE), it was decided to use Penalized Maximum Likelihood Estimation (PMLE) due to the high number of parameters. It was decided to use a Ridge regression with a penalty parameter $\lambda$ = 4. Moreover,  L-BFGS-B optimization method was used to estimate the parameters because, although it obtains slightly worse results than BFGS method, the parameters are estimated remarkably fast as it is designed to optimize a large number of parameters using a limited amount of memory.


```python
# Specify the initial values and method for the optimization.
kernel_model.fit_mle(np.zeros(total_vars), method="L-BFGS-B", PMLE="RIDGE", PMLE_lambda=4)
```

    Log-likelihood at zero: -439.4553
    Initial Log-likelihood: -439.4553
    Estimation Time for Point Estimation: 0.03 seconds.
    Final log-likelihood: -277.9502


    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/conditional_logit.py:376: UserWarning: NOTE: A penalized regression is being performed. The reported standard errors and robust standard errors ***WILL BE INCORRECT***.
      warnings.warn(_ridge_warning_msg)
    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/base_multinomial_cm_v2.py:1231: RuntimeWarning: invalid value encountered in sqrt
      self._store_inferential_results(np.sqrt(np.diag(self.cov)),
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in greater
      return (a < x) & (x < b)
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:903: RuntimeWarning: invalid value encountered in less
      return (a < x) & (x < b)
    /home/joseangel/.local/lib/python3.6/site-packages/scipy/stats/_distn_infrastructure.py:1912: RuntimeWarning: invalid value encountered in less_equal
      cond2 = cond0 & (x <= _a)
    /home/joseangel/.local/lib/python3.6/site-packages/PyKernelLogit/base_multinomial_cm_v2.py:1262: RuntimeWarning: invalid value encountered in sqrt
      self._store_inferential_results(np.sqrt(np.diag(self.robust_cov)),


### 3.3. Evaluation of the estimated model

Once the model has been estimated, the performance of the model can be assessed such as in the linear model. Therefore, get_statsmodels_summary() can be used to obtain the goodness of fit of the model and the estimated parameters.


```python
# Look at the estimation results
kernel_model.get_statsmodels_summary()
```




<table class="simpletable">
<caption>Multinomial Logit Model Regression Results</caption>
<tr>
  <th>Dep. Variable:</th>         <td>choice</td>          <th>  No. Observations:  </th>    <td>317</td>  
</tr>
<tr>
  <th>Model:</th>         <td>Multinomial Logit Model</td> <th>  Df Residuals:      </th>    <td>-3</td>   
</tr>
<tr>
  <th>Method:</th>                  <td>MLE</td>           <th>  Df Model:          </th>    <td>320</td>  
</tr>
<tr>
  <th>Date:</th>             <td>Fri, 28 Feb 2020</td>     <th>  Pseudo R-squ.:     </th>   <td>0.368</td> 
</tr>
<tr>
  <th>Time:</th>                 <td>14:15:37</td>         <th>  Pseudo R-bar-squ.: </th>  <td>-0.361</td> 
</tr>
<tr>
  <th>AIC:</th>                  <td>1,195.900</td>        <th>  Log-Likelihood:    </th> <td>-277.950</td>
</tr>
<tr>
  <th>BIC:</th>                  <td>2,398.749</td>        <th>  LL-Null:           </th> <td>-439.455</td>
</tr>
</table>
<table class="simpletable">
<tr>
      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  
</tr>
<tr>
  <th>ASC 2</th>     <td>   -0.7932</td> <td>  843.246</td> <td>   -0.001</td> <td> 0.999</td> <td>-1653.526</td> <td> 1651.939</td>
</tr>
<tr>
  <th>ASC 3</th>     <td>   -0.5352</td> <td>  925.261</td> <td>   -0.001</td> <td> 1.000</td> <td>-1814.013</td> <td> 1812.942</td>
</tr>
<tr>
  <th>ASC 4</th>     <td>   -0.0256</td> <td>  827.401</td> <td> -3.1e-05</td> <td> 1.000</td> <td>-1621.702</td> <td> 1621.651</td>
</tr>
<tr>
  <th>alpha_1</th>   <td>   -0.1119</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_2</th>   <td>   -0.0724</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_3</th>   <td>    0.1261</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_4</th>   <td>    0.0074</td> <td> 1.33e+06</td> <td> 5.59e-09</td> <td> 1.000</td> <td>-2.61e+06</td> <td> 2.61e+06</td>
</tr>
<tr>
  <th>alpha_5</th>   <td>    0.0525</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_6</th>   <td>    0.0348</td> <td>  9.4e+06</td> <td>  3.7e-09</td> <td> 1.000</td> <td>-1.84e+07</td> <td> 1.84e+07</td>
</tr>
<tr>
  <th>alpha_7</th>   <td>   -0.0053</td> <td> 7.93e+06</td> <td>-6.67e-10</td> <td> 1.000</td> <td>-1.55e+07</td> <td> 1.55e+07</td>
</tr>
<tr>
  <th>alpha_8</th>   <td>   -0.1580</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_9</th>   <td>   -0.0290</td> <td>  3.9e+06</td> <td>-7.44e-09</td> <td> 1.000</td> <td>-7.64e+06</td> <td> 7.64e+06</td>
</tr>
<tr>
  <th>alpha_10</th>  <td>   -0.2071</td> <td> 5.64e+06</td> <td>-3.67e-08</td> <td> 1.000</td> <td>-1.11e+07</td> <td> 1.11e+07</td>
</tr>
<tr>
  <th>alpha_11</th>  <td>    0.0867</td> <td> 5.19e+06</td> <td> 1.67e-08</td> <td> 1.000</td> <td>-1.02e+07</td> <td> 1.02e+07</td>
</tr>
<tr>
  <th>alpha_12</th>  <td>    0.1213</td> <td> 7.52e+05</td> <td> 1.61e-07</td> <td> 1.000</td> <td>-1.47e+06</td> <td> 1.47e+06</td>
</tr>
<tr>
  <th>alpha_13</th>  <td>    0.0644</td> <td> 8.12e+06</td> <td> 7.94e-09</td> <td> 1.000</td> <td>-1.59e+07</td> <td> 1.59e+07</td>
</tr>
<tr>
  <th>alpha_14</th>  <td>    0.2021</td> <td> 9.17e+05</td> <td>  2.2e-07</td> <td> 1.000</td> <td> -1.8e+06</td> <td>  1.8e+06</td>
</tr>
<tr>
  <th>alpha_15</th>  <td>    0.0202</td> <td> 9.52e+06</td> <td> 2.12e-09</td> <td> 1.000</td> <td>-1.87e+07</td> <td> 1.87e+07</td>
</tr>
<tr>
  <th>alpha_16</th>  <td>   -0.0955</td> <td> 9.07e+06</td> <td>-1.05e-08</td> <td> 1.000</td> <td>-1.78e+07</td> <td> 1.78e+07</td>
</tr>
<tr>
  <th>alpha_17</th>  <td>    0.0715</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_18</th>  <td>   -0.1768</td> <td> 6.45e+06</td> <td>-2.74e-08</td> <td> 1.000</td> <td>-1.26e+07</td> <td> 1.26e+07</td>
</tr>
<tr>
  <th>alpha_19</th>  <td>   -0.0726</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_20</th>  <td>   -0.1114</td> <td> 3.35e+06</td> <td>-3.33e-08</td> <td> 1.000</td> <td>-6.56e+06</td> <td> 6.56e+06</td>
</tr>
<tr>
  <th>alpha_21</th>  <td>    0.0443</td> <td> 7.08e+06</td> <td> 6.25e-09</td> <td> 1.000</td> <td>-1.39e+07</td> <td> 1.39e+07</td>
</tr>
<tr>
  <th>alpha_22</th>  <td>   -0.0446</td> <td>  1.6e+06</td> <td>-2.79e-08</td> <td> 1.000</td> <td>-3.14e+06</td> <td> 3.14e+06</td>
</tr>
<tr>
  <th>alpha_23</th>  <td>   -0.0163</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_24</th>  <td>    0.0758</td> <td> 3.61e+06</td> <td>  2.1e-08</td> <td> 1.000</td> <td>-7.07e+06</td> <td> 7.07e+06</td>
</tr>
<tr>
  <th>alpha_25</th>  <td>    0.0586</td> <td> 3.68e+06</td> <td> 1.59e-08</td> <td> 1.000</td> <td>-7.22e+06</td> <td> 7.22e+06</td>
</tr>
<tr>
  <th>alpha_26</th>  <td>   -0.0488</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_27</th>  <td>   -0.1262</td> <td> 4.75e+06</td> <td>-2.66e-08</td> <td> 1.000</td> <td>-9.31e+06</td> <td> 9.31e+06</td>
</tr>
<tr>
  <th>alpha_28</th>  <td>    0.0352</td> <td> 4.83e+06</td> <td> 7.29e-09</td> <td> 1.000</td> <td>-9.47e+06</td> <td> 9.47e+06</td>
</tr>
<tr>
  <th>alpha_29</th>  <td>   -0.0126</td> <td> 7.15e+06</td> <td>-1.76e-09</td> <td> 1.000</td> <td> -1.4e+07</td> <td>  1.4e+07</td>
</tr>
<tr>
  <th>alpha_30</th>  <td>    0.1004</td> <td>  7.1e+06</td> <td> 1.42e-08</td> <td> 1.000</td> <td>-1.39e+07</td> <td> 1.39e+07</td>
</tr>
<tr>
  <th>alpha_31</th>  <td>    0.1157</td> <td> 2.27e+06</td> <td> 5.09e-08</td> <td> 1.000</td> <td>-4.45e+06</td> <td> 4.45e+06</td>
</tr>
<tr>
  <th>alpha_32</th>  <td>   -0.0167</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_33</th>  <td>   -0.2332</td> <td> 3.58e+06</td> <td>-6.51e-08</td> <td> 1.000</td> <td>-7.01e+06</td> <td> 7.01e+06</td>
</tr>
<tr>
  <th>alpha_34</th>  <td>   -0.1810</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_35</th>  <td>    0.0023</td> <td> 2.79e+06</td> <td> 8.41e-10</td> <td> 1.000</td> <td>-5.46e+06</td> <td> 5.46e+06</td>
</tr>
<tr>
  <th>alpha_36</th>  <td>    0.0215</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_37</th>  <td>    0.0613</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_38</th>  <td>   -0.0230</td> <td>  4.6e+06</td> <td>-4.99e-09</td> <td> 1.000</td> <td>-9.02e+06</td> <td> 9.02e+06</td>
</tr>
<tr>
  <th>alpha_39</th>  <td>    0.1023</td> <td> 3.97e+06</td> <td> 2.57e-08</td> <td> 1.000</td> <td>-7.79e+06</td> <td> 7.79e+06</td>
</tr>
<tr>
  <th>alpha_40</th>  <td>    0.2244</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_41</th>  <td>    0.0752</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_42</th>  <td>   -0.0097</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_43</th>  <td>   -0.0938</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_44</th>  <td>    0.1832</td> <td> 3.83e+06</td> <td> 4.79e-08</td> <td> 1.000</td> <td> -7.5e+06</td> <td>  7.5e+06</td>
</tr>
<tr>
  <th>alpha_45</th>  <td>   -0.2462</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_46</th>  <td>    0.0980</td> <td> 5.29e+06</td> <td> 1.85e-08</td> <td> 1.000</td> <td>-1.04e+07</td> <td> 1.04e+07</td>
</tr>
<tr>
  <th>alpha_47</th>  <td>   -0.1189</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_48</th>  <td>   -0.0588</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_49</th>  <td>    0.0069</td> <td>  5.3e+06</td> <td>  1.3e-09</td> <td> 1.000</td> <td>-1.04e+07</td> <td> 1.04e+07</td>
</tr>
<tr>
  <th>alpha_50</th>  <td>   -0.0620</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_51</th>  <td>   -0.0952</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_52</th>  <td>   -0.3174</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_53</th>  <td>   -0.1900</td> <td> 2.59e+06</td> <td>-7.35e-08</td> <td> 1.000</td> <td>-5.07e+06</td> <td> 5.07e+06</td>
</tr>
<tr>
  <th>alpha_54</th>  <td>   -0.0395</td> <td> 5.76e+06</td> <td>-6.84e-09</td> <td> 1.000</td> <td>-1.13e+07</td> <td> 1.13e+07</td>
</tr>
<tr>
  <th>alpha_55</th>  <td>   -0.0124</td> <td> 1.43e+06</td> <td>-8.66e-09</td> <td> 1.000</td> <td> -2.8e+06</td> <td>  2.8e+06</td>
</tr>
<tr>
  <th>alpha_56</th>  <td>    0.0650</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_57</th>  <td>    0.0311</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_58</th>  <td>    0.0825</td> <td> 1.88e+06</td> <td>  4.4e-08</td> <td> 1.000</td> <td>-3.67e+06</td> <td> 3.67e+06</td>
</tr>
<tr>
  <th>alpha_59</th>  <td>   -0.1329</td> <td> 3.39e+06</td> <td>-3.92e-08</td> <td> 1.000</td> <td>-6.65e+06</td> <td> 6.65e+06</td>
</tr>
<tr>
  <th>alpha_60</th>  <td>   -0.0867</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_61</th>  <td>    0.0450</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_62</th>  <td>   -0.0297</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_63</th>  <td>    0.0498</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_64</th>  <td>   -0.0822</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_65</th>  <td>   -0.1865</td> <td> 4.45e+06</td> <td>-4.19e-08</td> <td> 1.000</td> <td>-8.73e+06</td> <td> 8.73e+06</td>
</tr>
<tr>
  <th>alpha_66</th>  <td>    0.0331</td> <td> 7.61e+06</td> <td> 4.35e-09</td> <td> 1.000</td> <td>-1.49e+07</td> <td> 1.49e+07</td>
</tr>
<tr>
  <th>alpha_67</th>  <td>    0.0592</td> <td> 5.14e+06</td> <td> 1.15e-08</td> <td> 1.000</td> <td>-1.01e+07</td> <td> 1.01e+07</td>
</tr>
<tr>
  <th>alpha_68</th>  <td>    0.0697</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_69</th>  <td>   -0.0273</td> <td> 1.99e+06</td> <td>-1.37e-08</td> <td> 1.000</td> <td> -3.9e+06</td> <td>  3.9e+06</td>
</tr>
<tr>
  <th>alpha_70</th>  <td>    0.2125</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_71</th>  <td>    0.0106</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_72</th>  <td>   -0.0717</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_73</th>  <td>   -0.0563</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_74</th>  <td>   -0.0099</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_75</th>  <td>   -0.0957</td> <td> 2.48e+06</td> <td>-3.86e-08</td> <td> 1.000</td> <td>-4.85e+06</td> <td> 4.85e+06</td>
</tr>
<tr>
  <th>alpha_76</th>  <td>    0.1815</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_77</th>  <td>    0.1622</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_78</th>  <td>   -0.0225</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_79</th>  <td>   -0.1379</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_80</th>  <td>   -0.0905</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_81</th>  <td>    0.0599</td> <td> 3.21e+06</td> <td> 1.87e-08</td> <td> 1.000</td> <td>-6.28e+06</td> <td> 6.28e+06</td>
</tr>
<tr>
  <th>alpha_82</th>  <td>   -0.1238</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_83</th>  <td>   -0.0421</td> <td> 3.89e+06</td> <td>-1.08e-08</td> <td> 1.000</td> <td>-7.63e+06</td> <td> 7.63e+06</td>
</tr>
<tr>
  <th>alpha_84</th>  <td>   -0.1254</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_85</th>  <td>   -0.1435</td> <td>  2.4e+06</td> <td>-5.98e-08</td> <td> 1.000</td> <td>-4.71e+06</td> <td> 4.71e+06</td>
</tr>
<tr>
  <th>alpha_86</th>  <td>    0.0636</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_87</th>  <td>   -0.0943</td> <td> 5.46e+06</td> <td>-1.73e-08</td> <td> 1.000</td> <td>-1.07e+07</td> <td> 1.07e+07</td>
</tr>
<tr>
  <th>alpha_88</th>  <td>   -0.0520</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_89</th>  <td>    0.0431</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_90</th>  <td>   -0.0021</td> <td> 1.77e+06</td> <td>-1.21e-09</td> <td> 1.000</td> <td>-3.47e+06</td> <td> 3.47e+06</td>
</tr>
<tr>
  <th>alpha_91</th>  <td>   -0.1926</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_92</th>  <td>    0.1309</td> <td> 3.14e+06</td> <td> 4.17e-08</td> <td> 1.000</td> <td>-6.15e+06</td> <td> 6.15e+06</td>
</tr>
<tr>
  <th>alpha_93</th>  <td>   -0.1069</td> <td> 4.75e+06</td> <td>-2.25e-08</td> <td> 1.000</td> <td>-9.31e+06</td> <td> 9.31e+06</td>
</tr>
<tr>
  <th>alpha_94</th>  <td>   -0.0025</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_95</th>  <td>   -0.0657</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_96</th>  <td>    0.0782</td> <td> 9.13e+06</td> <td> 8.56e-09</td> <td> 1.000</td> <td>-1.79e+07</td> <td> 1.79e+07</td>
</tr>
<tr>
  <th>alpha_97</th>  <td>   -0.1390</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_98</th>  <td>   -0.2913</td> <td> 4.43e+06</td> <td>-6.57e-08</td> <td> 1.000</td> <td>-8.69e+06</td> <td> 8.69e+06</td>
</tr>
<tr>
  <th>alpha_99</th>  <td>   -0.1440</td> <td> 4.52e+06</td> <td>-3.19e-08</td> <td> 1.000</td> <td>-8.86e+06</td> <td> 8.86e+06</td>
</tr>
<tr>
  <th>alpha_100</th> <td>   -0.0950</td> <td> 2.27e+06</td> <td>-4.18e-08</td> <td> 1.000</td> <td>-4.46e+06</td> <td> 4.46e+06</td>
</tr>
<tr>
  <th>alpha_101</th> <td>   -0.1456</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_102</th> <td>    0.0483</td> <td> 9.61e+06</td> <td> 5.02e-09</td> <td> 1.000</td> <td>-1.88e+07</td> <td> 1.88e+07</td>
</tr>
<tr>
  <th>alpha_103</th> <td>   -0.2813</td> <td> 2.85e+06</td> <td>-9.88e-08</td> <td> 1.000</td> <td>-5.58e+06</td> <td> 5.58e+06</td>
</tr>
<tr>
  <th>alpha_104</th> <td>   -0.1262</td> <td> 3.07e+06</td> <td>-4.11e-08</td> <td> 1.000</td> <td>-6.02e+06</td> <td> 6.02e+06</td>
</tr>
<tr>
  <th>alpha_105</th> <td>   -0.1589</td> <td> 8.88e+05</td> <td>-1.79e-07</td> <td> 1.000</td> <td>-1.74e+06</td> <td> 1.74e+06</td>
</tr>
<tr>
  <th>alpha_106</th> <td>    0.0296</td> <td> 1.96e+06</td> <td> 1.51e-08</td> <td> 1.000</td> <td>-3.84e+06</td> <td> 3.84e+06</td>
</tr>
<tr>
  <th>alpha_107</th> <td>   -0.0162</td> <td> 6.84e+06</td> <td>-2.37e-09</td> <td> 1.000</td> <td>-1.34e+07</td> <td> 1.34e+07</td>
</tr>
<tr>
  <th>alpha_108</th> <td>    0.0208</td> <td>  2.4e+06</td> <td> 8.66e-09</td> <td> 1.000</td> <td>-4.71e+06</td> <td> 4.71e+06</td>
</tr>
<tr>
  <th>alpha_109</th> <td>    0.1239</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_110</th> <td>    0.0235</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_111</th> <td>    0.0044</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_112</th> <td>   -0.2971</td> <td> 8.99e+05</td> <td>-3.31e-07</td> <td> 1.000</td> <td>-1.76e+06</td> <td> 1.76e+06</td>
</tr>
<tr>
  <th>alpha_113</th> <td>    0.2028</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_114</th> <td>   -0.0756</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_115</th> <td>    0.0083</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_116</th> <td>   -0.0351</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_117</th> <td>    0.0773</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_118</th> <td>    0.0768</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_119</th> <td>    0.0453</td> <td> 5.07e+06</td> <td> 8.94e-09</td> <td> 1.000</td> <td>-9.93e+06</td> <td> 9.93e+06</td>
</tr>
<tr>
  <th>alpha_120</th> <td>    0.0896</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_121</th> <td>    0.0323</td> <td> 2.24e+06</td> <td> 1.44e-08</td> <td> 1.000</td> <td>-4.39e+06</td> <td> 4.39e+06</td>
</tr>
<tr>
  <th>alpha_122</th> <td>    0.0729</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_123</th> <td>    0.1574</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_124</th> <td>    0.0126</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_125</th> <td>    0.0614</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_126</th> <td>   -0.3628</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_127</th> <td>   -0.1861</td> <td>  2.8e+06</td> <td>-6.65e-08</td> <td> 1.000</td> <td>-5.49e+06</td> <td> 5.49e+06</td>
</tr>
<tr>
  <th>alpha_128</th> <td>    0.1717</td> <td> 3.29e+06</td> <td> 5.22e-08</td> <td> 1.000</td> <td>-6.45e+06</td> <td> 6.45e+06</td>
</tr>
<tr>
  <th>alpha_129</th> <td>   -0.0119</td> <td> 5.28e+06</td> <td>-2.26e-09</td> <td> 1.000</td> <td>-1.03e+07</td> <td> 1.03e+07</td>
</tr>
<tr>
  <th>alpha_130</th> <td>    0.1482</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_131</th> <td>    0.0230</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_132</th> <td>   -0.2162</td> <td> 3.36e+06</td> <td>-6.43e-08</td> <td> 1.000</td> <td>-6.59e+06</td> <td> 6.59e+06</td>
</tr>
<tr>
  <th>alpha_133</th> <td>    0.1891</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_134</th> <td>    0.0137</td> <td>  2.4e+06</td> <td> 5.72e-09</td> <td> 1.000</td> <td> -4.7e+06</td> <td>  4.7e+06</td>
</tr>
<tr>
  <th>alpha_135</th> <td>    0.1030</td> <td> 3.32e+06</td> <td>  3.1e-08</td> <td> 1.000</td> <td>-6.51e+06</td> <td> 6.51e+06</td>
</tr>
<tr>
  <th>alpha_136</th> <td>    0.1470</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_137</th> <td>    0.1630</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_138</th> <td>    0.0529</td> <td> 1.87e+06</td> <td> 2.83e-08</td> <td> 1.000</td> <td>-3.67e+06</td> <td> 3.67e+06</td>
</tr>
<tr>
  <th>alpha_139</th> <td>   -0.1658</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_140</th> <td>    0.0811</td> <td> 3.29e+06</td> <td> 2.47e-08</td> <td> 1.000</td> <td>-6.44e+06</td> <td> 6.44e+06</td>
</tr>
<tr>
  <th>alpha_141</th> <td>    0.2265</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_142</th> <td>   -0.0778</td> <td> 6.18e+06</td> <td>-1.26e-08</td> <td> 1.000</td> <td>-1.21e+07</td> <td> 1.21e+07</td>
</tr>
<tr>
  <th>alpha_143</th> <td>   -0.2297</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_144</th> <td>    0.0126</td> <td> 2.17e+06</td> <td>  5.8e-09</td> <td> 1.000</td> <td>-4.25e+06</td> <td> 4.25e+06</td>
</tr>
<tr>
  <th>alpha_145</th> <td>   -0.1028</td> <td> 1.54e+06</td> <td>-6.68e-08</td> <td> 1.000</td> <td>-3.02e+06</td> <td> 3.02e+06</td>
</tr>
<tr>
  <th>alpha_146</th> <td>   -0.0012</td> <td> 5.51e+06</td> <td> -2.1e-10</td> <td> 1.000</td> <td>-1.08e+07</td> <td> 1.08e+07</td>
</tr>
<tr>
  <th>alpha_147</th> <td>   -0.1505</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_148</th> <td>    0.0560</td> <td> 2.66e+06</td> <td>  2.1e-08</td> <td> 1.000</td> <td>-5.22e+06</td> <td> 5.22e+06</td>
</tr>
<tr>
  <th>alpha_149</th> <td>   -0.0485</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_150</th> <td>   -0.0721</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_151</th> <td>   -0.0257</td> <td> 2.12e+06</td> <td>-1.21e-08</td> <td> 1.000</td> <td>-4.16e+06</td> <td> 4.16e+06</td>
</tr>
<tr>
  <th>alpha_152</th> <td>    0.0594</td> <td> 4.08e+06</td> <td> 1.46e-08</td> <td> 1.000</td> <td>-7.99e+06</td> <td> 7.99e+06</td>
</tr>
<tr>
  <th>alpha_153</th> <td>   -0.1511</td> <td> 2.47e+06</td> <td>-6.13e-08</td> <td> 1.000</td> <td>-4.83e+06</td> <td> 4.83e+06</td>
</tr>
<tr>
  <th>alpha_154</th> <td>   -0.1401</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_155</th> <td>    0.0431</td> <td> 3.23e+06</td> <td> 1.33e-08</td> <td> 1.000</td> <td>-6.33e+06</td> <td> 6.33e+06</td>
</tr>
<tr>
  <th>alpha_156</th> <td>    0.0569</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_157</th> <td>   -0.0333</td> <td> 1.64e+06</td> <td>-2.04e-08</td> <td> 1.000</td> <td>-3.21e+06</td> <td> 3.21e+06</td>
</tr>
<tr>
  <th>alpha_158</th> <td>   -0.1185</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_159</th> <td>    0.0800</td> <td> 4.28e+06</td> <td> 1.87e-08</td> <td> 1.000</td> <td>-8.38e+06</td> <td> 8.38e+06</td>
</tr>
<tr>
  <th>alpha_160</th> <td>    0.0015</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_161</th> <td>   -0.0234</td> <td> 1.49e+06</td> <td>-1.57e-08</td> <td> 1.000</td> <td>-2.92e+06</td> <td> 2.92e+06</td>
</tr>
<tr>
  <th>alpha_162</th> <td>    0.0708</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_163</th> <td>   -0.0849</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_164</th> <td>    0.0766</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_165</th> <td>   -0.1192</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_166</th> <td>   -0.0742</td> <td> 3.94e+06</td> <td>-1.88e-08</td> <td> 1.000</td> <td>-7.72e+06</td> <td> 7.72e+06</td>
</tr>
<tr>
  <th>alpha_167</th> <td>   -0.2277</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_168</th> <td>    0.1165</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_169</th> <td>    0.0760</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_170</th> <td>   -0.0497</td> <td> 1.54e+06</td> <td>-3.24e-08</td> <td> 1.000</td> <td>-3.01e+06</td> <td> 3.01e+06</td>
</tr>
<tr>
  <th>alpha_171</th> <td>    0.1372</td> <td> 1.83e+06</td> <td> 7.49e-08</td> <td> 1.000</td> <td>-3.59e+06</td> <td> 3.59e+06</td>
</tr>
<tr>
  <th>alpha_172</th> <td>    0.0494</td> <td> 1.38e+06</td> <td> 3.57e-08</td> <td> 1.000</td> <td>-2.71e+06</td> <td> 2.71e+06</td>
</tr>
<tr>
  <th>alpha_173</th> <td>    0.0543</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_174</th> <td>   -0.0303</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_175</th> <td>   -0.0426</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_176</th> <td>   -0.2416</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_177</th> <td>   -0.2136</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_178</th> <td>    0.0035</td> <td> 4.27e+06</td> <td> 8.16e-10</td> <td> 1.000</td> <td>-8.37e+06</td> <td> 8.37e+06</td>
</tr>
<tr>
  <th>alpha_179</th> <td>   -0.0043</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_180</th> <td>   -0.2604</td> <td> 1.97e+06</td> <td>-1.33e-07</td> <td> 1.000</td> <td>-3.85e+06</td> <td> 3.85e+06</td>
</tr>
<tr>
  <th>alpha_181</th> <td>   -0.0043</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_182</th> <td>    0.1750</td> <td> 1.94e+06</td> <td> 9.02e-08</td> <td> 1.000</td> <td>-3.81e+06</td> <td> 3.81e+06</td>
</tr>
<tr>
  <th>alpha_183</th> <td>   -0.1624</td> <td> 5.57e+06</td> <td>-2.92e-08</td> <td> 1.000</td> <td>-1.09e+07</td> <td> 1.09e+07</td>
</tr>
<tr>
  <th>alpha_184</th> <td>   -0.0081</td> <td> 1.59e+06</td> <td>-5.08e-09</td> <td> 1.000</td> <td>-3.11e+06</td> <td> 3.11e+06</td>
</tr>
<tr>
  <th>alpha_185</th> <td>    0.0542</td> <td> 2.22e+06</td> <td> 2.44e-08</td> <td> 1.000</td> <td>-4.36e+06</td> <td> 4.36e+06</td>
</tr>
<tr>
  <th>alpha_186</th> <td>   -0.0353</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_187</th> <td>    0.0813</td> <td> 2.06e+06</td> <td> 3.94e-08</td> <td> 1.000</td> <td>-4.04e+06</td> <td> 4.04e+06</td>
</tr>
<tr>
  <th>alpha_188</th> <td>    0.0463</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_189</th> <td>   -0.0821</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_190</th> <td>   -0.1220</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_191</th> <td>   -0.1454</td> <td> 1.93e+06</td> <td>-7.55e-08</td> <td> 1.000</td> <td>-3.77e+06</td> <td> 3.77e+06</td>
</tr>
<tr>
  <th>alpha_192</th> <td>    0.0037</td> <td> 1.31e+06</td> <td> 2.83e-09</td> <td> 1.000</td> <td>-2.56e+06</td> <td> 2.56e+06</td>
</tr>
<tr>
  <th>alpha_193</th> <td>   -0.1975</td> <td>  2.8e+06</td> <td>-7.06e-08</td> <td> 1.000</td> <td>-5.48e+06</td> <td> 5.48e+06</td>
</tr>
<tr>
  <th>alpha_194</th> <td>   -0.2787</td> <td> 2.29e+06</td> <td>-1.22e-07</td> <td> 1.000</td> <td>-4.48e+06</td> <td> 4.48e+06</td>
</tr>
<tr>
  <th>alpha_195</th> <td>   -0.1792</td> <td> 3.07e+06</td> <td>-5.84e-08</td> <td> 1.000</td> <td>-6.01e+06</td> <td> 6.01e+06</td>
</tr>
<tr>
  <th>alpha_196</th> <td>   -0.0844</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_197</th> <td>    0.0059</td> <td> 4.68e+06</td> <td> 1.27e-09</td> <td> 1.000</td> <td>-9.18e+06</td> <td> 9.18e+06</td>
</tr>
<tr>
  <th>alpha_198</th> <td>   -0.0317</td> <td>  1.6e+06</td> <td>-1.99e-08</td> <td> 1.000</td> <td>-3.13e+06</td> <td> 3.13e+06</td>
</tr>
<tr>
  <th>alpha_199</th> <td>    0.0823</td> <td> 6.69e+06</td> <td> 1.23e-08</td> <td> 1.000</td> <td>-1.31e+07</td> <td> 1.31e+07</td>
</tr>
<tr>
  <th>alpha_200</th> <td>   -0.0092</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_201</th> <td>   -0.0512</td> <td> 2.49e+06</td> <td>-2.06e-08</td> <td> 1.000</td> <td>-4.88e+06</td> <td> 4.88e+06</td>
</tr>
<tr>
  <th>alpha_202</th> <td>   -0.2186</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_203</th> <td>    0.0616</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_204</th> <td>    0.1598</td> <td> 5.78e+06</td> <td> 2.77e-08</td> <td> 1.000</td> <td>-1.13e+07</td> <td> 1.13e+07</td>
</tr>
<tr>
  <th>alpha_205</th> <td>   -0.0086</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_206</th> <td>   -0.2852</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_207</th> <td>    0.1323</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_208</th> <td>   -0.0135</td> <td> 2.79e+06</td> <td>-4.84e-09</td> <td> 1.000</td> <td>-5.48e+06</td> <td> 5.48e+06</td>
</tr>
<tr>
  <th>alpha_209</th> <td>   -0.2285</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_210</th> <td>    0.0053</td> <td> 1.21e+06</td> <td> 4.36e-09</td> <td> 1.000</td> <td>-2.38e+06</td> <td> 2.38e+06</td>
</tr>
<tr>
  <th>alpha_211</th> <td>   -0.0127</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_212</th> <td>   -0.2706</td> <td> 3.89e+06</td> <td>-6.95e-08</td> <td> 1.000</td> <td>-7.63e+06</td> <td> 7.63e+06</td>
</tr>
<tr>
  <th>alpha_213</th> <td>    0.0254</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_214</th> <td>   -0.0553</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_215</th> <td>    0.1737</td> <td> 2.69e+06</td> <td> 6.46e-08</td> <td> 1.000</td> <td>-5.27e+06</td> <td> 5.27e+06</td>
</tr>
<tr>
  <th>alpha_216</th> <td>    0.0488</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_217</th> <td>   -0.1708</td> <td> 1.32e+06</td> <td>-1.29e-07</td> <td> 1.000</td> <td>-2.58e+06</td> <td> 2.58e+06</td>
</tr>
<tr>
  <th>alpha_218</th> <td>   -0.0962</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_219</th> <td>   -0.1178</td> <td> 3.46e+06</td> <td> -3.4e-08</td> <td> 1.000</td> <td>-6.79e+06</td> <td> 6.79e+06</td>
</tr>
<tr>
  <th>alpha_220</th> <td>    0.0096</td> <td> 3.62e+06</td> <td> 2.64e-09</td> <td> 1.000</td> <td> -7.1e+06</td> <td>  7.1e+06</td>
</tr>
<tr>
  <th>alpha_221</th> <td>   -0.0139</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_222</th> <td>    0.0855</td> <td> 6.59e+05</td> <td>  1.3e-07</td> <td> 1.000</td> <td>-1.29e+06</td> <td> 1.29e+06</td>
</tr>
<tr>
  <th>alpha_223</th> <td>   -0.0232</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_224</th> <td>   -0.1281</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_225</th> <td>   -0.0588</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_226</th> <td>    0.1737</td> <td> 2.16e+06</td> <td> 8.03e-08</td> <td> 1.000</td> <td>-4.24e+06</td> <td> 4.24e+06</td>
</tr>
<tr>
  <th>alpha_227</th> <td>   -0.1555</td> <td> 2.96e+06</td> <td>-5.25e-08</td> <td> 1.000</td> <td>-5.81e+06</td> <td> 5.81e+06</td>
</tr>
<tr>
  <th>alpha_228</th> <td>    0.0411</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_229</th> <td>    0.0263</td> <td> 2.17e+06</td> <td> 1.21e-08</td> <td> 1.000</td> <td>-4.24e+06</td> <td> 4.24e+06</td>
</tr>
<tr>
  <th>alpha_230</th> <td>   -0.1260</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_231</th> <td>   -0.0563</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_232</th> <td>   -0.1642</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_233</th> <td>   -0.1848</td> <td> 6.35e+06</td> <td>-2.91e-08</td> <td> 1.000</td> <td>-1.25e+07</td> <td> 1.25e+07</td>
</tr>
<tr>
  <th>alpha_234</th> <td>   -0.0164</td> <td> 1.67e+06</td> <td>-9.81e-09</td> <td> 1.000</td> <td>-3.28e+06</td> <td> 3.28e+06</td>
</tr>
<tr>
  <th>alpha_235</th> <td>    0.1883</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_236</th> <td>   -0.1571</td> <td> 1.31e+06</td> <td> -1.2e-07</td> <td> 1.000</td> <td>-2.57e+06</td> <td> 2.57e+06</td>
</tr>
<tr>
  <th>alpha_237</th> <td>   -0.3025</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_238</th> <td>   -0.0345</td> <td> 4.94e+06</td> <td>-6.98e-09</td> <td> 1.000</td> <td>-9.69e+06</td> <td> 9.69e+06</td>
</tr>
<tr>
  <th>alpha_239</th> <td>    0.0552</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_240</th> <td>    0.0464</td> <td> 3.44e+06</td> <td> 1.35e-08</td> <td> 1.000</td> <td>-6.74e+06</td> <td> 6.74e+06</td>
</tr>
<tr>
  <th>alpha_241</th> <td>   -0.1885</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_242</th> <td>    0.0461</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_243</th> <td>    0.0189</td> <td> 4.42e+06</td> <td> 4.27e-09</td> <td> 1.000</td> <td>-8.67e+06</td> <td> 8.67e+06</td>
</tr>
<tr>
  <th>alpha_244</th> <td>    0.1661</td> <td> 9.94e+05</td> <td> 1.67e-07</td> <td> 1.000</td> <td>-1.95e+06</td> <td> 1.95e+06</td>
</tr>
<tr>
  <th>alpha_245</th> <td>   -0.1414</td> <td> 5.13e+05</td> <td>-2.76e-07</td> <td> 1.000</td> <td>-1.01e+06</td> <td> 1.01e+06</td>
</tr>
<tr>
  <th>alpha_246</th> <td>    0.0408</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_247</th> <td>    0.0032</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_248</th> <td>   -0.2314</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_249</th> <td>   -0.2004</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_250</th> <td>   -0.0854</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_251</th> <td>   -0.0119</td> <td>  1.2e+05</td> <td>-9.85e-08</td> <td> 1.000</td> <td>-2.36e+05</td> <td> 2.36e+05</td>
</tr>
<tr>
  <th>alpha_252</th> <td>   -0.0351</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_253</th> <td>   -0.0369</td> <td> 2.36e+06</td> <td>-1.56e-08</td> <td> 1.000</td> <td>-4.63e+06</td> <td> 4.63e+06</td>
</tr>
<tr>
  <th>alpha_254</th> <td>    0.2275</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_255</th> <td>   -0.1291</td> <td> 2.21e+06</td> <td>-5.84e-08</td> <td> 1.000</td> <td>-4.33e+06</td> <td> 4.33e+06</td>
</tr>
<tr>
  <th>alpha_256</th> <td>    0.1254</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_257</th> <td>   -0.0045</td> <td> 2.57e+06</td> <td>-1.75e-09</td> <td> 1.000</td> <td>-5.03e+06</td> <td> 5.03e+06</td>
</tr>
<tr>
  <th>alpha_258</th> <td>    0.0640</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_259</th> <td>    0.0309</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_260</th> <td>    0.1127</td> <td> 4.31e+06</td> <td> 2.62e-08</td> <td> 1.000</td> <td>-8.45e+06</td> <td> 8.45e+06</td>
</tr>
<tr>
  <th>alpha_261</th> <td>    0.1426</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_262</th> <td>    0.0249</td> <td> 1.71e+06</td> <td> 1.46e-08</td> <td> 1.000</td> <td>-3.35e+06</td> <td> 3.35e+06</td>
</tr>
<tr>
  <th>alpha_263</th> <td>    0.1937</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_264</th> <td>   -0.0702</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_265</th> <td>    0.0252</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_266</th> <td>    0.0804</td> <td> 2.88e+06</td> <td> 2.79e-08</td> <td> 1.000</td> <td>-5.65e+06</td> <td> 5.65e+06</td>
</tr>
<tr>
  <th>alpha_267</th> <td>   -0.0415</td> <td>  2.5e+06</td> <td>-1.66e-08</td> <td> 1.000</td> <td>-4.89e+06</td> <td> 4.89e+06</td>
</tr>
<tr>
  <th>alpha_268</th> <td>    0.0553</td> <td> 3.26e+06</td> <td>  1.7e-08</td> <td> 1.000</td> <td>-6.39e+06</td> <td> 6.39e+06</td>
</tr>
<tr>
  <th>alpha_269</th> <td>    0.0414</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_270</th> <td>   -0.0161</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_271</th> <td>   -0.1356</td> <td> 2.94e+06</td> <td> -4.6e-08</td> <td> 1.000</td> <td>-5.77e+06</td> <td> 5.77e+06</td>
</tr>
<tr>
  <th>alpha_272</th> <td>    0.0215</td> <td> 4.56e+06</td> <td> 4.72e-09</td> <td> 1.000</td> <td>-8.95e+06</td> <td> 8.95e+06</td>
</tr>
<tr>
  <th>alpha_273</th> <td>    0.0413</td> <td> 5.42e+06</td> <td> 7.61e-09</td> <td> 1.000</td> <td>-1.06e+07</td> <td> 1.06e+07</td>
</tr>
<tr>
  <th>alpha_274</th> <td>    0.1034</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_275</th> <td>   -0.0340</td> <td> 7.19e+06</td> <td>-4.73e-09</td> <td> 1.000</td> <td>-1.41e+07</td> <td> 1.41e+07</td>
</tr>
<tr>
  <th>alpha_276</th> <td>   -0.1941</td> <td> 2.78e+06</td> <td>-6.99e-08</td> <td> 1.000</td> <td>-5.45e+06</td> <td> 5.45e+06</td>
</tr>
<tr>
  <th>alpha_277</th> <td>   -0.2046</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_278</th> <td>   -0.2574</td> <td> 2.34e+06</td> <td> -1.1e-07</td> <td> 1.000</td> <td>-4.58e+06</td> <td> 4.58e+06</td>
</tr>
<tr>
  <th>alpha_279</th> <td>   -0.0351</td> <td> 4.55e+06</td> <td>-7.71e-09</td> <td> 1.000</td> <td>-8.93e+06</td> <td> 8.93e+06</td>
</tr>
<tr>
  <th>alpha_280</th> <td>   -0.2097</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_281</th> <td>    0.0694</td> <td> 6.12e+06</td> <td> 1.13e-08</td> <td> 1.000</td> <td> -1.2e+07</td> <td>  1.2e+07</td>
</tr>
<tr>
  <th>alpha_282</th> <td>   -0.2733</td> <td> 1.79e+06</td> <td>-1.53e-07</td> <td> 1.000</td> <td> -3.5e+06</td> <td>  3.5e+06</td>
</tr>
<tr>
  <th>alpha_283</th> <td>    0.0476</td> <td> 2.02e+06</td> <td> 2.36e-08</td> <td> 1.000</td> <td>-3.95e+06</td> <td> 3.95e+06</td>
</tr>
<tr>
  <th>alpha_284</th> <td>    0.0117</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_285</th> <td>    0.0404</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_286</th> <td>    0.0497</td> <td>  4.9e+06</td> <td> 1.01e-08</td> <td> 1.000</td> <td> -9.6e+06</td> <td>  9.6e+06</td>
</tr>
<tr>
  <th>alpha_287</th> <td>   -0.0898</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_288</th> <td>    0.1765</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_289</th> <td>   -0.2823</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_290</th> <td>    0.0345</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_291</th> <td>    0.0331</td> <td> 2.21e+06</td> <td>  1.5e-08</td> <td> 1.000</td> <td>-4.33e+06</td> <td> 4.33e+06</td>
</tr>
<tr>
  <th>alpha_292</th> <td>   -0.0943</td> <td> 5.16e+06</td> <td>-1.83e-08</td> <td> 1.000</td> <td>-1.01e+07</td> <td> 1.01e+07</td>
</tr>
<tr>
  <th>alpha_293</th> <td>   -0.2573</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_294</th> <td>    0.0413</td> <td> 7.09e+05</td> <td> 5.83e-08</td> <td> 1.000</td> <td>-1.39e+06</td> <td> 1.39e+06</td>
</tr>
<tr>
  <th>alpha_295</th> <td>   -0.0415</td> <td> 2.93e+06</td> <td>-1.42e-08</td> <td> 1.000</td> <td>-5.75e+06</td> <td> 5.75e+06</td>
</tr>
<tr>
  <th>alpha_296</th> <td>    0.0145</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_297</th> <td>   -0.0695</td> <td> 2.04e+06</td> <td>-3.41e-08</td> <td> 1.000</td> <td>   -4e+06</td> <td>    4e+06</td>
</tr>
<tr>
  <th>alpha_298</th> <td>    0.1025</td> <td> 2.09e+06</td> <td>  4.9e-08</td> <td> 1.000</td> <td> -4.1e+06</td> <td>  4.1e+06</td>
</tr>
<tr>
  <th>alpha_299</th> <td>   -0.0479</td> <td> 2.39e+06</td> <td>   -2e-08</td> <td> 1.000</td> <td>-4.69e+06</td> <td> 4.69e+06</td>
</tr>
<tr>
  <th>alpha_300</th> <td>    0.0355</td> <td> 4.62e+06</td> <td> 7.68e-09</td> <td> 1.000</td> <td>-9.06e+06</td> <td> 9.06e+06</td>
</tr>
<tr>
  <th>alpha_301</th> <td>   -0.0802</td> <td> 5.25e+06</td> <td>-1.53e-08</td> <td> 1.000</td> <td>-1.03e+07</td> <td> 1.03e+07</td>
</tr>
<tr>
  <th>alpha_302</th> <td>   -0.0184</td> <td> 6.87e+05</td> <td>-2.68e-08</td> <td> 1.000</td> <td>-1.35e+06</td> <td> 1.35e+06</td>
</tr>
<tr>
  <th>alpha_303</th> <td>    0.0760</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_304</th> <td>   -0.0437</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_305</th> <td>    0.0244</td> <td> 2.52e+06</td> <td>  9.7e-09</td> <td> 1.000</td> <td>-4.93e+06</td> <td> 4.93e+06</td>
</tr>
<tr>
  <th>alpha_306</th> <td>    0.1834</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_307</th> <td>   -0.1459</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_308</th> <td>    0.0737</td> <td> 1.36e+06</td> <td>  5.4e-08</td> <td> 1.000</td> <td>-2.67e+06</td> <td> 2.67e+06</td>
</tr>
<tr>
  <th>alpha_309</th> <td>    0.0103</td> <td>  7.4e+06</td> <td> 1.39e-09</td> <td> 1.000</td> <td>-1.45e+07</td> <td> 1.45e+07</td>
</tr>
<tr>
  <th>alpha_310</th> <td>   -0.0891</td> <td> 4.45e+06</td> <td>   -2e-08</td> <td> 1.000</td> <td>-8.72e+06</td> <td> 8.72e+06</td>
</tr>
<tr>
  <th>alpha_311</th> <td>   -0.1262</td> <td> 2.99e+06</td> <td>-4.23e-08</td> <td> 1.000</td> <td>-5.85e+06</td> <td> 5.85e+06</td>
</tr>
<tr>
  <th>alpha_312</th> <td>    0.0527</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_313</th> <td>    0.0296</td> <td> 4.83e+06</td> <td> 6.14e-09</td> <td> 1.000</td> <td>-9.46e+06</td> <td> 9.46e+06</td>
</tr>
<tr>
  <th>alpha_314</th> <td>    0.0723</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_315</th> <td>   -0.0476</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
<tr>
  <th>alpha_316</th> <td>   -0.2535</td> <td> 2.85e+06</td> <td>-8.89e-08</td> <td> 1.000</td> <td>-5.59e+06</td> <td> 5.59e+06</td>
</tr>
<tr>
  <th>alpha_317</th> <td>   -0.2573</td> <td>      nan</td> <td>      nan</td> <td>   nan</td> <td>      nan</td> <td>      nan</td>
</tr>
</table>



It is more interesting to compute the confusion matrix and the precision, recall and F-score metric such as in the previous model. Nonetheless, the first step must be to compute the long format kernel matrix for the test set that is necessary to compute the previous metrics using KLR. This can be done using the following code.


```python
# Comute the long format kernel matrix for the test set.
K_long_format_test = pkl.long_format_with_kernel_matrix(test_set_scaled, 
                                                        custom_alt_id, 
                                                        obs_id_column, 
                                                        choice_column, 
                                                        variables, 
                                                        kernel_type = "RBF", 
                                                        Z = train_set_scaled,
                                                        length_scale = 1)
```


```python
# Compute the confunsion matrix using the test set
confusion_matrix = kernel_model.confusion_matrix(K_long_format_test)

# The generated confusion matrix can be shown directly as a pandas dataframe
confusion_matrix
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>57</td>
      <td>0</td>
      <td>0</td>
      <td>7</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>0</td>
      <td>0</td>
      <td>4</td>
    </tr>
    <tr>
      <th>3</th>
      <td>7</td>
      <td>0</td>
      <td>11</td>
      <td>8</td>
    </tr>
    <tr>
      <th>4</th>
      <td>8</td>
      <td>0</td>
      <td>6</td>
      <td>25</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Visualize the confusion matrix using seaborn.
# Generate a new figure and set the figure size.
plt.figure(figsize=(10,5))

# Generate a heatmap for the confusion matrix using seaborn
ax = sns.heatmap(confusion_matrix, annot=True, linewidth=0.5, cmap='PuBu')

# labels, title and ticks
ax.set_xlabel('Predicted alternative')
ax.set_ylabel('True alternative')
ax.set_title('Confusion Matrix')
```


```python
# Compute the precision, recall and F1-score for the linear model using a
# Micro-averaging technique. The F1-score is the F-score for beta = 1.
kernel_model.precision_recall_fscore(K_long_format_test, "micro", beta = 1)
```




    (0.6838235294117647, 0.6838235294117647, 0.6838235294117647)



### 3.4. Prediction using the estimated model

Again, once the model has been estimated, new instances can be predicted using the model.


```python
# The different observations from the test_set are represented on each row.
# The columns represent each of the alternatives. The value from each cell
# is the probability that the alternative on the corresponding column was
# selected on that observation.
pd.DataFrame(data=kernel_model.predict(K_long_format_test).reshape(-1, 4))
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.916604</td>
      <td>0.017250</td>
      <td>0.040786</td>
      <td>0.025359</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.739171</td>
      <td>0.024550</td>
      <td>0.082291</td>
      <td>0.153987</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.953754</td>
      <td>0.033085</td>
      <td>0.004249</td>
      <td>0.008912</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.314859</td>
      <td>0.203336</td>
      <td>0.105414</td>
      <td>0.376390</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.149059</td>
      <td>0.049907</td>
      <td>0.112112</td>
      <td>0.688923</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>131</th>
      <td>0.217392</td>
      <td>0.137808</td>
      <td>0.116807</td>
      <td>0.527993</td>
    </tr>
    <tr>
      <th>132</th>
      <td>0.612135</td>
      <td>0.105613</td>
      <td>0.090062</td>
      <td>0.192190</td>
    </tr>
    <tr>
      <th>133</th>
      <td>0.764801</td>
      <td>0.114906</td>
      <td>0.084024</td>
      <td>0.036269</td>
    </tr>
    <tr>
      <th>134</th>
      <td>0.660301</td>
      <td>0.128328</td>
      <td>0.030257</td>
      <td>0.181114</td>
    </tr>
    <tr>
      <th>135</th>
      <td>0.828948</td>
      <td>0.071262</td>
      <td>0.072905</td>
      <td>0.026885</td>
    </tr>
  </tbody>
</table>
<p>136 rows × 4 columns</p>
</div>



### 3.5. Computation of the Willingness to Pay (WTP) indicator using the estimated KLR model

As shown previously, the equation to compute the WTP is:
\begin{equation}
    WTP = \frac{\delta_{i n}^{c}}{\delta_{i n}^{x}}=-\frac{\left(\partial V_{i n} / \partial x_{i n}\right)\left(c_{i n}, x_{i n}\right)}{\left(\partial V_{i n} / \partial c_{i n}\right)\left(c_{i n}, x_{i n}\right)}
\end{equation}
where $c_{in}$ is the cost of the alternative $i$ for a decision-maker $n$ and $x_{in}$ is the value of another variable from the model. The WTP can be computed directly using a close expression for the linear RUM (as shown in section 2.5, nonetheless, for the KLR models it is necessary to compute the gradient vector of the partial derivatives of the systematic utility function. 

Since KLR is a non-linear model, the first step is to define the point where the WTP is going to be computed. Any observation from a long format dataframe can be selected as this WTP point and, therefore, the WTP for that decision-maker can be computed. Moreover, the analyst can use any point for the WTP calculation, but this point should be provided in a long format dataframe.

The following lines of code shows how to compute the WTP for any observation from the dataframe using KLR. The first step is to select the WTP point.


```python
# Compute the WTP of any observation from the dataframe
WTP_point = long_mode_choice.loc[long_mode_choice[obs_id_column] == 12]

WTP_point
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>custom_id</th>
      <th>mode_id</th>
      <th>choice</th>
      <th>cost</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>44</th>
      <td>12</td>
      <td>1</td>
      <td>1</td>
      <td>1.864544</td>
      <td>17.530151</td>
    </tr>
    <tr>
      <th>45</th>
      <td>12</td>
      <td>2</td>
      <td>0</td>
      <td>0.568055</td>
      <td>26.731061</td>
    </tr>
    <tr>
      <th>46</th>
      <td>12</td>
      <td>3</td>
      <td>0</td>
      <td>2.167807</td>
      <td>55.347093</td>
    </tr>
    <tr>
      <th>47</th>
      <td>12</td>
      <td>4</td>
      <td>0</td>
      <td>1.978343</td>
      <td>32.651246</td>
    </tr>
  </tbody>
</table>
</div>



Next, the PyKernelLogit function calculate_WTP_kernel can be used to calculate the WTP indicador for an observation using KLR by numerically approximating the gradient vector of the systematic utility. This function takes following input arguments:
* WTP_cost_column. Column name with contains the cost or price of the alternative.

* WTP_x_column. Column name with contains the variable to be analyzed.

* alt. Indicates the alternative for which the WTP is computed.

* model. A MNDC_Model instance corresponding with the trained model.

* variables. A Python dictionary where each key represents each of the alternatives IDs and its associated value contains a list of characteristics from the long dataframe WTP_point and Z to be used on that alternative.

* WTP_point. A dataframe in long format with only one observation (1 row per each alternative of that observation) for which the Willingness to Pay is going to be computed.

* Z. A reference matrix $\mathbf{Z}_{i}$ to be used when computing the kernel matrix.

* kernel_type. The type of kernel function to be used.

* scaler. If none scaler was used to fit the kernel model, this parameter must be None. Otherwise, an instance of the scaler function must be provided here. The provided scaler must contain a transform function which scale the features of the dataset according to the scaler definition.

* epsilon. The $\varepsilon$ parameter. The increment of the coordinate vector used for determining the function gradient.  

As it was described previously the Value of time (VOT) reflects the price that a traveler is willing to pay to decrease the travel time in one unit. In this example, it is interesting to compute the **VOT** as the model contains the cost and time variables.



```python
# Comuthe the VOT for a KLR model, i.e. the WTP using cost an time variables.
# Take into account that travel time is in minutes, therefore, the VOT must be
# multiplied by 60 to convert it to hours.
VOT = -60 * pkl.calculate_WTP_kernel(WTP_cost_column = 'cost', 
                                    WTP_x_column = 'time', 
                                    alt = 1, 
                                    model = kernel_model, 
                                    variables = variables, 
                                    WTP_point = WTP_point, 
                                    Z = train_set_scaled, 
                                    kernel_type = "RBF", 
                                    scaler = scaler,
                                    scaled_fetures=['cost', 'time'])

VOT
```




    7.617601618103002



The analyst can evalute the WTP also for an hypothetical observation which is not present at the dataframe, but this point should be provided in a long format dataframe. In this case, the obs_id_col, alt_id_col and choice_col must be created by the user manually. To illustrate this concept, the following code computes the WTP for an observation for which the value of each variable is the mean value of that variable in the Mode dataframe.


```python
# Define the WTP_points columns. 
WTP_point_columns = [obs_id_column, custom_alt_id, choice_column, 'cost', 'time']
WTP_point = pd.DataFrame(columns=WTP_point_columns)

# Append to WTP_points dataframe the mean value for each dataframe variable
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 1, # Fixed to 1 in the first alternative. Necessary to create a long type dataframe
                               custom_alt_id: 1, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.car'].mean(),
                               'time': mode_choice['time.car'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 2, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.carpool'].mean(),
                               'time': mode_choice['time.carpool'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 3, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.bus'].mean(),
                               'time': mode_choice['time.bus'].mean(),
                               }, ignore_index=True)
WTP_point = WTP_point.append({ obs_id_column: 1, # Fixed to 1, necessary to create a long type dataframe
                               choice_column: 0, # Fixed to 0 in the rest of alternatives
                               custom_alt_id: 4, # It indicates the alternative for which the data is stored
                               'cost': mode_choice['cost.rail'].mean(),
                               'time': mode_choice['time.rail'].mean(),
                               }, ignore_index=True)

# Show the generated dataframe
WTP_point
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>custom_id</th>
      <th>mode_id</th>
      <th>choice</th>
      <th>cost</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>4.873487</td>
      <td>37.044117</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1.0</td>
      <td>2.0</td>
      <td>0.0</td>
      <td>1.686348</td>
      <td>39.771005</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1.0</td>
      <td>3.0</td>
      <td>0.0</td>
      <td>2.035669</td>
      <td>39.923409</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1.0</td>
      <td>4.0</td>
      <td>0.0</td>
      <td>2.211683</td>
      <td>39.504507</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Comuthe the VOT for a KLR model, i.e. the WTP using cost an time variables.
VOT = -60 * pkl.calculate_WTP_kernel(WTP_cost_column = 'cost', 
                                    WTP_x_column = 'time', 
                                    alt = 1, 
                                    model = kernel_model, 
                                    variables = variables, 
                                    WTP_point = WTP_point, 
                                    Z = train_set_scaled, 
                                    kernel_type = "RBF", 
                                    scaler = scaler,
                                    scaled_fetures=['cost', 'time'])

VOT
```




    8.133675662298685



## About this Jupyter notebook

This Jupyter notebook was developed by José Ángel Martín Baos on September 2019 to demonstrate the key functionalities of the PyKernelLogit Python package (https://pypi.org/project/pykernellogit/). This package is based on the PyLogit Python package (https://pypi.org/project/pylogit/) developed by Timothy Brathwaite (Timothy Brathwaite and Joan L. Walker. Asymmetric, closed-form, finite-parameter models of multinomial choice. Journal of Choice Modelling. Volume 29, December 2018.). This notebook is based on the example notebook for the PyLogit package.

* José Ángel Martín Baos (https://joseangelmartinb.github.io/). Faculty of Computer Science. University of Castilla-La Mancha. 


b24dce44e631bd7dd0a523238bb5aec6b73f5e09
ipynb
Jupyter Notebook
examples/notebooks-kernel/Main PyKernelLogit Example.ipynb
JoseAngelMartinB/PyKernelLogit
81316291249e2caf3801b4f31761b6cf71284f54
BSD-3-Clause
2020-06-01T07:47:12.000Z
2020-06-01T07:47:12.000Z
examples/notebooks-kernel/Main PyKernelLogit Example.ipynb
JoseAngelMartinB/PyKernelLogit
81316291249e2caf3801b4f31761b6cf71284f54
BSD-3-Clause
examples/notebooks-kernel/Main PyKernelLogit Example.ipynb
JoseAngelMartinB/PyKernelLogit
81316291249e2caf3801b4f31761b6cf71284f54
BSD-3-Clause
Qwen/Qwen-72B
1. YES
2. YES