# Synthetic data generation

We try to build a potential data scenario for the application of contextual multi-armed bandit strategies.

The scenario should be characterized by
1. **a context** (i.e. a collection of attributes describing the situation where a specific action is proposed). In this case the context is characterised by a collection of customer features, which could be either continuos, ordinal or categorical.
2. **a series of actions** to propose to the customer (e.g. no offer, mild offer and deep offer) given the context.

The dataset we build should represent a situation where we know in advance which are the actions that have been proposed to each customer and the value of the corresponding rewards. This would allow us to test bench any kind of contextual multi armed bandit solution via the execution of controlled experiments.

The notebook has the following structure

1. [Loading modules](#Loading-modules)
2. [Synthetic data generation](#Synthetic-data-generation)
3. [Input space inspection](#Input-space-inspection)
    * [Continuos features](#Continuos-features)
    * [Ordinal features](#Ordinal-features)
    * [Categorical features](#Categorical-features)
4. [Prediction power vs synthetic noise](#Prediction-power-vs-synthetic-noise)


## Loading modules


```python
import os
import sys
sys.path.append('/Users/fci02/Documents/GitHub/decisioning-analysis/test_and_learn/production_modules/')

from contextual_mab.data_utils.data_generation  import generate_synthetic_data
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from ccp_analysis.ccp_tree_analysis import compute_auc_metrics

```

## Synthetic data generation

We generate a synthetic dataset with the following shape

1. $3$ available actions
2. $n = 1000$ data points per action played
3. the **global** expected rewards of the actions are $[5\%,3\%,1\%]$
4. The features characterizing the context are
    * 3 continuos, normally distributed with mean vector $[0,10,-3]$ and standard deviations $[1,5.,.5]$
    * 2 ordinal features, with specified frequencies
        \begin{align}
        \text{O_1} &= [1,2,3,4,5,6,7,8,9,10] \quad p = [0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1]\\
        \text{O_2} &= [1,2,3] \quad p = [0.3,0.4,0.3]
        \end{align}
    * 3 categorical features, with specified frequencies
        \begin{align}
        \text{C_1} &= (\text{Yes,No}) \quad p = [0.4,.61]\\
        \text{C_2} &= (\text{Green,Yellow}) \quad p = [0.2,0.8]\\
        \text{C_3} &= (\text{A,B,C}) \quad p = [0.4,0.1,0.5]
        \end{align}
5. For each action $a$, the reward outcome are computed in the following way:
    * a random vector of weights is generated ${\bf w}_{a} \sim \mathcal{N}({\bf 0},{\bf 1})$
    * for a context matrix $X_a$, a probabilities vector is computed via sigmoid function
        \begin{align}
        {\bf z} = {\bf w}^T_a X_a+{\bf \epsilon}\\
        {\bf p} = \frac{1}{1+e^{-{\bf z}}}
        \end{align}
       where $\epsilon$ is a noise term 
       \begin{equation}
       {\bf \epsilon} \sim \mathcal{N}(0, \alpha)
       \end{equation}
       and $\alpha$ is provided by the user. For each row $i$ of $X_a$, a reward value is then assigned using the following criterion
       \begin{align}
       r_i &= 1 \quad \text{if } p_i > (1-\bar{r}_a)\text{-quantile}\\
       r_i &= 0 \quad \text{if } p_i <= (1-\bar{r}_a)\text{-quantile}
       \end{align}
      where $\bar{r}_a$ is provided by the user. In this way we ensure the global expected reward of each action reflects the input value provided by the user.
      For example if $\bar{r}_a=0.1$ then the rule assigns a non zero reward only to those rows whith probabilities belonging to the last decile. 


```python
n = 10000
nc = 3
cw = [.05,.03,.01]

ct_prm = {'X1':{'loc':0.,'scale':1.},
          'X2':{'loc':10.,'scale':5.},
          'X3':{'loc':-.3,'scale':.5}}

ord_prm = {'O1':{'start':1,'stop':11,'weights':[.1]*10},
           'O2':{'start':1,'stop':4,'weights':[.3,.4,.3]}}

catg_prm = {'C1':{'levels':['Yes','No'],'weights':[.4,.6]},
            'C2':{'levels':['Green','Yellow'],'weights':[.2,.8]},
            'C3':{'levels':['A','B','C'],'weights':[.4,.1,.5]}}
seed = 0
noise_scale=5.
```


```python
synthetic_data = generate_synthetic_data(n,nc,cw,
                       ct_prm,ord_prm,catg_prm,seed,noise_scale,
                       return_inputs=False,output_info=True)
```

    Total number of data: 30000
    Actions played: [1, 2, 3]
    Global expected rewards per action



<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>reward</th>
    </tr>
    <tr>
      <th>action_code</th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>0.05</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.03</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.01</td>
    </tr>
  </tbody>
</table>
</div>



```python
synthetic_data.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>X1</th>
      <th>X2</th>
      <th>X3</th>
      <th>O1</th>
      <th>O2</th>
      <th>C1</th>
      <th>C2</th>
      <th>C3</th>
      <th>reward</th>
      <th>action_code</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.764052</td>
      <td>11.856161</td>
      <td>-0.838830</td>
      <td>8</td>
      <td>1</td>
      <td>Yes</td>
      <td>Green</td>
      <td>C</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.400157</td>
      <td>11.523919</td>
      <td>-0.721146</td>
      <td>6</td>
      <td>2</td>
      <td>No</td>
      <td>Green</td>
      <td>A</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.978738</td>
      <td>12.520623</td>
      <td>-0.428904</td>
      <td>1</td>
      <td>3</td>
      <td>Yes</td>
      <td>Yellow</td>
      <td>C</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2.240893</td>
      <td>10.676498</td>
      <td>0.056038</td>
      <td>5</td>
      <td>2</td>
      <td>Yes</td>
      <td>Yellow</td>
      <td>C</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1.867558</td>
      <td>13.268794</td>
      <td>-0.027104</td>
      <td>7</td>
      <td>3</td>
      <td>No</td>
      <td>Yellow</td>
      <td>C</td>
      <td>0</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>



## Input space inspection

### Continuos features


```python
fig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(24,8))

for feature, ax in zip(['X1','X2','X3'],[ax1,ax2,ax3]):
    sns.distplot(synthetic_data[feature],ax=ax,label='mean: {0:.3f}\nstd: {1:.3f}'.format(synthetic_data[feature].mean(),synthetic_data[feature].std()))
    ax.set_title('Histogram of {}'.format(feature))
    ax.legend()
plt.tight_layout()
plt.show()
```

### Ordinal features


```python
fig, (ax1,ax2) = plt.subplots(1,2,figsize=(24,8))

for feature, ax in zip(['O1','O2'],[ax1,ax2]):
    sns.countplot(synthetic_data[feature],palette="rocket",ax=ax,)
    ax.set_title('Histogram of {}'.format(feature))
    ncount = len(synthetic_data[feature])
    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), 
            ha='center', va='bottom',fontsize=14)
plt.tight_layout()
plt.show()
```

### Categorical features


```python
fig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(24,8))

for feature, ax in zip(['C1','C2','C3'],[ax1,ax2,ax3]):
    sns.countplot(synthetic_data[feature],palette="rocket",ax=ax,)
    ax.set_title('Histogram of {}'.format(feature))
    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), 
            ha='center', va='bottom',fontsize=14)
plt.tight_layout()
plt.show()
```

## Prediction power vs synthetic noise

We can define more or less difficult to learn reward distributions by tuning the value of the random noise term $\epsilon$. This would help us in mimicking more realistic scenarios.


```python
noise_scales = [.1,1.,5,10.,15]
fig, (ax1, ax2, ax3) = plt.subplots(1,3,figsize=(24,8))

for action, ax in zip(range(1,4),[ax1, ax2, ax3]):
    noise_perfs = {}
    for noise_scale in noise_scales:
        synthetic_data = generate_synthetic_data(n,nc,cw,
                                                 ct_prm,ord_prm,catg_prm,seed,noise_scale)
        subset = synthetic_data[synthetic_data['action_code']==action]
        X = pd.get_dummies(subset.drop(columns=['reward','action_code']),drop_first=True).values
        y = subset['reward'].values
        X_train, X_test, y_train, y_test =train_test_split(X,y,test_size=.3,stratify=y)
        model = LogisticRegression(solver='lbfgs')
        model.fit(X_train,y_train)
        noise_perfs[str(noise_scale)] = compute_auc_metrics(model,X_train, X_test, y_train, y_test)

    data = pd.DataFrame(noise_perfs)
    w=.1
    ax.bar(np.arange(1,6),data.loc['train',:],width=w,label='train AUC',align='center')
    ax.bar(np.arange(1,6)+w,data.loc['test',:],width=w,label='test AUC',align='center')
    ax.legend()
    ax.set_title('Action {}'.format(action),fontsize=20)
    ax.grid()
    ax.set_xticks(np.arange(1,6))
    ax.set_xticklabels([str(noise_scale) for noise_scale in noise_scales])
    ax.set_xlabel('$\sigma_\epsilon$', fontsize=20)
plt.tight_layout()
plt.show()
```

It seems we can capture the variability of the reward enough well even for large value of the noise term variance. Larger values have been tried, however in this case it is necessary to use larger sample size per action in order to be able to see non zero outcomes of the reward.

725dac3e625db5661b6e4e8e3b68a4571a5b8c81
ipynb
Jupyter Notebook
examples/Synthetic_data_contextual_MAB.ipynb
Fracappo87/RL
4642bdfbdf4019e476be57c1b4cb8936b325f1bd
BSD-3-Clause
examples/Synthetic_data_contextual_MAB.ipynb
Fracappo87/RL
4642bdfbdf4019e476be57c1b4cb8936b325f1bd
BSD-3-Clause
examples/Synthetic_data_contextual_MAB.ipynb
Fracappo87/RL
4642bdfbdf4019e476be57c1b4cb8936b325f1bd
BSD-3-Clause
Qwen/Qwen-72B
1. YES
2. YES