# Tutorial: optimal piecewise binning with continuous target

## Basic

To get us started, let's load a well-known dataset from the UCI repository and transform the data into a ``pandas.DataFrame``.


```python
import pandas as pd
from sklearn.datasets import load_boston
```


```python
data = load_boston()
df = pd.DataFrame(data.data, columns=data.feature_names)
```

We choose a variable to discretize and the continuous target.


```python
variable = "LSTAT"
x = df[variable].values
y = data.target
```

Import and instantiate an ``ContinuousOptimalPWBinning`` object class and we pass the variable name. The ``ContinuousOptimalPWBinning`` can **ONLY** handle numerical variables. This differs from the ``ContinuousOptimalBinning`` object class.


```python
from optbinning import ContinuousOptimalPWBinning
```


```python
optb = ContinuousOptimalPWBinning(name=variable)
```

We fit the optimal binning object with arrays ``x`` and ``y``.


```python
optb.fit(x, y)
```




    ContinuousOptimalPWBinning(name='LSTAT')



You can check if an optimal solution has been found via the ``status`` attribute:


```python
optb.status
```




    'OPTIMAL'



You can also retrieve the optimal split points via the ``splits`` attribute:


```python
optb.splits
```




    array([ 4.6500001 ,  5.49499989,  6.86500001,  9.7249999 , 11.67499971,
           13.0999999 , 16.08500004, 19.89999962, 23.31500053])



#### The binning table

The optimal binning algorithms return a binning table; a binning table displays the binned data and several metrics for each bin. Class ``ContinuousOptimalPWBinning`` returns an object ``PWContinuousBinningTable`` via the ``binning_table`` attribute.


```python
binning_table = optb.binning_table
```


```python
type(binning_table)
```




    optbinning.binning.piecewise.binning_statistics.PWContinuousBinningTable



The `binning_table` is instantiated, but not built. Therefore, the first step is to call the method `build`, which returns a ``pandas.DataFrame``.


```python
binning_table.build()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Bin</th>
      <th>Count</th>
      <th>Count (%)</th>
      <th>Sum</th>
      <th>Std</th>
      <th>Min</th>
      <th>Max</th>
      <th>Zeros count</th>
      <th>c0</th>
      <th>c1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>(-inf, 4.65)</td>
      <td>50</td>
      <td>0.098814</td>
      <td>1985.9</td>
      <td>8.198651</td>
      <td>22.8</td>
      <td>50.0</td>
      <td>0</td>
      <td>56.750078</td>
      <td>-4.823690</td>
    </tr>
    <tr>
      <th>1</th>
      <td>[4.65, 5.49)</td>
      <td>28</td>
      <td>0.055336</td>
      <td>853.2</td>
      <td>6.123541</td>
      <td>21.9</td>
      <td>50.0</td>
      <td>0</td>
      <td>74.546279</td>
      <td>-8.650830</td>
    </tr>
    <tr>
      <th>2</th>
      <td>[5.49, 6.87)</td>
      <td>45</td>
      <td>0.088933</td>
      <td>1188.6</td>
      <td>5.136259</td>
      <td>20.6</td>
      <td>48.8</td>
      <td>0</td>
      <td>27.338265</td>
      <td>-0.059744</td>
    </tr>
    <tr>
      <th>3</th>
      <td>[6.87, 9.72)</td>
      <td>89</td>
      <td>0.175889</td>
      <td>2274.9</td>
      <td>6.845250</td>
      <td>11.9</td>
      <td>50.0</td>
      <td>0</td>
      <td>35.670910</td>
      <td>-1.273531</td>
    </tr>
    <tr>
      <th>4</th>
      <td>[9.72, 11.67)</td>
      <td>49</td>
      <td>0.096838</td>
      <td>1057.9</td>
      <td>2.994842</td>
      <td>15.0</td>
      <td>31.0</td>
      <td>0</td>
      <td>36.084157</td>
      <td>-1.316024</td>
    </tr>
    <tr>
      <th>5</th>
      <td>[11.67, 13.10)</td>
      <td>35</td>
      <td>0.069170</td>
      <td>697.5</td>
      <td>2.592139</td>
      <td>14.5</td>
      <td>27.9</td>
      <td>0</td>
      <td>20.719580</td>
      <td>-0.000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>[13.10, 16.09)</td>
      <td>66</td>
      <td>0.130435</td>
      <td>1289.9</td>
      <td>3.541705</td>
      <td>10.2</td>
      <td>30.7</td>
      <td>0</td>
      <td>36.325829</td>
      <td>-1.191317</td>
    </tr>
    <tr>
      <th>7</th>
      <td>[16.09, 19.90)</td>
      <td>69</td>
      <td>0.136364</td>
      <td>1129.3</td>
      <td>3.607949</td>
      <td>8.3</td>
      <td>27.5</td>
      <td>0</td>
      <td>25.801457</td>
      <td>-0.537020</td>
    </tr>
    <tr>
      <th>8</th>
      <td>[19.90, 23.32)</td>
      <td>28</td>
      <td>0.055336</td>
      <td>368.4</td>
      <td>3.912839</td>
      <td>5.0</td>
      <td>21.7</td>
      <td>0</td>
      <td>32.669942</td>
      <td>-0.882170</td>
    </tr>
    <tr>
      <th>9</th>
      <td>[23.32, inf)</td>
      <td>47</td>
      <td>0.092885</td>
      <td>556.0</td>
      <td>4.006586</td>
      <td>5.0</td>
      <td>23.7</td>
      <td>0</td>
      <td>13.271245</td>
      <td>-0.050143</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Special</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Missing</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>Totals</th>
      <td></td>
      <td>506</td>
      <td>1.000000</td>
      <td>11401.6</td>
      <td></td>
      <td>5.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>-</td>
      <td>-</td>
    </tr>
  </tbody>
</table>
</div>



Let's describe the columns of this binning table:

- Bin: the intervals delimited by the optimal split points.
- Count: the number of records for each bin.
- Count (%): the percentage of records for each bin.
- Sum: the target sum for each bin.
- Std: the target std for each bin.
- Min: the target min value for each bin.
- Max: the target max value for each bin.
- Zeros count: the number of zeros for each bin.
- $c_0$: the first coefficient of the prediction polynomial.
- $c_1$: the second coefficient of the prediction polynomial.

The  for bin $i$ is defined as $P_i = c_0 + c_1 x_i$, where $x_i \in \text{Bin}_{i}$. In general,
\begin{equation}
P_i = \sum_{j=0}^d c_j x_i^j,
\end{equation}
where $d$ is the degree of the event rate polynomial.

The last row shows the total number of records, sum and mean.

You can use the method ``plot`` to visualize the histogram and mean curve.


```python
binning_table.plot()
```

##### Mean transformation

Now that we have checked the binned data, we can transform our original data into mean values.


```python
x_transform_mean = optb.transform(x)
```

## Advanced

Many of the advanced options have been covered in the previous tutorials with a binary target. **Check it out!** In this section, we focus on the mean monotonicity trend and the mean difference between bins.

#### Binning table statistical analysis

The ``analysis`` method performs a statistical analysis of the binning table, computing the Information Value (IV) and Herfindahl-Hirschman Index (HHI). Additionally, several statistical significance tests between consecutive bins of the contingency table are performed using the Student's t-test. The piecewise binning also includes several performance metrics for regression problems.


```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------
    
      General metrics
    
        Mean absolute error              3.68295105
        Mean squared error              26.03331040
        Median absolute error            2.72954866
        Explained variance               0.69161991
        R^2                              0.69161991
        MPE                             -0.05213932
        MAPE                             0.17801859
        SMAPE                            0.08405929
        MdAPE                            0.13009259
        SMdAPE                           0.06660004
        HHI                              0.11313253
        HHI (normalized)                 0.03250821
        Quality score                    0.21464367
    
      Significance tests
    
        Bin A  Bin B  t-statistic      p-value
            0      1     5.644492 3.313748e-07
            1      2     2.924528 5.175586e-03
            2      3     0.808313 4.206096e-01
            3      4     4.714124 6.127108e-06
            4      5     2.712699 8.189719e-03
            5      6     0.622294 5.353397e-01
            6      7     5.162973 8.663417e-07
            7      8     3.742511 4.988278e-04
            8      9     1.408305 1.643801e-01
    


#### Mean monotonicity

The monotonic_trend option permits forcing a monotonic trend to the mean curve. The default setting “auto” should be the preferred option, however, some business constraints might require to impose different trends. The default setting “auto” chooses the monotonic trend most likely to minimize the L1-norm from the options “ascending”, “descending”, “peak” and “valley” using a machine-learning-based classifier.


```python
variable = "INDUS"
x = df[variable].values
y = data.target
```


```python
optb = ContinuousOptimalPWBinning(name=variable, monotonic_trend="auto")
optb.fit(x, y)
```




    ContinuousOptimalPWBinning(name='INDUS')




```python
binning_table = optb.binning_table
binning_table.build()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Bin</th>
      <th>Count</th>
      <th>Count (%)</th>
      <th>Sum</th>
      <th>Std</th>
      <th>Min</th>
      <th>Max</th>
      <th>Zeros count</th>
      <th>c0</th>
      <th>c1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>(-inf, 3.35)</td>
      <td>63</td>
      <td>0.124506</td>
      <td>1994.0</td>
      <td>8.569841</td>
      <td>16.5</td>
      <td>50.0</td>
      <td>0</td>
      <td>31.428209</td>
      <td>-0.000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>[3.35, 5.04)</td>
      <td>57</td>
      <td>0.112648</td>
      <td>1615.2</td>
      <td>8.072710</td>
      <td>17.2</td>
      <td>50.0</td>
      <td>0</td>
      <td>42.928218</td>
      <td>-3.432839</td>
    </tr>
    <tr>
      <th>2</th>
      <td>[5.04, 6.66)</td>
      <td>66</td>
      <td>0.130435</td>
      <td>1723.7</td>
      <td>7.879078</td>
      <td>16.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>25.626711</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>[6.66, 8.01)</td>
      <td>31</td>
      <td>0.061265</td>
      <td>692.0</td>
      <td>4.604886</td>
      <td>14.4</td>
      <td>35.2</td>
      <td>0</td>
      <td>52.694010</td>
      <td>-4.064159</td>
    </tr>
    <tr>
      <th>4</th>
      <td>[8.01, 16.57)</td>
      <td>100</td>
      <td>0.197628</td>
      <td>2045.5</td>
      <td>3.547348</td>
      <td>11.9</td>
      <td>28.7</td>
      <td>0</td>
      <td>21.858675</td>
      <td>-0.212150</td>
    </tr>
    <tr>
      <th>5</th>
      <td>[16.57, 18.84)</td>
      <td>132</td>
      <td>0.260870</td>
      <td>2165.3</td>
      <td>8.507336</td>
      <td>5.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>21.603058</td>
      <td>-0.196723</td>
    </tr>
    <tr>
      <th>6</th>
      <td>[18.84, inf)</td>
      <td>57</td>
      <td>0.112648</td>
      <td>1165.9</td>
      <td>9.519086</td>
      <td>7.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>17.896791</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Special</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Missing</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>Totals</th>
      <td></td>
      <td>506</td>
      <td>1.000000</td>
      <td>11401.6</td>
      <td></td>
      <td>5.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>-</td>
      <td>-</td>
    </tr>
  </tbody>
</table>
</div>




```python
binning_table.plot()
```


```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------
    
      General metrics
    
        Mean absolute error              5.38628982
        Mean squared error              59.01564719
        Median absolute error            4.02296554
        Explained variance               0.30092446
        R^2                              0.30092446
        MPE                             -0.12086396
        MAPE                             0.27859705
        SMAPE                            0.12156690
        MdAPE                            0.18811572
        SMdAPE                           0.09658000
        HHI                              0.16875752
        HHI (normalized)                 0.06485221
        Quality score                    0.74831364
    
      Significance tests
    
        Bin A  Bin B  t-statistic  p-value
            0      1     2.180865 0.031181
            1      2     1.537968 0.126745
            2      3     2.976660 0.003739
            3      4     2.075258 0.044180
            4      5     4.934158 0.000002
            5      6    -2.770230 0.006721
    


A smoother curve, keeping the valley monotonicity, can be achieved by using ``monotonic_trend="convex"``.


```python
optb = ContinuousOptimalPWBinning(name=variable, monotonic_trend="convex")
optb.fit(x, y)
```




    ContinuousOptimalPWBinning(monotonic_trend='convex', name='INDUS')




```python
binning_table = optb.binning_table
binning_table.build()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Bin</th>
      <th>Count</th>
      <th>Count (%)</th>
      <th>Sum</th>
      <th>Std</th>
      <th>Min</th>
      <th>Max</th>
      <th>Zeros count</th>
      <th>c0</th>
      <th>c1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>(-inf, 3.35)</td>
      <td>63</td>
      <td>0.124506</td>
      <td>1994.0</td>
      <td>8.569841</td>
      <td>16.5</td>
      <td>50.0</td>
      <td>0</td>
      <td>35.478615</td>
      <td>-1.760813</td>
    </tr>
    <tr>
      <th>1</th>
      <td>[3.35, 5.04)</td>
      <td>57</td>
      <td>0.112648</td>
      <td>1615.2</td>
      <td>8.072710</td>
      <td>17.2</td>
      <td>50.0</td>
      <td>0</td>
      <td>35.478615</td>
      <td>-1.760813</td>
    </tr>
    <tr>
      <th>2</th>
      <td>[5.04, 6.66)</td>
      <td>66</td>
      <td>0.130435</td>
      <td>1723.7</td>
      <td>7.879078</td>
      <td>16.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>35.478615</td>
      <td>-1.760813</td>
    </tr>
    <tr>
      <th>3</th>
      <td>[6.66, 8.01)</td>
      <td>31</td>
      <td>0.061265</td>
      <td>692.0</td>
      <td>4.604886</td>
      <td>14.4</td>
      <td>35.2</td>
      <td>0</td>
      <td>35.478615</td>
      <td>-1.760813</td>
    </tr>
    <tr>
      <th>4</th>
      <td>[8.01, 16.57)</td>
      <td>100</td>
      <td>0.197628</td>
      <td>2045.5</td>
      <td>3.547348</td>
      <td>11.9</td>
      <td>28.7</td>
      <td>0</td>
      <td>24.492313</td>
      <td>-0.388383</td>
    </tr>
    <tr>
      <th>5</th>
      <td>[16.57, 18.84)</td>
      <td>132</td>
      <td>0.260870</td>
      <td>2165.3</td>
      <td>8.507336</td>
      <td>5.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>18.898426</td>
      <td>-0.050792</td>
    </tr>
    <tr>
      <th>6</th>
      <td>[18.84, inf)</td>
      <td>57</td>
      <td>0.112648</td>
      <td>1165.9</td>
      <td>9.519086</td>
      <td>7.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>18.898426</td>
      <td>-0.050792</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Special</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Missing</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>Totals</th>
      <td></td>
      <td>506</td>
      <td>1.000000</td>
      <td>11401.6</td>
      <td></td>
      <td>5.0</td>
      <td>50.0</td>
      <td>0</td>
      <td>-</td>
      <td>-</td>
    </tr>
  </tbody>
</table>
</div>




```python
binning_table.plot()
```


```python
binning_table.analysis()
```

    -------------------------------------------------
    OptimalBinning: Continuous Binning Table Analysis
    -------------------------------------------------
    
      General metrics
    
        Mean absolute error              5.46871836
        Mean squared error              60.47066184
        Median absolute error            4.02086478
        Explained variance               0.28368894
        R^2                              0.28368894
        MPE                             -0.12278289
        MAPE                             0.28138163
        SMAPE                            0.12314489
        MdAPE                            0.19095659
        SMdAPE                           0.09398799
        HHI                              0.16875752
        HHI (normalized)                 0.06485221
        Quality score                    0.74831364
    
      Significance tests
    
        Bin A  Bin B  t-statistic  p-value
            0      1     2.180865 0.031181
            1      2     1.537968 0.126745
            2      3     2.976660 0.003739
            3      4     2.075258 0.044180
            4      5     4.934158 0.000002
            5      6    -2.770230 0.006721
    


9dd9ea6976328fb2d665db7f8d0b616e6a37ea3d
ipynb
Jupyter Notebook
doc/source/tutorials/tutorial_piecewise_continuous.ipynb
jensgk/optbinning
5ccd892fa4ee0a745ab539cee10a2069b35de6da
Apache-2.0
2020-01-23T21:32:59.000Z
2022-03-30T06:33:21.000Z
doc/source/tutorials/tutorial_piecewise_continuous.ipynb
jensgk/optbinning
5ccd892fa4ee0a745ab539cee10a2069b35de6da
Apache-2.0
2020-01-23T22:14:35.000Z
2022-03-29T14:05:04.000Z
doc/source/tutorials/tutorial_piecewise_continuous.ipynb
jensgk/optbinning
5ccd892fa4ee0a745ab539cee10a2069b35de6da
Apache-2.0
2020-01-27T15:37:08.000Z
2022-03-30T06:33:25.000Z
Qwen/Qwen-72B
1. YES
2. YES