# Labeling: Excess Return Over Median


_*Fig. 1:*_ Distribution of excess over median return for 22 stock tickers from period between Jan 2019 and May 2020.

## Abstract

In this notebook, we demonstrate labeling financial returns data according to excess over median. Using cross-sectional data on returns of many different stocks, each observation is labeled according to whether (or how much) its return exceeds the median return. Correlations can then be found between features and the likelihood that a stock will outperform the market.

## Introduction
This technique is used in the following paper:
["The benefits of tree-based models for stock selection"](https://link.springer.com/article/10.1057/jam.2012.17) by _Zhu et al._ (2012). 

In that paper, independent composite features are constructed as weighted averages of various parameters in fundamental and quantitative analysis, such as PE ratio, corporate cash flows, debt etc. The composite features are applied as parameters in linear regression or a decision tree to predict whether a stock will outperform the market median return.


## How it works

A dataframe containing forward total stock returns is calculated from close prices. The median return of all companies at time $t$ in the dataframe is used to represent the market return, and excess returns are calculated by subtracting the median return from each stock's return over the time period $t$ \[Zhu et al. 2012\]. The numerical returns over median can then be used as is (for regression analysis), or can be relabeled simply to its sign (for classification analysis).

At time $t$:

$$P_t = \{p_{t,0}, p_{t,1}, ..., p_{t,n}\}$$
$$m_t = median(P_t)$$
$$L(P_t) = \{p_{t,0} - m_t, p_{t,1} - m_t, ...,p_{t,n} - m_t\}$$

If categorical rather than numerical labels are desired:

$$
     \begin{equation}
     \begin{split}
       L(p_{t,n}) = \begin{cases}
       -1 &\ \text{if} \ \ p_{t,n} - m_t < 0\\
       0 &\ \text{if} \ \ p_{t,n} - m_t = 0\\
       1 &\ \text{if} \ \ p_{t,n} - m_t > 0\\
       \end{cases}
     \end{split}
     \end{equation}
$$

---
## Examples of use


```python
import numpy as np
import pandas as pd
import yfinance as yf

from mlfinlab.labeling import excess_over_median

import matplotlib.pyplot as plt
```


```python
# Load price data for 22 stocks
tickers = "AAPL MSFT COST PFE SYY F GE BABA AMD CCL ZM FB WFC JPM NVDA CVX AAL UBER C UA VZ NOK"

data = yf.download(tickers, start="2019-01-20", end="2020-05-25",
                   group_by="ticker")
data = data.loc[:, (slice(None), 'Adj Close')]
data.columns = data.columns.droplevel(1)
data.head()
```

    [*********************100%***********************]  22 of 22 completed





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AMD</th>
      <th>FB</th>
      <th>BABA</th>
      <th>NVDA</th>
      <th>MSFT</th>
      <th>AAL</th>
      <th>C</th>
      <th>JPM</th>
      <th>COST</th>
      <th>F</th>
      <th>...</th>
      <th>UA</th>
      <th>ZM</th>
      <th>UBER</th>
      <th>NOK</th>
      <th>VZ</th>
      <th>AAPL</th>
      <th>PFE</th>
      <th>CVX</th>
      <th>SYY</th>
      <th>CCL</th>
    </tr>
    <tr>
      <th>Date</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2019-01-22</th>
      <td>19.760000</td>
      <td>147.570007</td>
      <td>152.149994</td>
      <td>148.102676</td>
      <td>103.568062</td>
      <td>32.219025</td>
      <td>59.116608</td>
      <td>98.963676</td>
      <td>209.413116</td>
      <td>7.837517</td>
      <td>...</td>
      <td>18.590000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.846427</td>
      <td>54.096535</td>
      <td>150.266403</td>
      <td>39.946537</td>
      <td>105.162872</td>
      <td>60.661575</td>
      <td>51.750248</td>
    </tr>
    <tr>
      <th>2019-01-23</th>
      <td>19.799999</td>
      <td>144.300003</td>
      <td>152.029999</td>
      <td>148.620316</td>
      <td>104.577469</td>
      <td>31.146364</td>
      <td>59.384232</td>
      <td>98.713730</td>
      <td>209.107468</td>
      <td>7.689988</td>
      <td>...</td>
      <td>18.400000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>5.924772</td>
      <td>54.827435</td>
      <td>150.874130</td>
      <td>39.842587</td>
      <td>104.273567</td>
      <td>60.884239</td>
      <td>51.512947</td>
    </tr>
    <tr>
      <th>2019-01-24</th>
      <td>20.850000</td>
      <td>145.830002</td>
      <td>155.860001</td>
      <td>157.131958</td>
      <td>104.077667</td>
      <td>33.124386</td>
      <td>59.938599</td>
      <td>98.771400</td>
      <td>207.352509</td>
      <td>7.929724</td>
      <td>...</td>
      <td>18.860001</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>6.032495</td>
      <td>54.172474</td>
      <td>149.678253</td>
      <td>38.699097</td>
      <td>106.258125</td>
      <td>60.535717</td>
      <td>52.224846</td>
    </tr>
    <tr>
      <th>2019-01-25</th>
      <td>21.930000</td>
      <td>149.009995</td>
      <td>159.210007</td>
      <td>159.431610</td>
      <td>105.028282</td>
      <td>34.423378</td>
      <td>61.190701</td>
      <td>99.396294</td>
      <td>206.129929</td>
      <td>8.169458</td>
      <td>...</td>
      <td>19.490000</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>6.463387</td>
      <td>53.536488</td>
      <td>154.638153</td>
      <td>38.406132</td>
      <td>105.986641</td>
      <td>60.041988</td>
      <td>52.689953</td>
    </tr>
    <tr>
      <th>2019-01-28</th>
      <td>20.180000</td>
      <td>147.470001</td>
      <td>158.919998</td>
      <td>137.390930</td>
      <td>102.980049</td>
      <td>35.988075</td>
      <td>61.028221</td>
      <td>99.867378</td>
      <td>207.806046</td>
      <td>7.985046</td>
      <td>...</td>
      <td>19.370001</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>6.355664</td>
      <td>52.274014</td>
      <td>153.207047</td>
      <td>37.357147</td>
      <td>105.003731</td>
      <td>60.284012</td>
      <td>53.534740</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 22 columns</p>
</div>



We find the excess return over median for all tickers in the time period, calculate the mean and standard deviation of returns, and plot the distribution.


```python
excess1 = excess_over_median(data)
excess1.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AMD</th>
      <th>FB</th>
      <th>BABA</th>
      <th>NVDA</th>
      <th>MSFT</th>
      <th>AAL</th>
      <th>C</th>
      <th>JPM</th>
      <th>COST</th>
      <th>F</th>
      <th>...</th>
      <th>UA</th>
      <th>ZM</th>
      <th>UBER</th>
      <th>NOK</th>
      <th>VZ</th>
      <th>AAPL</th>
      <th>PFE</th>
      <th>CVX</th>
      <th>SYY</th>
      <th>CCL</th>
    </tr>
    <tr>
      <th>Date</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2019-01-22</th>
      <td>0.001406</td>
      <td>-0.022777</td>
      <td>-0.001406</td>
      <td>0.002877</td>
      <td>0.009129</td>
      <td>-0.033911</td>
      <td>0.003909</td>
      <td>-0.003143</td>
      <td>-0.002077</td>
      <td>-0.019441</td>
      <td>...</td>
      <td>-0.010838</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.012783</td>
      <td>0.012893</td>
      <td>0.003427</td>
      <td>-0.003220</td>
      <td>-0.009074</td>
      <td>0.003053</td>
      <td>-0.005203</td>
    </tr>
    <tr>
      <th>2019-01-23</th>
      <td>0.043061</td>
      <td>0.000634</td>
      <td>0.015223</td>
      <td>0.047302</td>
      <td>-0.014748</td>
      <td>0.053538</td>
      <td>-0.000634</td>
      <td>-0.009385</td>
      <td>-0.018362</td>
      <td>0.021206</td>
      <td>...</td>
      <td>0.015031</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.008213</td>
      <td>-0.021915</td>
      <td>-0.017895</td>
      <td>-0.038669</td>
      <td>0.009063</td>
      <td>-0.015693</td>
      <td>0.003851</td>
    </tr>
    <tr>
      <th>2019-01-24</th>
      <td>0.034036</td>
      <td>0.004044</td>
      <td>0.003731</td>
      <td>-0.003127</td>
      <td>-0.008629</td>
      <td>0.021453</td>
      <td>0.003127</td>
      <td>-0.011436</td>
      <td>-0.023659</td>
      <td>0.012470</td>
      <td>...</td>
      <td>0.015642</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.053666</td>
      <td>-0.029502</td>
      <td>0.015375</td>
      <td>-0.025333</td>
      <td>-0.020317</td>
      <td>-0.025918</td>
      <td>-0.008857</td>
    </tr>
    <tr>
      <th>2019-01-25</th>
      <td>-0.070535</td>
      <td>-0.001071</td>
      <td>0.007443</td>
      <td>-0.128981</td>
      <td>-0.010238</td>
      <td>0.054719</td>
      <td>0.006609</td>
      <td>0.014004</td>
      <td>0.017396</td>
      <td>-0.013309</td>
      <td>...</td>
      <td>0.003107</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>-0.007402</td>
      <td>-0.014317</td>
      <td>0.000010</td>
      <td>-0.018049</td>
      <td>-0.000010</td>
      <td>0.013295</td>
      <td>0.025297</td>
    </tr>
    <tr>
      <th>2019-01-28</th>
      <td>-0.040490</td>
      <td>-0.016647</td>
      <td>-0.007242</td>
      <td>-0.040851</td>
      <td>-0.014771</td>
      <td>-0.002062</td>
      <td>-0.004429</td>
      <td>0.008386</td>
      <td>0.003412</td>
      <td>0.017142</td>
      <td>...</td>
      <td>-0.017121</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>0.014840</td>
      <td>-0.026909</td>
      <td>-0.004770</td>
      <td>0.036963</td>
      <td>0.002564</td>
      <td>0.002062</td>
      <td>0.003112</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 22 columns</p>
</div>



We can visualize the distribution as a histogram.


```python
s2 = pd.Series(excess1.iloc[:-1, :].values.flatten())
ax2 = s2.plot.hist(bins=50)
ax2.set_xlim(-0.2,0.2)
ax2.set_xlabel('Return Over Median')
ax2.set_title('Distribution of Return Over Median for 22 Stocks')
plt.savefig('distribution_over_median.png')
```

Instead of returning the numerical value of excess return over median, we can also simply return the sign. Using categorical rather than numerical labels alleviates problems that can arise due to extreme outlier returns [Zhu et al. 2012].


```python
excess2 = excess_over_median(data, binary=True)
excess2.head()
```

    c:\users\ruifan\mlfinlab\mlfinlab\labeling\excess_over_median.py:41: RuntimeWarning: invalid value encountered in sign
      return np.sign(returns_over_median)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AMD</th>
      <th>FB</th>
      <th>BABA</th>
      <th>NVDA</th>
      <th>MSFT</th>
      <th>AAL</th>
      <th>C</th>
      <th>JPM</th>
      <th>COST</th>
      <th>F</th>
      <th>...</th>
      <th>UA</th>
      <th>ZM</th>
      <th>UBER</th>
      <th>NOK</th>
      <th>VZ</th>
      <th>AAPL</th>
      <th>PFE</th>
      <th>CVX</th>
      <th>SYY</th>
      <th>CCL</th>
    </tr>
    <tr>
      <th>Date</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2019-01-22</th>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>...</td>
      <td>-1.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
    </tr>
    <tr>
      <th>2019-01-23</th>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>...</td>
      <td>1.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>2019-01-24</th>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>...</td>
      <td>1.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
    </tr>
    <tr>
      <th>2019-01-25</th>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>...</td>
      <td>1.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>2019-01-28</th>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>...</td>
      <td>-1.0</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>1.0</td>
      <td>-1.0</td>
      <td>-1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
      <td>1.0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 22 columns</p>
</div>



We can verify that the number of positive labels matches the number of negative labels. Note: for larger data sets, there is increased chance that some tickers will have the exact same return for a given time index. If that return is also equal to the the median, the number of positive labels may not match exactly with the number of negatives, but should be very close.


```python
excess2.stack().value_counts()
```




    -1.0    3629
     1.0    3629
     0.0      19
    dtype: int64



---
## Conclusion

This notebook presents the method to label data according to excess return over median. This method can return either numerical or categorical labels for observations. Zhu et al. utilize these labels to predict monthly stock returns using linear regression and decision trees based on composite features as independent variables. In this process:
 - Forward rates of return for assets are calculated for the entire selection of stocks indexed by time bars.
 - At each time index, the median rate of return for all stocks is calculated. The median is subtracted from each stock's return to find the excess return over median.
 - If categorical labels are desired, the excess returns are converted to their signs.

This method is useful for labelling data used for training regression models and decision trees. Zhu et al. found that decision trees were slightly better at predicting outperformers than linear regression.

## References

1. Zhu, M., Philpotts, D. and Stevenson, M., 2012. The benefits of tree-based models for stock selection. Journal of Asset Management, [online] 13(6), pp.437-448. Available at: <https://link.springer.com/article/10.1057/jam.2012.17>.

0fbbc8c7148759b1eeb7bcc6fbaacdb0e42a8316
ipynb
Jupyter Notebook
jupyter-notebooks/hudson-and-thames-quant/Labelling/Labels Excess Over Median/excess_over_median.ipynb
BlackSwine/compendium
8ca631d79605c4e34ef2cec1dc5a469ab7639623
BSD-3-Clause-Clear Unlicense
jupyter-notebooks/hudson-and-thames-quant/Labelling/Labels Excess Over Median/excess_over_median.ipynb
BlackSwine/compendium
8ca631d79605c4e34ef2cec1dc5a469ab7639623
BSD-3-Clause-Clear Unlicense
2020-12-20T18:39:44.000Z
2020-12-20T18:39:44.000Z
Labelling/Labels Excess Over Median/excess_over_median.ipynb
tzw101/research
522cf53da9ba42b0b8fd79a8760ee0c39e6e2715
MIT
2020-12-31T15:07:10.000Z
2020-12-31T15:07:10.000Z
Qwen/Qwen-72B
1. YES
2. YES