# 第十一讲：机器学习应用建议

## 5. 机器学习应用的一些建议

本节主要是[演示文稿](http://cs229.stanford.edu/materials/ML-advice.pdf)，有以下几点需要注意：

* 介绍如何将机器学习算法应用在不同的应用问题上的一些建议；
* 本节的材料并不涉及很多数学运算，但仍旧是本课程最难理解的一部分内容；
* 本节的内容在今天可能仍有争议；
* 本节的一些方法并不有利于新的学习算法的研发，只是有利于现有算法的应用；
* 本节的三个重点：
    1. 学习算法的调试诊断；
    2. 误差分析及销蚀分析；
    3. 如何开始用机器学习解决问题
        * 过早统计优化（premature statistical optimization，与软件工程中的过早优化类似，指对机器学习系统中一个非常不重要的部分进行过度的优化）

### 5.1 调试学习算法

举个例子，比如在写垃圾邮件分类器时，我们小心的选择了$100$个单词作为特征值。（而不是使用全部$50000+$的英语单词）。然后使用贝叶斯逻辑回归算法$\displaystyle\operatorname*{max}_\theta\sum_{i=1}^m\log p\left(y^{(i)}\mid x^{(i)},\theta\right)-\lambda\lVert\theta\rVert^2$，再使用梯度下降法算出结果。测试模型发现，得到了不可接受的20%的错误率。

接下来怎么办？

通常的想法是想方设法的优化这个算法，包括：
* 获得更多的训练样本；
* 寻找更小的特征值集合；
* 寻找更大的特征值集合；
* 尝试改变特征值，比如“从邮件头部获取信息”V.S.“从邮件正文获取信息”；
* 让算法再多迭代几次（怀疑算法没有完全收敛）；
* 尝试换用牛顿法计算参数；
* 给$\lambda$赋不同的值；
* 尝试换用SVM算法。

这里仅列出了八项，而在实际的学习系统中，能够列出上百项可行的改进方法，其中的某些尝试可能会有效（比如获取更多的训练样本），但显然非常浪费时间，而且非常需要运气，就算解决了也可能仍旧不知道问题出在哪。

所以，我们应该先诊断问题所在，然后再想方法解决。那么，对于上面的例子：

假设我们怀疑问题出现在：
* 过拟合（高方差）；
* 特征值过少，无法有效分类（高偏差）

诊断：
* 高方差的特点：训练误差比测试误差小很多；
* 高偏差的特点：训练误差也很高；

高方差的典型学习曲线：

* 当训练集$m$增大时测试误差也会继续降低（而训练误差会持续增加，因为样本越多，越难以完美拟合），建议增加训练样本；
* 训练误差与测试误差中的差距过大。

高偏差的典型学习曲线：

* 即使是训练误差也是高的不可思议（还可以发现增加训练样本的效果已经很不明显了，而此时增加训练样本只会使训练效果越来越差，因为通常的训练误差是关于训练样本的单调增函数）；
* 而训练误差与测试误差间的差距很小。

（所以，可以通过判断训练误差与测试误差间的距离判断）

于是，对于这个使用梯度下降法实现的贝叶斯逻辑回归，回到前面列出的八项解决方案：
* 获得更多的训练样本；*（有助于修复高方差）*
* 寻找更小的特征值集合；*（有助于修复高方差）*
* 寻找更大的特征值集合；*（有助于修复高偏差）*
* 尝试改变特征值，比如“从邮件头部获取信息”V.S.“从邮件正文获取信息”；*（有助于修复高偏差）*
* 让算法再多迭代几次（怀疑算法没有完全收敛）；
* 尝试换用牛顿法计算参数；
* 给$\lambda$赋不同的值；
* 尝试换用SVM算法。

上面这个关于方差与偏差的诊断是一个常见的机器学习调试方法。而对于别的问题则需要我们构造自己的诊断方法，找出问题。

我们再举一个例子：
* 使用贝叶斯逻辑回归时，对垃圾邮件存在$2%$的误判，对非垃圾邮件也存在$2%$的误判（也就是对正常邮件的误判率过高）；
* 而使用带有线性核的SVM时，对垃圾邮件存在$10%$的误判，对非垃圾邮件存在$0.01%$的误判（在正常范围内）；
* 因为运算效率的问题，我们更希望使用逻辑回归。

这次，又该怎么办？

通常，伴随这种问题会产生更常见的问题：
* 收敛性问题：我们的算法（对于逻辑回归的梯度下降法）收敛了吗？

    通过直接观察目标函数值经常难以判断算法是否已经收敛。所以，除了直接观察目标函数关于迭代次数的曲线以外，我们可能需要更好的判断收敛的方法。

* 是否找到了正确的优化函数，比如在垃圾邮件分类器的例子中，我们真正关心的是这个加权准确率函数：

$$
a(\theta)=\sum_iw^{(i)}1\left\{h_\theta(x^{(i)})=y^{(i)}\right\}
$$

    （非垃圾邮件的权值$w^{(i)}$更高，因为我们更关心对垃圾邮件的预测，更希望模型能够准确挑出垃圾邮件。）
* 那么，是使用贝叶斯逻辑回归？惩罚项系数$\lambda$设置正确吗？

$$
\operatorname*{max}_\theta J(\theta)=\sum_{i=1}^m\log p\left(y^{(i)}\mid x^{(i)},\theta\right)-\lambda\lVert\theta\rVert^2
$$

* 还是使用SVM？惩罚项系数$C$设置正确吗？

$$
\begin{align}
\operatorname*{min}_{w,b}&\quad\left\lVert w^2\right\rVert+C\sum_{i=1}^m\xi_i\\
\mathrm{s.t.}&\quad y^{(i)}\left(w^Tx^{(i)}-b\right)\geq1-\xi_i
\end{align}
$$

我们从这个例子引出第二种诊断方法，判断究竟是算法没有收敛，还是在于一开始目标函数的选择上。假设尽管SVM比贝叶斯逻辑回归表现更好，但是我们还是希望使用贝叶斯逻辑回归：

令$\theta_\mathrm{SVM}$为由SVM求出的参数；令$\theta_\mathrm{BLR}$为贝叶斯逻辑回归生成的参数。继续列出我们关心的加权准确率：

$$
a(\theta)=\sum_iw^{(i)}1\left\{h_\theta(x^{(i)})=y^{(i)}\right\}
$$

因为SVM的表现更好，所以其加权准确率更高$a\left(\theta_\mathrm{SVM}\right)\gt a\left(\theta_\mathrm{BLR}\right)$。因为贝叶斯逻辑回归的目标函数为$\displaystyle\operatorname*{max}_\theta J(\theta)=\sum_{i=1}^m\log p\left(y^{(i)}\mid x^{(i)},\theta\right)-\lambda\lVert\theta\rVert^2$，所以我们诊断两个算法目标函数的结果：

$$
J\left(\theta_\mathrm{SVM}\right)\stackrel{?}{\gt} J\left(\theta_\mathrm{BLR}\right)
$$

* 第一种情况：$a\left(\theta_\mathrm{SVM}\right)\gt a\left(\theta_\mathrm{BLR}\right)$，且$J\left(\theta_\mathrm{SVM}\right)\gt J\left(\theta_\mathrm{BLR}\right)$。

    因为BLR的目标是最大化$J(\theta)$，则可以看出BLR并没有完成这个目标，即$\theta_\mathrm{BLR}$并没有使$J$取到最大值，算法的收敛出了问题。也就是说，这是优化算法的问题。

* 第二种情况：$a\left(\theta_\mathrm{SVM}\right)\gt a\left(\theta_\mathrm{BLR}\right)$，且$J\left(\theta_\mathrm{SVM}\right)\leq J\left(\theta_\mathrm{BLR}\right)$。

    这意味着BLR成功的最大化了$J(\theta)$，但是SVM虽然在最大化目标函数这方面做得没有BLR好，但是SVM的加权准确率$a(\theta)$竟依旧比BLR高。说明如果我们关心$a(\theta)$，那么$J(\theta)$就不是一个恰当的目标函数。也就是说，这是目标函数选择的问题。

再回到我们列出的八项解决方案：
* 获得更多的训练样本；*（有助于修复高方差）*
* 寻找更小的特征值集合；*（有助于修复高方差）*
* 寻找更大的特征值集合；*（有助于修复高偏差）*
* 尝试改变特征值，比如“从邮件头部获取信息”V.S.“从邮件正文获取信息”；*（有助于修复高偏差）*
* 让算法再多迭代几次（怀疑算法没有完全收敛）；*(有助于修复算法收敛问题)*
* 尝试换用牛顿法计算参数；*(有助于修复算法收敛问题)*
* 给$\lambda$赋不同的值；*(有助于修复目标函数选择问题)*
* 尝试换用SVM算法。*(有助于修复目标函数选择问题)*

再来简要的介绍一个**强化学习（RL: reinforcement learning）**的例子——用学习算法控制一个直升机，分为以下三步：

1. 搭建一个遥控直升机模拟器，类似一个带操纵杆的模拟器，可以模拟飞行；
2. 选择一个成本函数，也称作奖励函数（reward function），比如$J(\theta)=\left\lVert x-x_\mathrm{desired}\right\rVert$（其中的$x$代表直升机的位置，这里示例的函数表示直升机实际位置与期望位置的平方误差）；
3. 在模拟器中运行强化学习算法控制直升机并尝试最小化成本函数，$\theta_\mathrm{RL}=\mathrm{arg}\displaystyle\operatorname*{min}_\theta J(\theta)$。

现在，假设我们做了以上三步，发现得到的控制参数$\theta_\mathrm{RL}$的表现比人类直接操作差很多，接下来该怎么办？我们能直接想到的——是升级模拟器（也行是模拟器不够精确，没有考虑到更精确的空气动力学因素，需要收集到飞机附近更精确的气流扰动）？还是修改成本函数$J$（也许平方误差并不合适，也许人类操纵者考虑的不是误差平方，而是更加复杂的运算）？或是修改RL算法（也许RL算法不够好，也许它收敛的不够迅速）？

对于表现不好的参数$\theta_\mathrm{RL}$，我们先做出以下假设：
1. 假设模拟器已经够精确了（也就是我们对直升机的建模是准确的）；
2. 假设强化学习算法能够在模拟器中正确的控制直升机，从而求得最小化的$J(\theta)$；
3. 假设最小化$J(\theta)$确实可以正确的控制直升机（即强相关的）。

如果这些假设都成立，那么我们的算法理论上就能够很好的控制直升机了。但现实是$\theta_{RL}$并不能很好的控制直升机，所以上面的假设至少有一个是不成立的。我们希望找出问题，于是用到的诊断方法如下：

1. 检查模拟器，如果算法在模拟器中可以正常工作，但是在现实中不行，那么问题可能出在模拟器上。于是，我们应该改进模拟器；
2. 令$\theta_\mathrm{human}$为人工操作时的参数（让人在模拟器中操作飞机，并记录成本函数的值，在这里就是平方误差的均值），如果$J(\theta_\mathrm{human})\lt J(\theta_\mathrm{RL})$，则是增强学习算法出了问题。（算法并没有找到使成本函数$J$最小的参数取值。）
3. 如果$J(\theta_\mathrm{human})\geq J(\theta_\mathrm{RL})$，则应是成本函数出了问题。（最小化错误的函数并不能增强算法控制飞机的效果。）

不过，通常遇到问题时，我们需要自己寻找诊断方法查找问题。

有时，即使一个算法能够正常工作，我们也可以运行诊断方法来看算法是如何运作的，比如用于：
* 让我们理解程序的问题：比如手头有一个跑了几个月甚至几年的重要学习算法，那么直观的知道算法是如何运行的，知道算法在什么时候有效，在什么时候失效，对我们个人的提高都非常有帮助；
* 写论文时：诊断及误差分析都能帮助我们深入理解问题，并为论点提供有力证据。比起“如此，算法的表现很好”而言，说“算法因为组件X才有了如此优异的表现，这里是我的论据”则来的更有意思。

于是，也引出了误差分析——一个很好的机器学习实践，可以帮助我们更好的理解误差产生的根源。

### 5.2 误差分析

许多实际的人工智能、机器学习应用会通过“流水线”的方式将不同的学习组件组装起来，比如从图片中识别人脸的应用：

那么我们应该分析：
1. 误差由哪些组件造成，每个组件造成的误差是多少？
2. 对每一个组件使用基准真值测试，看每一个组件对整个系统准确率的影响。整个流程大概是这样的：
  * 启初，我们发现面部识别系统判断的准确率是$85%$；
  * 第一步，我们手动去除图片的背景，也就是直接给系统提供已经去除背景的图片，之后发现系统的准确率提高到$91%$；
  * 第二步，我们手动的告诉系统每一张图片中面部的坐标，之后发现系统准确率提升到$95%$；
  * 以此类推，我们不停的手动告知系统，用基准真值代替每一个组件通过计算得出的值，我们会接着告诉系统眼睛、鼻子、嘴巴的具体位置……系统最终会得到$100%$的准确率。

  然后，就会得到一张表，记录着每个组件对系统准确率的影响：

  $$
  \begin{array}
  {|c|c|}
  \hline
  \color{blue}{\textrm{Component}}&\color{blue}{\textrm{Accuracy}}\\
  \hline
  \textrm{Overall system}&85\%\\
  \hline
  \textrm{Preprocess(remove background)}&85.1\%\\
  \hline
  \textrm{Face detection}&91\%\\
  \hline
  \textrm{Eyes segmentation}&95\%\\
  \hline
  \textrm{Nose segmentation}&96\%\\
  \hline
  \textrm{Mouth segmentation}&97\%\\
  \hline
  \textrm{Logistic regression}&100\%\\
  \hline
  \end{array}
  $$

从这里我们就可以看出，如果我们改进预处理步骤（移除背景），对整个系统准确率的影响仅有$0.1%$；而如果我们改进面部检测算法，则整个系统将会提高$6%$的准确率。

### 5.3 销蚀分析

通过前面的介绍，我们知道**误差分析（error analysis）**是尝试解释系统当前表现与我们期待的完美表现之间的差距；

而**销蚀分析（ablative analysis）**是尝试解释系统基准表现（可能比当前差很多）与当前表现之间的差距。

还是用前面的例子，由于我们机智的向逻辑回归中加入了很多有趣的特征值，于是得到了一个非常好用的垃圾邮件分类器，其中的特征值包括：
* 拼写检查；
* 发件人主机信息；
* 邮件头信息；
* 邮件正文分析特征；
* 邮件携带的JavaScript分析特征；
* 邮件中内嵌的``标签分析特征；
* 等等……

我们现在想知道，上面那些特征是真正起作用的？

假设使用简单的逻辑回归，不加入上面任何特征时，系统具有$94%$的准确率。那么，这些特征是如何将系统准确率从$94%$提高到$99.9%$的？

销蚀分析的步骤，就是从系统中每次去掉一个组件，记录并观察系统是如何被瓦解的（留意准确率因何降低，特别是突降）。

$$
\begin{array}
{|c|c|}
\hline
\color{blue}{\textrm{Component}}&\color{blue}{\textrm{Accuracy}}\\
\hline
\textrm{Overall system}&99.9\%\\
\hline
\textrm{Spelling correction}&99\%\\
\hline
\textrm{Sender host features}&98.9\%\\
\hline
\textrm{Email header features}&98.9\%\\
\hline
\textrm{Email text parser features}&95\%\\
\hline
\textrm{Javascript parser}&94.5\%\\
\hline
\textrm{Features from images}&94\%\\
\hline
\end{array}
$$

从列表中可以清楚的看到，邮件正文分析特征对整个系统的提升最为显著。不过，调整销蚀顺序可能也会影响准确率，我们介绍的误差分析、销蚀分析并不是一成不变的。比如有些方法会从中剔除一个组件观察记录后放回，再剔除另一个组件，周而复始，调试方法有很多，关键是在相应的时候灵活使用、发明自己的调试方法。

### 5.4 着手处理学习问题的一些通用建议

着手解决一个学习问题通常有两种实现方式：

1. 精心设计
  * 花费很长的时间精确的设计特征值、收集有效的训练样本，并设计正确的算法架构；
  * 用代码实现设计，并希望它能够正常工作。

  这种实现方式的优点是：我们会得到更好的，扩展性更强的算法。这种方式通常会得到新的、优雅的学习算法，将会为机器学习的基础研究做出贡献。
2. 边编写边修改
  * 先快速实现一个简陋的原型；
  * 运行误差分析并诊断这个实现的问题所在，然后修复问题并重复这个步骤，直到得到一个满意的解决方案。

  这种实现方式的优点是：我们通常会更快的解决问题，对快速占领市场很有帮助。

就拿上面的面部识别的系统来说，对于设计如此复杂的一个系统，我们在一开始很难知道哪些组件容易实现、哪些组件需要花时间研究等等，只有通过实现一个快速原型，多试几次才能确定那些对系统影响显著的组件并给予深入研究。不过，这个建议并不适合研发新的学习算法，对于新的算法，就需要从一开始深入研究并精心设计系统的每一个步骤，而不是快速实现原型后不停的修补模型。

需要注意的是，同软件工程中的过早优化类似，我们要避免在系统设计阶段发生过早统计优化，比如上面面部识别例子中，如果我们在设计时过早的花费很多精力优化预处理模块，则在后面就会发现，这一步对整个系统影响甚微。

另外，我们不应该在理论上做过度的联想，比如我们想要实现一个自动投递邮件的机器人，那么会需要障碍躲避的知识和机器人操控的知识，如果要实现障碍躲避就需要实现障碍侦测和导航，如果要实现障碍侦测则需要避免不同时间光线对物体的光照色彩的影响，所以需要一套独立于色彩的物体识别系统，而为了实现这个系统，我们需要研究三维物体表面的微分几何，因为这可以帮助我们确定两个不同颜色的物体可能是同一个物体，为了理解上面的这些理论，我们需要学习非黎曼几何的复杂性……等等等等等等……

不过问题是，这些知识并不是真正强相关的，其中的某些箭头可能只是谋篇论文道听途说的，我们应该专注于解决眼前的工程问题，并且只相信我们能够亲自确定的存在关联的知识点，只有亲自确定后才选择是否深入研究某个特定领域的知识。

### 5.5 总结

* 通常，我们花费在调试及诊断学习算法上的时间都是值得的；
* 通常，我们需要发挥自己的聪明才智，去设计调试特定问题的方法；（对于一个机器学习问题，我们有时会花费三分之一、甚至一半的时间在调试及诊断上。）
* 误差分析及销蚀分析可以让我们深入的理解问题，看到究竟是那些因素影响算法的表现；
* 介绍了两种实现方式：
  * 精心设计后实现（可能伴有过早统计优化的风险）；
  * 快速实现原型后不断进行诊断、调试、修改；


# Stock market fluctuations

The fluctuations of stock prices represent an intriguing example of a complex random walk. Stock prices are influenced by transactions that are carried out over a broad range of time scales, from micro- to milliseconds for high-frequency hedge funds over several hours or days for day-traders to months or years for long-term investors. We therefore expect that the statistical properties of stock price fluctuations, like volatility and auto-correlation of returns, are not constant over time, but show significant fluctuations of their own. Time-varying parameter models can account for such changes in volatility and auto-correlation and update their parameter estimates in real-time.

There are, however, certain events that render previously gathered information about volatility or autocorrelation of returns completely useless. News announcements that are unexpected at least to some extent, for example, can induce increased trading activity, as market participants update their orders according to their interpretation of novel information. The resulting *"non-scheduled"* price corrections can often not be adequately described by the current estimates of volatility. Even a model that accounts for gradual variations in volatility cannot reproduce these large price corrections. Instead, when such an event happens, it becomes favorable to forget about previously gathered parameter estimates and completely start over. In this example, we use the *bayesloop* framework to specifically evaluate the probability of previously acquired information becoming useless. We interpret this probability value as a risk metric and evaluate it for each minute of an individual trading day (Nov 28, 2016) for the exchange-traded fund [SPY](https://www.google.com/finance?q=SPY). The announcement of macroeconomic indicators on this day results in a significant increase of our risk metric in intra-day trading.

**DISCLAIMER:** This website does not provide tax, legal or accounting advice. This material has been prepared for informational purposes only, and is not intended to provide, and should not be relied on for, tax, legal or accounting advice. You should consult your own tax, legal and accounting advisors before engaging in any transaction.

**Note:** The intraday pricing data used in this example is obtained via [Google Finance](https://www.google.com/finance):

```
https://www.google.com/finance/getprices?q=SPY&i=60&p=1d&f=d,c
```

This request returns a list of minute close prices (and date/time information; `f=d,c`) for SPY for the last trading period. The maximum look-back period is 14 days (`p=14d`) for requests of minute-scale data (`i=60`).

```python
%matplotlib inline
import numpy as np
import bayesloop as bl
import sympy.stats as stats
from tqdm import tqdm_notebook
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_color_codes()  # use seaborn colors

# minute-scale pricing data
prices = np.array(
     [ 221.14 ,  221.09 ,  221.17 ,  221.3  ,  221.3  ,  221.26 ,
       221.32 ,  221.17 ,  221.2  ,  221.27 ,  221.19 ,  221.12 ,
       221.08 ,  221.1  ,  221.075,  221.03 ,  221.04 ,  221.03 ,
       221.11 ,  221.14 ,  221.135,  221.13 ,  221.04 ,  221.15 ,
       221.21 ,  221.25 ,  221.21 ,  221.17 ,  221.21 ,  221.2  ,
       221.21 ,  221.17 ,  221.1  ,  221.13 ,  221.18 ,  221.15 ,
       221.2  ,  221.2  ,  221.23 ,  221.25 ,  221.25 ,  221.25 ,
       221.25 ,  221.22 ,  221.2  ,  221.15 ,  221.18 ,  221.13 ,
       221.1  ,  221.08 ,  221.13 ,  221.09 ,  221.08 ,  221.07 ,
       221.09 ,  221.1  ,  221.06 ,  221.1  ,  221.11 ,  221.18 ,
       221.26 ,  221.46 ,  221.38 ,  221.35 ,  221.3  ,  221.18 ,
       221.18 ,  221.18 ,  221.17 ,  221.175,  221.13 ,  221.03 ,
       220.99 ,  220.97 ,  220.9  ,  220.885,  220.9  ,  220.91 ,
       220.94 ,  220.935,  220.84 ,  220.86 ,  220.89 ,  220.91 ,
       220.89 ,  220.84 ,  220.83 ,  220.74 ,  220.755,  220.72 ,
       220.69 ,  220.72 ,  220.79 ,  220.79 ,  220.81 ,  220.82 ,
       220.8  ,  220.74 ,  220.75 ,  220.73 ,  220.69 ,  220.72 ,
       220.73 ,  220.69 ,  220.71 ,  220.72 ,  220.8  ,  220.81 ,
       220.79 ,  220.8  ,  220.79 ,  220.74 ,  220.77 ,  220.79 ,
       220.87 ,  220.86 ,  220.92 ,  220.92 ,  220.88 ,  220.87 ,
       220.88 ,  220.87 ,  220.94 ,  220.93 ,  220.92 ,  220.94 ,
       220.94 ,  220.9  ,  220.94 ,  220.9  ,  220.91 ,  220.85 ,
       220.85 ,  220.83 ,  220.85 ,  220.84 ,  220.87 ,  220.91 ,
       220.85 ,  220.77 ,  220.83 ,  220.79 ,  220.78 ,  220.78 ,
       220.79 ,  220.83 ,  220.87 ,  220.88 ,  220.9  ,  220.97 ,
       221.05 ,  221.02 ,  221.01 ,  220.99 ,  221.04 ,  221.05 ,
       221.06 ,  221.07 ,  221.12 ,  221.06 ,  221.07 ,  221.03 ,
       221.01 ,  221.03 ,  221.03 ,  221.01 ,  221.02 ,  221.04 ,
       221.04 ,  221.07 ,  221.105,  221.1  ,  221.09 ,  221.08 ,
       221.07 ,  221.08 ,  221.03 ,  221.06 ,  221.1  ,  221.11 ,
       221.11 ,  221.18 ,  221.2  ,  221.34 ,  221.29 ,  221.235,
       221.22 ,  221.2  ,  221.21 ,  221.22 ,  221.19 ,  221.17 ,
       221.19 ,  221.13 ,  221.13 ,  221.12 ,  221.14 ,  221.11 ,
       221.165,  221.19 ,  221.18 ,  221.19 ,  221.18 ,  221.15 ,
       221.16 ,  221.155,  221.185,  221.19 ,  221.2  ,  221.2  ,
       221.16 ,  221.18 ,  221.16 ,  221.11 ,  221.07 ,  221.095,
       221.08 ,  221.08 ,  221.09 ,  221.11 ,  221.08 ,  221.08 ,
       221.1  ,  221.08 ,  221.11 ,  221.07 ,  221.11 ,  221.1  ,
       221.09 ,  221.07 ,  221.14 ,  221.12 ,  221.08 ,  221.09 ,
       221.05 ,  221.08 ,  221.065,  221.05 ,  221.06 ,  221.085,
       221.095,  221.07 ,  221.05 ,  221.09 ,  221.1  ,  221.145,
       221.12 ,  221.14 ,  221.12 ,  221.12 ,  221.12 ,  221.11 ,
       221.14 ,  221.15 ,  221.13 ,  221.12 ,  221.11 ,  221.105,
       221.105,  221.13 ,  221.14 ,  221.1  ,  221.105,  221.105,
       221.11 ,  221.13 ,  221.15 ,  221.11 ,  221.13 ,  221.08 ,
       221.11 ,  221.12 ,  221.12 ,  221.12 ,  221.13 ,  221.15 ,
       221.18 ,  221.21 ,  221.18 ,  221.15 ,  221.15 ,  221.15 ,
       221.15 ,  221.15 ,  221.13 ,  221.13 ,  221.16 ,  221.13 ,
       221.11 ,  221.12 ,  221.09 ,  221.07 ,  221.06 ,  221.04 ,
       221.06 ,  221.09 ,  221.07 ,  221.045,  221.   ,  220.99 ,
       220.985,  220.95 ,  221.   ,  221.01 ,  221.005,  220.99 ,
       221.03 ,  221.055,  221.06 ,  221.03 ,  221.03 ,  221.03 ,
       221.   ,  220.95 ,  220.96 ,  220.97 ,  220.965,  220.97 ,
       220.94 ,  220.93 ,  220.9  ,  220.9  ,  220.9  ,  220.91 ,
       220.94 ,  220.92 ,  220.94 ,  220.91 ,  220.92 ,  220.935,
       220.875,  220.89 ,  220.91 ,  220.92 ,  220.93 ,  220.93 ,
       220.91 ,  220.9  ,  220.89 ,  220.9  ,  220.9  ,  220.93 ,
       220.94 ,  220.92 ,  220.93 ,  220.88 ,  220.88 ,  220.86 ,
       220.9  ,  220.92 ,  220.85 ,  220.83 ,  220.83 ,  220.795,
       220.81 ,  220.78 ,  220.7  ,  220.69 ,  220.6  ,  220.58 ,
       220.61 ,  220.63 ,  220.68 ,  220.63 ,  220.63 ,  220.595,
       220.66 ,  220.645,  220.64 ,  220.6  ,  220.579,  220.53 ,
       220.53 ,  220.5  ,  220.42 ,  220.49 ,  220.49 ,  220.5  ,
       220.475,  220.405,  220.4  ,  220.425,  220.385,  220.37 ,
       220.49 ,  220.46 ,  220.45 ,  220.48 ,  220.51 ,  220.48 ]
)

plt.figure(figsize=(8,2))
plt.plot(prices)
plt.ylabel('price [USD]')
plt.xlabel('Nov 28, 2016')
plt.xticks([30, 90, 150, 210, 270, 330, 390],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlim([0, 390]);
```

## Persistent random walk model

In this example, we choose to model the price evolution of SPY as a simple, well-known random walk model: the auto-regressive process of first order. We assume that subsequent log-return values $r_t$ of SPY obey the following recursive instruction:

$$
r_t = \rho_t \cdot r_{t-1} + \sqrt{1-\rho^2} \cdot \sigma_t \cdot \epsilon_t
$$

with the time-varying correlation coefficient $\rho_t$ and the time-varying volatility parameter $\sigma_t$. Here, $\epsilon_t$ is drawn from a standard normal distribution and represents the driving noise of the process and the scaling factor $\sqrt{1-\rho_t^2}$ makes sure that $\sigma_t$ is the standard deviation of the process. In *bayesloop*, we define this observation model as follows:
```
bl.om.ScaledAR1('rho', bl.oint(-1, 1, 100), 'sigma', bl.oint(0, 0.006, 400))
```
This implementation of the correlated random walk model will be discussed in detail in the next section.

Looking at the log-returns of our example, we find that the magnitude of the fluctuations (i.e. the volatility) is higher after market open and before market close. While these variations happen quite gradually, a large peak around 10:30am (and possibly another one around 12:30pm) represents an abrupt price correction.

```python
logPrices = np.log(prices)
logReturns = np.diff(logPrices)

plt.figure(figsize=(8,2))
plt.plot(np.arange(1, 390), logReturns, c='r')
plt.ylabel('log-returns')
plt.xlabel('Nov 28, 2016')
plt.xticks([30, 90, 150, 210, 270, 330, 390],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.yticks([-0.001, -0.0005, 0, 0.0005, 0.001])
plt.xlim([0, 390]);
```

## Online study

*bayesloop* provides the class `OnlineStudy` to analyze on-going data streams and perform model selection for each new data point. In contrast to other Study types, more than just one transition model can be assigned to an `OnlineStudy` instance, using the method `addTransitionModel`. Here, we choose to add two distinct scenarios:

- **normal:** Both volatility and correlation of subsequent returns are allowed to vary gradually over time, to account for periods with above average trading activity after market open and before market close. This scenario therefore represents a smoothly running market.

- **chaotic:** This scenario assumes that we know nothing about the value of volatility or correlation. The probability that this scenario gets assigned in each minute therefore represents the probability that previously gathered knowledge about market dynamics cannot explain the last price movement.

By evaluating how likely the *chaotic* scenario explains each new minute close price of SPY compared to the *normal* scenario, we can identify specific events that lead to extreme fluctuations in intra-day trading.

First, we create a new instance of the `OnlineStudy` class and set the observation model introduced above. The keyword argument `storeHistory` is set to `True`, because we want to access the parameter estimates of all time steps afterwards, not only the estimates of the last time step.

```python
S = bl.OnlineStudy(storeHistory=True)

L = bl.om.ScaledAR1('rho', bl.oint(-1, 1, 100),
                    'sigma', bl.oint(0, 0.006, 400))

S.set(L)
```

    + Created new study.
      --> Hyper-study
      --> Online study
    + Observation model: Scaled autoregressive process of first order (AR1). Parameter(s): ['rho', 'sigma']

**Note:** While the parameter `rho` is naturally constrained to the interval ]-1, 1[, the parameter boundaries of `sigma` have to be specified by the user. Typically, one can review past log-return data and estimate the upper boundary as a multiple of the standard deviation of past data points.

Both scenarios for the dynamic market behavior are implemented via the method `add`. The *normal* case consists of a combined transition model that allows both volatility and correlation to perform a Gaussian random walk. As the standard deviation (magnitude) of the parameter fluctuations is a-priori unknown, we supply a wide range of possible values (`bl.cint(0, 1.5e-01, 15)` for `rho`, which corresponds to 15 equally spaced values within the closed interval [0, 0.15], and 50 equally spaced values within the interval [0, 1.5$\cdot$10$^{-4}$] for `sigma`).

Since we have no prior assumptions about the standard deviations of the Gaussian random walks, we let *bayesloop* assign equal probability to all values. If one wants to analyze more than just one trading day, the (hyper-)parameter distributions from the end of one day can be used as the prior distribution for the next day! One might also want to suppress large variations of `rho` or `sigma` with an exponential prior, e.g.:

```
bl.tm.GaussianRandomWalk('s1', bl.cint(0, 1.5e-01, 15), target='rho',
                         prior=stats.Exponential('expon', 1./3.0e-02))
```

The *chaotic* case is implemented by the transition model `Independent`. This model sets a flat prior distribution for the parameters volatility and correlation in each time step. This way, previous knowledge about the parameters is not used when analyzing a new data point.

```python
T1 = bl.tm.CombinedTransitionModel(
         bl.tm.GaussianRandomWalk('s1', bl.cint(0, 1.5e-01, 15), target='rho'),
         bl.tm.GaussianRandomWalk('s2', bl.cint(0, 1.5e-04, 50), target='sigma')
     )

T2 = bl.tm.Independent()

S.add('normal', T1)
S.add('chaotic', T2)
```

    + Added transition model: normal (750 combination(s) of the following hyper-parameters: ['s1', 's2'])
    + Added transition model: chaotic (no hyper-parameters)

Before any data points are passed to the study instance, we further provide prior probabilities for the two scenarios. We expect about one news announcement containing unexpected information per day and set a prior probability of $1/390$ for the *chaotic* scenario (one normal trading day consists of 390 trading minutes).

```python
S.setTransitionModelPrior([389/390., 1/390.])
```

    + Set custom transition model prior.

Finally, we can supply log-return values to the study instance, data point by data point. We use the `step` method to infer new parameter estimates and the updated probabilities of the two scenarios. Note that in this example, we use a `for` loop to feed all data points to the algorithm because all data points are already available. In a real application of the `OnlineStudy` class, one can supply each new data point as it becomes available and analyze it in real-time.

```python
for r in tqdm_notebook(logReturns):
    S.step(r)
```

    + Start model fit
    + Not enough data points to start analysis. Will wait for more data.
        + Set uniform prior with parameter boundaries.

## Volatility spikes

Before we analyze how the probability values of our two market scenarios change over time, we check whether the inferred temporal evolution of the time-varying parameters is realistic. Below, the log-returns are displayed together with the inferred marginal distribution (shaded red) and mean value (black line) of the volatility parameter, using the method `plotParameterEvolution`.

```python
plt.figure(figsize=(8, 4.5))

# data plot
plt.subplot(211)
plt.plot(np.arange(1, 390), logReturns, c='r')

plt.ylabel('log-returns')
plt.xticks([30, 90, 150, 210, 270, 330, 390],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.yticks([-0.001, -0.0005, 0, 0.0005, 0.001])
plt.xlim([0, 390])

# parameter plot
plt.subplot(212)
S.plot('sigma', color='r')

plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.ylim([0, 0.00075])
plt.xlim([-2, 388]);
```

Note that the volatility estimates of the first few trading minutes are not as accurate as later ones, as we initialize the algorithm with a non-informative prior distribution. One could of course provide a custom prior distribution as a more realistic starting point. Despite this *fade-in* period, the period of increased volatility after market open is captured nicely, as well as the (more subtle) increase in volatility during the last 45 minutes of the trading day. Large individual log-return values also result in an *volatility spikes* (around 10:30am and more subtle around 12:30pm).

## Islands of stability

The persistent random walk model does not only provide information about the magnitude of price fluctuations, but further quantifies whether subsequent log-return values are correlated. A positive correlation coefficient indicates diverging price movements, as a price increase is more likely followed by another increase, compared to a decrease. In contrast, a negative correlation coefficient indicates *islands of stability*, i.e. trading periods during which prices do not diffuse randomly (as with a corr. coeff. of zero). Below, we plot the price evolution of SPY on November 28, together with the inferred marginal distribution (shaded blue) and the corresponding mean value (black line) of the time-varying correlation coefficient.

```python
plt.figure(figsize=(8, 4.5))

# data plot
plt.subplot(211)
plt.plot(prices)

plt.ylabel('price [USD]')
plt.xticks([30, 90, 150, 210, 270, 330, 390],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlim([0, 390])

# parameter plot
plt.subplot(212)
S.plot('rho', color='#0000FF')

plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.ylim([-0.4, 0.4])
plt.xlim([-2, 388]);
```

As a correlation coefficient that deviates significantly from zero would be immediately exploitable to predict future price movements, we mostly find correlation values near zero (in accordance with the efficient market hypothesis). However, between 1:15pm and 2:15pm, we find a short period of negative correlation with a value around -0.2. During this period, subsequent price movements tend to cancel each other out, resulting in an unusually strong price stability.

Using the `Parser` sub-module of *bayesloop*, we can evaluate the probability that subsequent return values are negatively correlated. In the figure below, we tag all time steps with a probability for `rho < 0` of 80% or higher and find that this indicator nicely identifies the period of increased market stability!

**Note:** The arbitrary threshold of 80% for our market indicator is of course chosen with hindsight in this example. In a real application, more than one trading day of data needs to be analyzed to create robust indicators!

```python
# extract parameter grid values (rho) and corresponding prob. values (p)
rho, p = S.getParameterDistributions('rho')

# evaluate Prob.(rho < 0) for all time steps
P = bl.Parser(S)
p_neg_rho = np.array([P('rho < 0.', t=t, silent=True) for t in range(1, 389)])

# plotting
plt.figure(figsize=(8, 4.5))
plt.subplot(211)
plt.axhline(y=0.8, lw=1.5, c='g')
plt.plot(p_neg_rho, lw=1.5, c='k')
plt.fill_between(np.arange(len(p_neg_rho)), 0, p_neg_rho > 0.8, lw=0, facecolor='g', alpha=0.5)

plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.ylabel('prob. of neg. corr.')
plt.xlim([-2, 388])

plt.subplot(212)
plt.plot(prices)
plt.fill_between(np.arange(2, len(p_neg_rho)+2), 220.2, 220.2 + (p_neg_rho > 0.8)*1.4, lw=0, facecolor='g', alpha=0.5)
plt.ylabel('price [USD]')
plt.xticks([30, 90, 150, 210, 270, 330, 390],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlim([0, 390])
plt.ylim([220.2, 221.6])
plt.xlabel('Nov 28, 2016');
```

## Automatic tuning

One major advantage of the `OnlineStudy` class is that it not only infers the time-varying parameters of the *low-level* correlated random walk (the observation model `ScaledAR1`), but further infers the magnitude (the standard deviation of the transition model `GaussianRandomWalk`) of the parameter fluctuations and thereby tunes the parameter dynamics as new data arrives. As we can see below (left sub-figures), both magnitudes - one for `rho` and one for `sigma` - start off at a high level. This is due to our choice of a uniform prior, assigning equal probability to all hyper-parameter values before seeing any data. Over time, the algorithm *learns* that the true parameter fluctuations are less severe than previously assumed and adjusts the hyper-parameters accordingly. This newly gained information, summarized in the hyper-parameter distributions of the last time step (right sub-figures), could then represent the prior distribution for the next trading day.

```python
plt.figure(figsize=(8, 4.5))
plt.subplot(221)
S.plot('s1', color='green')
plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.xlim([-2, 388])
plt.ylim([0, 0.06])

plt.subplot(222)
S.plot('s1', t=388, facecolor='green', alpha=0.7)
plt.yticks([])
plt.xticks([0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08], ['0', '1', '2', '3', '4', '5', '6', '7', '8'])
plt.xlabel('s1 ($\cdot 10^{-2}$)')
plt.xlim([-0.005, 0.08])

plt.subplot(223)
S.plot('s2', color='green')
plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.xlim([-2, 388])
plt.ylim([0, 0.0001])

plt.subplot(224)
S.plot('s2', t=388, facecolor='green', alpha=0.7)
plt.yticks([])
plt.xticks([0, 0.00001, 0.00002, 0.00003], ['0', '1', '2', '3'])
plt.xlabel('s2 ($\cdot 10^{-5}$)')
plt.xlim([0, 0.00003])

plt.tight_layout()
```

## Real-time model selection

Finally, we investigate which of our two market scenarios - *normal* vs. *chaotic* - can explain the price movements best. Using the method `plot('chaotic')`, we obtain the probability values for the *chaotic* scenario compared to the *normal* scenario, **with respect to all past data points**:

```python
plt.figure(figsize=(8, 2))
S.plot('chaotic', lw=2, c='k')
plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.xlim([0, 388])
plt.ylabel('p("chaotic")')
```

As expected, the probability that the *chaotic* scenario can explain all past log-return values at a given point in time quickly falls off to practically zero. Indeed, a correlated random walk with slowly changing volatility and correlation of subsequent returns is better suited to describe the price fluctuations of SPY **in the majority of time steps**.

However, we may also ask for the probability that each **individual** log-return value is produced by either of the two market scenarios by using the keyword argument `local=True`:

```python
plt.figure(figsize=(8, 2))
S.plot('chaotic', local=True, c='k', lw=2)
plt.xticks([28, 88, 148, 208, 268, 328, 388],
           ['10am', '11am', '12pm', '1pm', '2pm', '3pm', '4pm'])
plt.xlabel('Nov 28, 2016')
plt.xlim([0, 388])
plt.ylabel('p("chaotic")')
plt.axvline(58, 0, 1, zorder=1, c='r', lw=1.5, ls='dashed', alpha=0.7)
plt.axvline(178, 0, 1, zorder=1, c='r', lw=1.5, ls='dashed', alpha=0.7);
```

Here, we find clear peaks indicating an increased probability for the *chaotic* scenario, i.e. that previously gained information about the market dynamics has become useless. Lets assume that we are concerned about market behavior as soon as there is at least a 1% risk that *normal* market dynamics can not describe the current price movement. This leaves us with three distinct events in the following time steps:

```python
p = S.getTransitionModelProbabilities('chaotic', local=True)
np.argwhere(p > 0.01)
```

    array([[ 59],
           [181],
           [382]], dtype=int64)

These time steps translate to the following trading minutes: **10:31am, 12:33pm and 3:54pm**.

The first peak at 10:31am directly follows the release of the [Texas Manufacturing Outlook Survey](http://www.dallasfed.org/news/releases/2016/nr161128.cfm) of the [Dallas Fed](http://www.dallasfed.org/). The publication of this set of economic indicators has already been announced by [financial news pages](http://finance.yahoo.com/news/10-things-know-opening-bell-113600334.html) before the market opened. While the title of the survey (*"Texas Manufacturing Activity Continues to Expand"*) indicated good news, investors reacted sceptically, possibly due to negative readings in both the *new orders index* and *growth rate of orders index* (c.f. [this article](http://247wallst.com/economy/2016/11/29/better-news-for-manufacturing-activity-in-texas/)).

**Note:** This first peak would be more pronounced if we had supplied a prior distribution that constrains strong parameter fluctuations!

The underlying trigger for the second peak, shortly after 12:30pm, remains unknown. No major macroeconomic indicators were published at that time, at least according to some economic news sites (see e.g. [nytimes.com](http://markets.on.nytimes.com/research/economy/indicators/indicators.asp?monthOffset=-3) or [liveindex.org](https://liveindex.org/46172/2016/11/us-pre-market-news-28-nov-2016/)).

The last peak at 3:54pm is likely connected to the imminent market close at 4pm. To protect themselves from unexpected news after trading hours, market participants often close their open positions before market close, generally leading to an increased volatility. If large market participants follow this behavior, price corrections may no longer be explained by *normal* market dynamics.

This example has shown that *bayesloop*'s `OnlineStudy` class can identify periods of increased volatility as well as periods of increased price stability (accompanied by a negative correlation of subsequent returns), and further automatically tunes its current assumptions about market dynamics. Finally, we have demonstrated that *bayesloop* serves as a viable tool to detect *"anomalous"* price corrections, triggered by large market participants or news announcements, in real-time.