a news article (removing stop words) in Figure \[fig:topic\], taken from the Reuters21578 data set obtained at [@reuters21578]. As we can see, HTMM gets much more consistent and smooth inferred topics compared to that obtained from a bag-of-words model (cf. supplementary material for details). This result agrees with human understanding.

Second-order vs. Third-order Learning
=====================================

We start by arguing that for the same observation data $\{Y_t\}_{t=0}^T$, the estimate of the pairwise co-occurrence probability $\Pr[Y_t,Y_{t+1}]$ is always more accurate than that of the triple co-occurrence probability $\Pr[Y_{t-1},Y_t,Y_{t+1}]$.

Let us first explicitly describe the estimator we use for these probabilities. For each observation $Y_t$, we define a coordinate vector ${\boldsymbol{\psi}}_t\in{\mathbf{R}}^K$, and ${\boldsymbol{\psi}}_t=\bm{e}_k$ if $Y_t=y_k$. The natural estimator for the pairwise co-occurrence probability matrix ${\boldsymbol{\varOmega}}$ is $$\label{eq:omega2}
\widehat{{\boldsymbol{\varOmega}}} = \frac{1}{T}\sum_{t=0}^{T-1}{\boldsymbol{\psi}}_t{\boldsymbol{\psi}}_{t+1}^{{\!\top\!}},$$ and similarly for the triple co-occurrence probability $\underline{{\boldsymbol{\varOmega}}_3}$ $$\label{eq:omega3}
\widehat{\underline{{\boldsymbol{\varOmega}}_3}} = \frac{1}{T-1}\sum_{t=1}^{T-1}{\boldsymbol{\psi}}_{t-1}\circ{\boldsymbol{\psi}}_t\circ{\boldsymbol{\psi}}_{t+1},$$ where $\circ$ denotes vector outer-product. [^2]

The first observation is that both $\widehat{{\boldsymbol{\varOmega}}}$ and $\widehat{\underline{{\boldsymbol{\varOmega}}_3}}$ are unbiased estimators: Obviously $\operatorname{E}({\boldsymbol{\psi}}_t{\boldsymbol{\psi}}_{t+1}^{{\!\top\!}})={\boldsymbol{\varOmega}}$ and likewise for the triple-occurrences, and taking their averages does not change the expectation. However, the individual terms in the summation are not independent of each other, making it hard to determine how fast estimates converge to their expectation. The state-of-the-art 