the topics rather than the over-simplifying independence assumption.

We can see some immediate advantages for the HTMM. Since the Markovian assumption is only imposed on the topics, which are not exposed to us, the observations (words) are not independent from each other, which agrees with our intuition. On the other hand, we now understand that although word dependencies exist for a wide neighborhood, we only need to work with pair-wise co-occurrence probabilities, or 2-grams. This releases us from picking a window length $n$ in the $n$-gram model, while maintaining dependencies between words well beyond a neighborhood of $n$ words. It also includes the bag-of-words assumption as a special case: If the topics of the words are indeed independent, this just means that the transition probability has the special form ${\boldsymbol{\mathit{1}}}{\boldsymbol{p}}_d^{{\!\top\!}}$. The closest work to ours is by @gruber2007hidden, which is also termed hidden topic Markov model. However, they make a simplifying assumption that the transition probability takes the form $(1-\epsilon){\boldsymbol{I}}+ \epsilon{\boldsymbol{\mathit{1}}}{\boldsymbol{p}}_d^{{\!\top\!}}$, meaning the topic of the word is either the same as the previous one, or independently drawn from ${\boldsymbol{p}}_d$. Both models are special cases of our general HTMM.

In order to learn the shared topic matrix ${\boldsymbol{M}}$, we can 