Words features as an input to a neural network, learned separate embeddings for states and actions, and then computed the Q function from autocorrelations between these embeddings. Narasimhan et al. ([-@DBLP:journals/corr/NarasimhanKB15]) suggested to use a word level Long Short-Term Memory [@hochreiter1997long] to learn a representation end-to-end, and Zelinka et al. ([-@zelinka2018using]), combined these approaches.

**DRL with linear function approximation:** DRL methods such as the DQN have achieved state-of-the-art results in a variety of challenging, high-dimensional domains. This success is mainly attributed to the power of deep neural networks to learn rich domain representations for approximating the value function or policy [@mnih2015human; @zahavy2016graying; @zrihem2016visualizing]. Batch reinforcement learning methods with linear representations, on the other hand, are more stable and enjoy accurate uncertainty estimates. Yet, substantial feature engineering is necessary to achieve good results. A natural attempt at getting the best of both worlds is to learn a (linear) control policy on top of the representation of the last layer of a DNN. This approach was shown to refine the performance of DQNs [@levine2017shallow] and improve exploration [@azizzadenesheli2018efficient]. Similarly, for contextual linear bandits, Riquelme et al. showed that a neuro-linear Thompson sampling approach outperformed deep (and linear) bandit algorithms in practice [@riquelmedeep].

