general idea is to explicitly incorporate the reference trajectory on a moving horizon in our new Q-function that is learned without requiring the system dynamics. Due to considering not only the current tracking error but rather the reference trajectory, this Q-function yields a controller which does not only achieve reactive but predictive behavior.

In particular, we provide:

-   A novel moving horizon tracking Q-function whose minimizing control also minimizes the tracking costs.

-   Derivation and proof of the analytical solution of the novel Q-function given the cost function and the system dynamics for the linear-quadratic tracking case.

-   Convergence proofs for our learning algorithm that learns optimal tracking from data without knowledge of the system dynamics. Thus, the estimated Q-function parameters as well as the associated control law converge to the optimal solution.

The rest of this paper is organized as follows. In Section \[problem\_def\], we define the optimal tracking problem with unknown system dynamics. The novel Q-function for arbitrary reference tracking is defined and analyzed in Section \[qfunc\_section\]. In Section \[section\_learning\], we introduce our learning algorithm that is based on the previously defined Q-function and provide convergence proofs. Simulation results and a comparison of our method with an adaptive 