 Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.   In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate the applicability of the method to MuJoCo control problems. The problem becomes even more challenging if the results are dependent on the sequence of previous observations ), e.g. because of dynamic nature of the problem involving speed or acceleration, or the difference between the current and the previous control signal.In many real-world problems, it is possible to combine the reinforcement and the supervised learning. For the problem of autonomous driving, it is often possible to provide parallel signals of the autopilot in order to use this information to restrict the reinforcement learning solutions towards the sensible subsets of control actions. Similar things can also be done for robotic control. Such real world models can be analytical, or trained by machine learning techniques, and may use some other sensors, which are capable to provide alternative information (e.g., the model trained on LiDAR data can be used to train the vision based model). However, although there were some works using partially labelled datasets within the reinforcement learning framework BID23 ), as far as we believe, the proposed problem statement, injecting supervised data into reinforcement learning using regularisation of Q-functions, is different from the ones published before. In BID23 , the authors consider the problem of robotic control which does not involve video data, and their approach considers sharing the replay buffer between the reinforcement learning and demonstrator data.The novelty of the approach, presented in this paper, is given as follows:1. the regularised optimisation problem statement, combining reinforcement and supervised learning, is proposed;2. the training algorithm for the control method is proposed, based on this problem statement, and assessed on the control problems;3. the novel greedy actor-critic reinforcement learning algorithm is proposed as a part of the training algorithm, containing interlaced data collection, critic and actor update stages.The proposed method reduces the number of samples from millions or tens of millions, required to train the reinforcement learning model on visual data, to just hundreds of thousands, and also improves the quality against the supervised and reinforcement learning. The proposed method shows dramatic improvement in the number of samples for video data (down to just several hundred thousand) comparing to the reinforcement learning methods, as well as improves the performance comparing to both supervised and reinforcement learning. We believe that such approach, combining reinforcement and supervised learning, could help to succeed in the areas of complex spaces where the state-of-the-art reinforcement learning methods are not working yet, as well as towards practical usage for real world models such as autonomous cars or robots.However, there are still a few limitations of the proposed method. First, it still requires label data through all the course of training. We believe that in the future work it should be possible to reduce usage of training data to a limited number of labelled episodes. Such decrease of the training data could benefit to the range of practical tasks solvable by the proposed approach. The parameterisation for the experiments is given in TAB2 ; the parameters' verbal description is augmented with the names referencing to Algorithm 1.For supervised-only pretraining of the actor network, the Momentum algorithm is used BID15 ); for the rest of the stages, the Adam algorithm is used BID7 ). The proposed algorithm has been implemented in Python using TensorFlow framework BID0 ). For the stage of supervised pretraining, in order to improve convergence of the model at the initial stage, the additional soft update coefficient was introduced for exponential smoothing of the parameters of the network during gradient descent optimisation.
The new combination of reinforcement and supervised learning, dramatically decreasing the number of required samples for training on video This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy