Title: Closing the Gap Between SVRG and TD-SVRG with Gradient Splitting. Abstract: Temporal difference (TD) learning is a simple algorithm for policy evaluation
in reinforcement learning. The performance of TD learning is affected by high
variance and it can be naturally enhanced with variance reduction techniques, such
as the Stochastic Variance Reduced Gradient (SVRG) method. Recently, multiple
works have sought to fuse TD learning with SVRG to obtain a policy evaluation
method with a linear rate of convergence. However, the resulting convergence rate
is significantly weaker than what is achieved by SVRG in the setting of convex
optimization. In this work we utilize a recent interpretation of TD-learning as the
splitting of the gradient of an appropriately chosen function, thus simplifying the
algorithm and fusing TD with SVRG. We prove a linear convergence bound that
is identical to the convergence bound available for SVRG in the convex setting.