We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system. Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking. Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns. Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training. We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines. We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs. Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies. Deep reinforcement learning (RL) has emerged as a powerful approach for sequential decision-making problems, achieving impressive results in domains such as game playing (Mnih et al., 2015; BID8 and robotics (Levine et al., 2016; BID5 Lillicrap et al., 2015) . This paper concerns RL in input-driven environments. Informally, input-driven environments have dynamics that are partially dictated by an exogenous, stochastic input process. Queuing systems (Kleinrock, 1976; Kelly, 2011) are an example; their dynamics are governed by not only the decisions made within the system (e.g., scheduling, load balancing) but also the arrival process that brings work (e.g., jobs, customers, packets) into the system. Input-driven environments also arise naturally in many other domains: network control and optimization BID18 Mao et al., 2017) , robotics control with stochastic disturbances BID3 , locomotion in environments with complex terrains and obstacles (Heess et al., 2017) , vehicular traffic control BID1 BID19 , tracking moving targets, and more (see FIG0 . We focus on model-free policy gradient RL algorithms BID17 Mnih et al., 2016; BID5 , which have been widely adopted and benchmarked for a variety of RL tasks (Duan et al., 2016; Wu & Tian, 2017) . A key challenge for these methods is the high variance in the gradient estimates, as such variance increases sample complexity and can impede effective learning BID6 Mnih et al., 2016) . A standard approach to reduce variance is to subtract a "baseline" from the total reward (or "return") to estimate the policy gradient BID16 . The most common choice of a baseline is the value function -the expected return starting from the state.Our main insight is that a state-dependent baseline -such as the value function -is a poor choice in input-driven environments, whose state dynamics and rewards are partially dictated by the input process. In such environments, comparing the return to the value function baseline may provide limited information about the quality of actions. The return obtained after taking a good action may be poor (lower than the baseline) if the input sequence following the action drives the system to unfavorable states; similarly, a bad action might end up with a high return with an advantageous input sequence. Intuitively, a good baseline for estimating the policy gradient should take the specific instance of the input process -the sequence of input values -into account. We call such a baseline an input-dependent baseline; it is a function of both the state and the entire future input sequence.We formally define input-driven Markov decision processes, and we prove that an input-dependent baseline does not introduce bias in standard policy gradient algorithms such as Advantage Actor (Harchol-Balter & Vesilo, 2010) with stochastic job arrival as the input process; (b) adaptive bitrate video streaming (Mao et al., 2017) with stochastic network bandwidth as the input process; (c) Walker2d in wind with a stochastic force (wind) applied to the walker as the input process; (d) HalfCheetah on floating tiles with the stochastic process that controls the buoyancy of the tiles as the input process; (e) 7-DoF arm tracking moving target with the stochastic target position as the input process. Environments (c)-(e ) use the MuJoCo physics simulator BID13 .Critic (A2C) (Mnih et al., 2016) and Trust Region Policy Optimization (TRPO) BID5 , provided that the input process is independent of the states and actions. We derive the optimal input-independent baseline and a simpler one to work with in practice; this takes the form of a conditional value function -the expected return given the state and the future input sequence.Input-dependent baselines are harder to learn than their state-dependent counterparts; they are highdimensional functions of the sequence of input values. To learn input-dependent baselines efficiently, we propose a simple approach based on meta-learning (Finn et al., 2017; BID15 . The idea is to learn a "meta baseline" that can be specialized to a baseline for a specific input instantiation using a small number of training episodes with that input. This approach applies to applications in which an input sequence can be repeated during training, e.g., applications that use simulations or experiments with previously-collected input traces for training (McGough et al., 2017) .We compare our input-dependent baseline to the standard value function baseline for the five tasks illustrated in FIG0 . These tasks are derived from queuing systems (load balancing heterogeneous servers (Harchol-Balter & Vesilo, 2010) ), computer networks (bitrate adaptation for video streaming (Mao et al., 2017) ), and variants of standard continuous control RL benchmarks in the MuJoCo physics simulator BID13 . We adapted three widely-used MuJoCo benchmarks (Duan et al., 2016; Clavera et al., 2018a; Heess et al., 2017) to add a stochastic input element that makes these tasks significantly more challenging. For example, we replaced the static target in a 7-DoF robotic arm target-reaching task with a randomly-moving target that the robot aims to track over time. Our results show that input-dependent baselines consistently provide improved training stability and better eventual policies. Input-dependent baselines are applicable to a variety of policy gradient methods, including A2C, TRPO, PPO, robust adversarial RL methods such as RARL BID3 , and meta-policy optimization such as MB- MPO (Clavera et al., 2018b) . Video demonstrations of our experiments are available at https://sites.google.com/view/input-dependent-baseline/. We introduced input-driven Markov Decision Processes in which stochastic input processes influence state dynamics and rewards. In this setting, we demonstrated that an input-dependent baseline can significantly reduce variance for policy gradient methods, improving training stability and the quality of learned policies. Our work provides an important ingredient for using RL successfully in a variety of domains, including queuing networks and computer systems, where an input workload is a fundamental aspect of the system, as well as domains where the input process is more implicit, like robotics control with disturbances or random obstacles.We showed that meta-learning provides an efficient way to learn input-dependent baselines for applications where input sequences can be repeated during training. Investigating efficient architectures for input-dependent baselines for cases where the input process cannot be repeated in training is an interesting direction for future work. Consider a walker in a 1D grid world, where the state s t ∈ Z at time t denotes the position of the walker, and action a t ∈ {−1, +1} denotes the intent to either move forward or backward. Additionally let z t ∈ {−1, +1} be a uniform i.i.d. "exogenous input" that perturbs the position of the walker. For an action a t and input z t , the state of the walker in the next step is given by s t+1 = s t + a t + z t . The objective of the game is to move the walker forward; hence, the reward is r t = a t + z t at each time step. γ ∈ [0, 1] is a discount factor.While the optimal policy for this game is clear (a t = +1 for all t), consider learning such a policy using policy gradient. For simplicity, let the policy be parametrized as π θ (a t = +1|s t ) = e θ /(1+e θ ), with θ initialized to 0 at the start of training. In the following, we evaluate the variance of the policy gradient estimate at the start of training under (i) the standard value function baseline, and (ii) a baseline that is the expected cumulative reward conditioned on all future z t inputs.Variance under standard baseline. The value function in this case is identically 0 at all states. This is because E[ DISPLAYFORM0 γ t (a t + z t )] = 0 since both actions a t and inputs z t are i.i.d. with mean 0. Also note that ∇ θ log π θ (a t = +1) = 1/2 and ∇ θ log π θ (a t = −1) = −1/2; hence ∇ θ log π θ (a t ) = a t /2. Therefore the variance of the policy gradient estimate can be written as DISPLAYFORM1 Variance under input-dependent baseline. Now, consider an alternative "input-dependent" baseline DISPLAYFORM2 Intuitively this baseline captures the average reward incurred when experiencing a particular fixed z sequence. We refer the reader to §4 for a formal discussion and analysis of input-dependent baselines. Evaluating the baseline we get V ( DISPLAYFORM3 Therefore the variance of the policy gradient estimate in this case is DISPLAYFORM4 Reduction in variance. To analyze the variance reduction between the two cases (Equations FORMULA16 and FORMULA19 ), we note that DISPLAYFORM5 This follows because DISPLAYFORM6 Therefore the covariance term in Equation (7) is 0. Hence the variance reduction from Equation FORMULA20 can be written as DISPLAYFORM7 Thus the input-dependent baseline reduces variance of the policy gradient estimate by an amount proportional to the variance of the external input. In this toy example, we have chosen z t to be binaryvalued, but more generally the variance of z t could be arbitrarily large and might be a dominating factor of the overall variance in the policy gradient estimation.
For environments dictated partially by external input processes, we derive an input-dependent baseline that provably reduces the variance for policy gradient methods and improves the policy performance in a wide range of RL tasks. The authors consider the problem of learning in input-driven environments, show how the PG theorem still applies for an input-aware critic, and show that input-dependent baselines are the best to use in conjecture with that critic. This paper introduces the notion of input-dependent baselines in Policy Gradient Methods in RL, and proposes different methods to train the input dependent baseline function to help clear variance from external factor perturbation.