Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc. Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations. This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for partial and complex observations. DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which allows for reasoning with partial observations over multiple time steps. While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model that is training directly for decision making. We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, because it circumvents the difficulty of modelling observations explicitly. In most cases, DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark that we introduce. We further show that DPFRL performs well for visual navigation with real-world data. Deep Reinforcement Learning (DRL) has attracted significant interest with applications ranging from game playing (Mnih et al., 2013; Silver et al., 2017) to robot control and visual navigation Kahn et al., 2018; Savva et al., 2019) . However, more natural or real-world environments pose significant challenges for current DRL methods (Arulkumaran et al., 2017) , in part because they require (1) reasoning in a partially observable environment (2) reasoning with complex observations, e.g. visually rich images. For example, a robot navigating in a new environment has to (1) localize and plan a path having only partial information of the environment (2) extract the traversable space from image pixels, where the relevant geometric features are tightly coupled with irrelevant visual features, such as wall textures and lighting. Decision making under partial observability can be formulated as a partially observable Markov decision process (POMDP). Solving POMDPs requires tracking the posterior distribution of the states, called the belief. Most POMDP RL methods track the belief, represented as a vector, using a recurrent neural network (RNN) (Hausknecht & Stone, 2015; Zhu et al., 2018) . RNNs are model-free generic function approximators, and without appropriate structural priors they may need large amounts of data to learn to track a complex belief. Model-based DRL methods aim to reduce the sample complexity by learning an environment model simultaneously with the policy. In particular, to deal with partial observability, recently proposed DVRL that learns a generative observation model incorporated into the policy through a Bayes filter. Because the Bayes filter tracks the belief explicitly, DVRL performs much better than generic RNNs under partial observability. However, a Bayes filter normally assumes a generative observation model, that defines the probability p(o | h t ) of receiving an observation o = o t given the history h t of past observations and actions (Fig. 1b ). Learning this model can be very challenging since the strong generative assumption requires modeling the whole observation space, including features irrelevant for RL. When o t is an image, p(o | h t ) is a distribution over all possible images, e.g., parameterized by independent pixel-wise Gaussians with learned mean and variance. This means, e.g., to navigate in a previously unseen environment, we need to learn the < l a t e x i t s h a 1 _ b a s e 6 4 = " r o 4 2 6 D H M J G U u G 8 x G G s K s G W z i 0 / s = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q J e y K 4 I e i 1 4 8 V r D b Q r u U b J p t Y 7 P Z J Z k V S u l / 8 O J B E a / + H 2 / + G 9 N 2 D 9 r 6 Q u D h n R k y 8 4 a p F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D I 9 8 k m W a 8 y R K Z 6 H Z I D Z d C 8 S Y K l L y d a k 7 j U P J W O L q d 1 V t P X B u R q A c c p z y I 6 U C J S D C K 1 v L 9 a t j D 8 1 6 5 4 t b c u c g q e D l U I F e j V / 7 q 9 h O W x V w h k 9 S Y j u e m G E y o R s E k n 5 a 6 m e E p Z S M 6 4 B 2 L i s b c B J P 5 t l N y Z p 0 + i R J t n 0 I y d 3 9 P T G h s z D g O b W d M c W i W a z P z v 1 o n w + g 6 m A i V Z s g V W 3 w U Z Z J g Q m a n k 7 7 Q n K E c W 6 B M C 7 s r Y U O q K U M b U M m G 4 C 2 f v A r + R c 2 z f H 9 Z q d / k c R T h B E 6 h C h 5 c Q R 3 u o A F N Y P A I z / A K b 0 7 i v D j v z s e i t e D k M 8 f w R 8 7 n D 7 8 m j p I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r o 4 2 6 D H M J G U u G 8 x G G s K s G W z i 0 / s = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q J e y K 4 I e i 1 4 8 V r D b Q r u U b J p t Y 7 P Z J Z k V S u l / 8 O J B E a / + H 2 / + G 9 N 2 D 9 r 6 Q u D h n R k y 8 4 a p F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D I 9 8 k m W a 8 y R K Z 6 H Z I D Z d C 8 S Y K l L y d a k 7 j U P J W O L q d 1 V t P X B u R q A c c p z y I 6 U C J S D C K 1 v L 9 a t j D 8 1 6 5 4 t b c u c g q e D l U I F e j V / 7 q 9 h O W x V w h k 9 S Y j u e m G E y o R s E k n 5 a 6 m e E p Z S M 6 4 B 2 L i s b c B J P 5 t l N y Z p 0 + i R J t n 0 I y d 3 9 P T G h s z D g O b W d M c W i W a z P z v 1 o n w + g 6 m A i V Z s g V W 3 w U Z Z J g Q m a n k 7 7 Q n K E c W 6 B M C 7 s r Y U O q K U M b U M m G 4 C 2 f v A r + R c 2 z f H 9 Z q d / k c R T h B E 6 h C h 5 c Q R 3 u o A F N Y P A I z / A K b 0 7 i v D j v z s e i t e D k M 8 f w R 8 7 n D 7 8 m j p I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r o 4 2 6 D H M J G U u G 8 x G G s K s G W z i 0 / s = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q J e y K 4 I e i 1 4 8 V r D b Q r u U b J p t Y 7 P Z J Z k V S u l / 8 O J B E a / + H 2 / + G 9 N 2 D 9 r 6 Q u D h n R k y 8 4 a p F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D I 9 8 k m W a 8 y R K Z 6 H Z I D Z d C 8 S Y K l L y d a k 7 j U P J W O L q d 1 V t P X B u R q A c c p z y I 6 U C J S D C K 1 v L 9 a t j D 8 1 6 5 4 t b c u c g q e D l U I F e j V / 7 q 9 h O W x V w h k 9 S Y j u e m G E y o R s E k n 5 a 6 m e E p Z S M 6 4 B 2 L i s b c B J P 5 t l N y Z p 0 + i R J t n 0 I y d 3 9 P T G h s z D g O b W d M c W i W a z P z v 1 o n w + g 6 m A i V Z s g V W 3 w U Z Z J g Q m a n k 7 7 Q n K E c W 6 B M C 7 s r Y U O q K U M b U M m G 4 C 2 f v A r + R c 2 z f H 9 Z q d / k c R T h B E 6 h C h 5 c Q R 3 u o A F N Y P A I z / A K b 0 7 i v D j v z s e i t e D k M 8 f w R 8 7 n D 7 8 m j p I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r o 4 2 6 D H M J G U u G 8 x G G s K s G W z i 0 / s = " > A A A B 7 X i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q J e y K 4 I e i 1 4 8 V r D b Q r u U b J p t Y 7 P Z J Z k V S u l / 8 O J B E a / + H 2 / + G 9 N 2 D 9 r 6 Q u D h n R k y 8 4 a p F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D I 9 8 k m W a 8 y R K Z 6 H Z I D Z d C 8 S Y K l L y d a k 7 j U P J W O L q d 1 V t P X B u R q A c c p z y I 6 U C J S D C K 1 v L 9 a t j D 8 1 6 5 4 t b c u c g q e D l U I F e j V / 7 q 9 h O W x V w h k 9 S Y j u e m G E y o R s E k n 5 a 6 m e E p Z S M 6 4 B 2 L i s b c B J P 5 t l N y Z p 0 + i R J t n 0 I y d 3 9 P T G h s z D g O b W d M c W i W a z P z v 1 o n w + g 6 m A i V Z s g V W 3 w U Z Z J g Q m a n k 7 7 Q n K E c W 6 B M C 7 s r Y U O q K U M b U M m G 4 C 2 f v A r + R c 2 z f H 9 Z q d / k c R T h B E 6 h C h 5 c Q R 3 u o A F N Y P A I z / A K b 0 7 i v D j v z s e i t e D k M 8 f w R 8 7 n D 7 8 m j p I = < / l a t e x i t > ⇡(b t ) < l a t e x i t s h a 1 _ b a s e 6 4 = " d M D G r E g / u U t R q i s D I A Z M / + 5 R W i g = " > A A A B 7 3 i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F q J e S i K D H o h e P F e w H t K F s t p t 2 6 W Y T d y d C C f 0 T X j w o 4 t W / 4 8 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E C o O u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U c v E q W a 8 y W I Z 6 0 5 A D Z d C 8 S Y K l L y T a E 6 j Q P J 2 M L 6 d 1 d t P X B s R q w e c J N y P 6 F C J U D C K 1 u r 0 E l E N + n j e L 1 f c m j s X W Q U v h w r k a v T L X 7 1 B z N K I K 2 S S G t P 1 3 A T 9 j G o U T P J p q Z c a n l A 2 p k P e t a h o x I 2 f z f e d k j P r D E g Y a / s U k r n 7 e y K j 0 X l x 3 p 2 P R W v B y W e O 4 Y + c z x 9 g o Y + F < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d M D G r E g / u U t R q i s D I A Z M / + 5 R W i g = " > A A A B 7 3 i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F q J e S i K D H o h e P F e w H t K F s t p t 2 6 W Y T d y d C C f 0 T X j w o 4 t W / 4 8 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E C o O u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U c v E q W a 8 y W I Z 6 0 5 A D Z d C 8 S Y K l L y T a E 6 j Q P J 2 M L 6 d 1 d t P X B s R q w e c J N y P 6 F C J U D C K 1 u r 0 E l E N + n j e L 1 f c m j s X W Q U v h w r k a v T L X 7 1 B z N K I K 2 S S G t P 1 3 A T 9 j G o U T P J p q Z c a n l A 2 p k P e t a h o x I 2 f z f e d k j P r D E g Y a / s U k r n 7 e y K j 0 X l x 3 p 2 P R W v B y W e O 4 Y + c z x 9 g o Y + F < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d M D G r E g / u U t R q i s D I A Z M / + 5 R W i g = " > A A A B 7 3 i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F q J e S i K D H o h e P F e w H t K F s t p t 2 6 W Y T d y d C C f 0 T X j w o 4 t W / 4 8 1 / 4 7 b N Q V t f W H h 4 Z 4 a d e Y N E C o O u + + 0 U 1 t Y 3 N r e K 2 6 W d 3 b 3 9 g / L h U c v E q W a 8 y W I Z 6 0 5 A D Z d C 8 S Y K l L y T a E 6 j Q P J 2 M L 6 d 1 d t P X B s R q w e c J N y P 6 F C J U D C K 1 u r 0 E l E N + n j e L 1 f c m j s X W Q U v h w r k a v T L X 7 1 B z N K I K 2 S S G t P 1 3 A T 9 j G o U T P J p q Z c a n l A 2 p k P e t a h o x I 2 f z f e d k j P r D E g Y a / s U k r n 7 e y K j l L y T a E 6 j Q P J 2 M L 6 d 1 d t P X B s R q w e c J N y P 6 F C J U D C K 1 u r 0 E l E N + n j e L 1 f c m j s X W Q U v h w r k a v T L X 7 1 B z N K I K 2 S S G t P 1 3 A T 9 j G o U T P J p q Z c a n l A 2 p k P e t a h o x I 2 f z f e d k j P r D E g Y a / s U k r n 7 e y K j distribution of all possible environments with their visual appearance, lighting condition, etc. -a much harder task than learning to extract features relevant to navigation, e.g. the traversable space. We introduce the Discriminative Particle Filter Reinforcement Learning (DPFRL), a POMDP RL method that learns to explicitly track a latent belief, while circumventing the difficulty of generative observation modeling, and learns to make decisions based on features of the latent belief (Fig. 1a) . DPFRL approximates the belief by a set of weighted learnable latent particles {(h , and it tracks the particle belief by a non-parametric Bayes filter algorithm, a particle filter, encoded as a differentiable computational graph in the neural network architecture. Transition and observation models for the particle filter are neural networks learned jointly end-to-end, optimized for the overall policy. Importantly, we use a discriminatively parameterized observation model, f obs (o t , h t ), a neural network that takes in o t and h t and outputs a single value, a direct estimate of the log-likelihood as shown in Fig. 1c . The discriminative parameterization relaxes the generative assumption and avoids explicitly modeling the entire complex observation space when computing observation likelihood. The intuition is similar to that of, e.g., energy-based models (LeCun et al., 2006) and contrastive predictive coding (Oord et al., 2018) , but here the learning signal comes directly from the RL objective, backpropagating through the differentiable particle filter, thus f obs (o t , h t ) only needs to model the observation features relevant to decision making. In addition, to summarize the particle belief, we introduce novel learnable features based on Moment-Generating Functions (MGFs) (Bulmer, 1979) . MGF features are computationally efficient and permutation invariant, and they can be directly optimized to provide useful higher-order moment information for learning the policy. MGF features could be also used as learned features of any empirical distribution in application beyond RL. We evaluate DPFRL on a range of POMDP RL domains: a continuous control task from , Flickering Atari Games (Hausknecht & Stone, 2015) , Natural Flickering Atari Games, a new domain with more complex observations that we introduce, and the Habitat visual navigation domain using real-world data (Savva et al., 2019) . DPFRL outperforms state-of-the-art POMDP RL methods in most cases. Results show that the particle filter structure is effective for handling partial observations, and the discriminative parameterization allows for complex observations. We summarize our contributions as follows: 1) a differentiable particle filter based method with a discriminatively parameterized observation model for RL with partial and complex observations. 2) effective MGF features for empirical distributions, e.g., particle distributions 3) a new RL benchmark, Natural Flickering Atari Games, that introduces both partial observability and complex visual observations to the popular Atari domain. We will open source the code to enable future work. We have introduced DPFRL, a principled framework for POMDP RL in natural environments. DPFRL combines the strength of Bayesian filtering and end-to-end RL: it performs explicit belief tracking with discriminative learnable particle filters optimized directly for the RL policy. DPFRL achieved state-of-the-art results on POMDP RL benchmarks from prior work, Mountain Hike and a number of Flickering Atari Games, and it significantly outperformed alternative methods in a new, more challenging domain, Natural Flickering Atari Games, as well as for visual navigation using real-world data. We have proposed a novel MGF feature for extracting statistics from an empirical distribution. MGF feature extraction could be applied beyond RL, e.g., for general sequence prediction. DPFRL does not perform well in some particular cases, e.g., DoubleDunk. While our discriminatively parameterized observation function is less susceptible to observation noise, it does not allow for additional learning signals that improve sample efficiency, e.g., through a reconstruction loss. Future work may combine generative and discriminative modeling with the principled DPFRL framework.
We introduce DPFRL, a framework for reinforcement learning under partial and complex observations with a fully differentiable discriminative particle filter Introduces ideas for training DLR agents with latent state variables, modeled as a belief distribution, so they can handle partially observed environments. This paper introduces a principled method for POMDP RL: Discriminative Particle Filter Reinforcement Learning that allows for reasoning with partial observations over multiple time steps, achieving state-of-the-art on benchmarks.