Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs. 
 LfD algorithms generally assume learning from single task demonstrations. In practice, however, it is more efficient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering. Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior. In this paper we present an LfD approach for learning multiple modes of behavior from visual data. Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network. We present an efficient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.
 We demonstrate our method on real robot visual object reaching tasks, and show that
 it can reliably learn the multiple behavior modes in the demonstration data. Video results are available at https://vimeo.com/240212286/fd401241b9. A key problem in robotic control is to simplify the problem of programming a complex behavior. Traditional control engineering approaches, which rely on accurate manual modeling of the system environment, are very challenging to apply in modern robotic applications where most sensory inputs come from images and other high-dimensional signals such as tactile feedback.In contrast, imitation learning, or learning from demonstration (LfD) approaches BID31 aim to directly learn a control policy from mentor or expert demonstrations. The key advantages of LfD are simplicity and data-efficiency, and indeed, LfD has been successfully used for learning complex robot skills such as locomotion BID32 , driving BID27 BID30 , flying BID0 , and manipulation BID21 BID4 BID25 . Recently, advances in deep representation learning BID12 have facilitated LfD methods with high dimensional perception, such as mapping raw images directly to controls BID9 . These advances are capable of learning generalizable skills BID18 , and offer a promising approach for modern industrial challenges such as pick and place tasks BID5 .One challenge in LfD, however, is learning different modes of the same task. For example, consider learning to pick up an object from a pile. The demonstrator can choose to pick up a different object each time, yet we expect LfD to understand that these are similar demonstrations of the same pick-up skill, only with a different intention in mind. Moreover , we want the learned robot behavior to display a similar multi-modal 1 nature.Standard approaches for LfD with image inputs, such as learning with deep neural networks (NNs) BID27 BID9 BID18 , are not suitable for learning multimodal behaviors. In their essence, NNs learn a deterministic mapping from observation to control, which cannot represent the inherently multi-modal latent intention in the demonstrations. In practice , this manifests as an 'averaging' of the different modes in the data BID3 , leading to an undesirable policy.A straightforward approach for tackling the multi-modal problem in LfD is to add a label for each mode in the data. Thus, in the pick-up task above, the demonstrator would also explicitly specify the object she intends to pick-up beforehand. Such an approach has several practical shortcomings: it requires the demonstrator to record more data, and requires the possible intentions to be specified in advance, making it difficult to use the same recorded data for different tasks. More importantly , such a solution is conceptually flawed -it solves an algorithmic challenge by placing additional burden on the client.In this work, we propose an approach for LfD with multi-modal demonstrations that does not require any additional data labels. Our method is based on a stochastic neural network model, which represents the latent intention as a random activation in the network. We propose a novel and efficient learning algorithm for training stochastic networks, and present a network architecture suitable for LfD with raw image inputs, where the intention takes the form of a stochastic attention over features in the image.We show that our method can reliably reproduce behavior with multiple intentions in real-robot object reaching tasks. Moreover, in scenarios where multiple intentions exist in the demonstration data, the stochastic neural networks perform better than their deterministic counterparts. We presented an approach for learning from demonstrations that contain multiple modes of performing the same task. Our method is based on stochastic neural networks, and represents the mode Figure 2 : Comparison of IDS and SNN algorithms. We plot three different errors during training (on the training data), for the same model trained using IDS and SNN algorithm. Left: the respective training loss for each method. Since the max in IDS upper bounds the softmax in SNN, the loss plot for IDS lower bounds SNN. Middle: the IDS loss on the training data, for both models. Since the SNN is trained on a different loss function (softmax), its performance is worse. This shows an important point: if, at test time, we use optimistic sampling to sample z from best samples during training, we should expect IDS to perform better than SNN. Right: the average log-likelihood loss during training. The SNN wins here, since the softmax encourages to increase the likelihood of 'incorrect' z values. This provides additional motivation for using optimistic sampling.of performing the task by a stochastic vector -the intention, which is given as input to a feedforward neural network. We presented a simple and efficient algorithm for training our models, and a particular implementation suitable for vision-based inputs. As we demonstrated in real-robot experiments, our method can reliably learn to reproduce the different modes in the demonstration data, and outperforms standard approaches in cases where such different modes exist.In future work we intend to investigate the extension of this approach to more complex manipulation tasks such as grasping and assembly, and domains with a very large number of objects in the scene. An interesting point in our model is tying the features to the intention by an attention mechanism, and we intend to further investigate recurrent attention mechanisms (Xu et al., 2015) that could offer better generalization at inference time. F (Q, θ) =E Q log P (u 1:T , z|x 1:T ; θ) Q(z|x 1:T , u 1:T ) ≤E P (·|x 1:T ,u 1:T ;θ) [log P (y 1:T |x 1:T ,θ)] DISPLAYFORM0 where F is the Kullback Liebler divergence between P (u 1:T |z, x 1:T ; θ) and Q(z|u 1:T , x 1:T ) given as follows:F (Q, θ) = −D KL (Q||P (·|x 1:T , u 1:T ; θ)) + log P (y 1:T |x 1:T ,θ). Most importantly, it has also been shown in Theorem 2 of BID23 that if Q and θ form a pair of local maximizer to F , then θ is also a local maximum of the original likelihood maximization problem. To maximize F w.r.t Q, one has the closed form solution based on Bayes theorem: Q * (z|u 1:T , x 1:T ; θ old ) =P (z|y 1:T , x 1:T , θ) DISPLAYFORM1 Here, {z 1 , . . . , z N } is a sequence of latent random variables sampled i.i.d. from the distribution P (z).Given parameter θ, denoted by θ old , immediately the posterior distribution Q that maximizes F is given by: Q * (z|x 1:T , u 1:T ) = P (z|x 1:T , u 1:T ; θ old ). In this case, the above loss function is equivalent to the complete data log-likelihood * (θ, θ old ) := E P (·|u 1:T ,x 1:T ;θold) log P (x 1:T , z|u 1:T ; θ) P (z|x 1:T , u 1:T ; θ old ) , which is a lower bound of the log likelihood. Furthermore , if θ = θ old , then clearly * (θ old , θ old ) is equal to the log-likelihood log P (y 1:T |x 1:T ,θ old ).Tang & Salakhutdinov (2013) present a generalized EM algorithm to train a SNN. In the E-step, the following approximate posterior distribution is used:Q(z|u 1:T , x 1:T ; θ old ) :=r(z; x 1:T ,y 1:T , θ old )P (z), wherer (z; x 1:T ,y 1:T , θ old ) = r(z; x 1:T , u 1: DISPLAYFORM2 r(z i ; x 1:T , u 1:T , θ old ) is the the importance sampling weight.Recall that for our distribution model, r(z; x 1:T , u 1:T , θ old ) ∝ exp(−d(f (x, z; θ), u)), therefore we obtain that the importance weights correspond to a soft-max over the prediction error.In the M-step, the θ parameters are updated with the gradient vector with respect to the following optimization: θ ∈ arg max θ∈Θˆ (θ, θ old ), wherê DISPLAYFORM3 (z i ; x 1:T ,y 1:T , θ old ) log P (y 1:T , z i |x 1:T , θ)is the empirical expected log likelihood, andQ is the posterior distribution from the E-step. Here we drop the last term in F because in our case Q that does not depend on θ. Correspondingly, the gradient estimate is given by: DISPLAYFORM4 (z i )∇ θ log r(z i ; x 1:T , u 1:T , θ), the equality is due to the facts that log P (y 1:T , z|x 1:T , θ) = log r(z; x 1:T ,y 1:T , θ) + log P (z) and distribution P (z) is independent of θ.To better understand this estimator, we will analyze the bias and variance of the gradient estimator. Based on the construction of importance sampling weight, immediately the gradient estimator is consistent. Furthermore, under certain regular assumptions, the bias is O(N −1/2 ). (This means the gradient estimator is asymptotically unbiased.) Furthermore, the variance of this estimator is given by DISPLAYFORM5 where the integrand is given by v(z; θ) =r(z; x 1:T ,y 1:T , θ old ) · (∇ θ log r(z; x 1:T , u 1:T , θ)) 2 ≥ 0.
multi-modal imitation learning from unstructured demonstrations using stochastic neural network modeling intention.  A new sampling-based approach for inference in latent variable models that applies to multi-modal imitation learning and works better than deterministic neural networks and stochastic neural networks for a real visual robotics task. This paper shows how to learn several modalities using imitation learning from visual data using stochastic Neural Networks, and a method for learning from demonstrations where several modalities of the same task are given.