This paper is concerned with the robustness of VAEs to adversarial attacks. We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as β-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)). This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions. Unsupervised learning of disentangled latent variables in generative models remains an open research problem, as is an exact mathematical definition of disentangling (Higgins et al., 2018) . Intuitively, a disentangled generative model has a one-to-one correspondence between each input dimension of the generator and some interpretable aspect of the data generated. For VAE-derived models (Kingma & Welling, 2013; Rezende et al., 2014) this is often based around rewarding independence between latent variables. Factor VAE (Kim & Mnih, 2018) , β-TCVAE (Chen et al., 2018) and HFVAE (Esmaeili et al., 2019) have shown that the evidence lower bound can be decomposed to obtain a term capturing the degree of independence between latent variables of the model, the total correlation. By up-weighting this term, we can obtain better disentangled representations under various metrics compared to β-VAEs (Higgins et al., 2017a) . Disentangled representations, much like PCA or factor analysis, are not only human-interpretable but also offer more informative and robust latent space representations. In addition, information theoretic interpretations of deep learning show that having a disentangled hidden layer within a discriminative deep learning model increases robustness to adversarial attack (Alemi et al., 2017) . Adversarial attacks on deep generative models, more difficult than those on discriminative models (Tabacof et al., 2016; Gondim-Ribeiro et al., 2018; Kos et al., 2018) , attempt to fool a model into reconstructing a chosen target image by adding distortions to the original input image. Generally, the most effective attack mode involves making the latent-space representation of the distorted input match that of the target image (Gondim-Ribeiro et al., 2018; Kos et al., 2018) . This kind of attack is particularly relevant to applications where the encoder's output is used downstream. Projections of data from VAEs, disentangled or not, are used for tasks such as: text classification (Xu et al., 2017) ; discrete optimisation (Kusner et al., 2017) ; image compression (Theis et al., 2017; Townsend et al., 2019) ; and as the perceptual part of a reinforcement learning algorithm (Ha & Schmidhuber, 2018; Higgins et al., 2017b) , the latter of which uses a disentangled VAE's encoder to improve the robustness of the agent to domain shift. Here we demonstrate that β-TCVAEs are significantly more robust to 'latent-space' attack than standard VAEs, and are generally more robust to attacks that act to maximise the evidence lower bound for the adversarial input. The robustness of these disentangled models is highly relevant because of the use-cases for VAEs highlighted above. However, imposing additional disentangling constraints on a VAE training objective degrades the quality of resulting drawn or reconstructed images (Higgins et al., 2017a; Chen et al., 2018) . We sought whether more powerful, expressive models, can help ameliorate this and in doing so built Figure 1: Latent-space adversarial attacks on Chairs, 3D Faces and CelebA for different models, including our proposed Seatbelt-VAE. β = 10 for β-TCVAE, β-TCDLGM and Seatbelt-VAE. L is the number of stochastic layers. Clockwise within each plot we show the initial input, its reconstruction, the adversarial input, the adversarial distortion added to make it (shown normalised), the adversarial input's reconstruction, and the target image. Following Tabacof et al. (2016) ; Gondim-Ribeiro et al. (2018) we attack with different degrees of penalisation on the magnitude of the adversarial distortion; in choosing the distortion to show, we pick the one with the penalisation that resulted in the value of the attack objective just better than the mean. See Section 5 for more details. a hierarchical disentangled VAE, Seatbelt-VAE, drawing on works like Ladder VAEs (Sønderby et al., 2016) and BIVA (Maaløe et al., 2019) . We demonstrate that Seatbelt-VAEs are more robust to adversarial attacks than β-TCVAEs and β-TCDLGMs (the latter a simple generalisation we make of β-TC penalisation to hierarchical VAEs). See Figure 1 for a demonstration. Rather than being concerned with human-interpretable controlled generation by our models, which has been the focus of much research into disentangling, instead we are interested in the robustness afforded by disentangled representations. Thus our key contributions are: • A demonstration that β-TCVAEs are significantly more robust to adversarial attacks via their latents than vanilla VAEs. • The introduction of Seatbelt-VAE, a hierarchical version of the β-TCVAE, designed to further increase robustness to various types of adversarial attack, while also giving better perceptual quality of reconstructions even when regularised. We have presented the increases in robustness to adversarial attack afforded by β-TCVAEs. This increase in robustness is strongest for attacks via the latent space. While disentangled models are often motivated by their ability to provide interpretable conditional generation, many use cases for VAEs centre on the learnt latent representation of data. Given the use of these representations as inputs for other tasks, the latent attack mode is the most important to protect against. Recent work by Shamir et al. (2019) gives a constructive proof for the existence of adversarial inputs for deep neural network classifiers with small Hamming distances. The proof holds with deterministic defence procedures that work as additional deterministic layers of the networks, and in the presence of adversarial training (Szegedy et al., 2014; Ganin et al., 2016; Tramèr et al., 2018; Shaham et al., 2018) . Shamir et al. (2019) thus give a theoretical grounding for using stochastic methods to defend against adversarial inputs. As VAEs are already used to defend deep net classifiers (Schott et al., 2019; Ghosh et al., 2019) , more robust VAEs, like β-TCVAEs, could find use in this area. We introduce Seatbelt-VAE, a particular hierarchical VAE disentangled on the top-most layer with skip connections down to the decoder. This model further increases robustness to adversarial attacks, while also increasing the quality of reconstructions. The performance of our model under adversarial attack to robustness is mirrored in robustness to uncorrelated noise: these models are effective denoising autoencoders as well. We hope this work stimulates further interest in defending and attacking VAEs. (5) cf. Eq (7) in the main paper. The likelihood is conditioned on all z layers: Now we have: Apply βTC decomposition to T as in Chen et al. (2018) . j indexes over units in z L . (A.14) for our chosen generative model. As in Chen et al. (2018), we choose to weight T b , the total correlation for q φ (z L ), by a prefactor β. Giving us the ELBO for Seatbelt-VAEs, Eq (10).
We show that disentangled VAEs are more robust than vanilla VAEs to adversarial attacks that aim to trick them into decoding the adversarial input to a chosen target. We then develop an even more robust hierarchical disentangled VAE, Seatbelt-VAE. The authors propose a new VAE model called seatbelt-VAE, showing to be more robust for latent attack than benchmarks.