For many applications, in particular in natural science, the task is to
 determine hidden system parameters from a set of measurements. Often,
 the forward process from parameter- to measurement-space is well-defined,
 whereas the inverse problem is ambiguous: multiple parameter sets can
 result in the same measurement. To fully characterize this ambiguity, the full
 posterior parameter distribution, conditioned on an observed measurement,
 has to be determined. We argue that a particular class of neural networks
 is well suited for this task – so-called Invertible Neural Networks (INNs).
 Unlike classical neural networks, which attempt to solve the ambiguous
 inverse problem directly, INNs focus on learning the forward process, using
 additional latent output variables to capture the information otherwise
 lost. Due to invertibility, a model of the corresponding inverse process is
 learned implicitly. Given a specific measurement and the distribution of
 the latent variables, the inverse pass of the INN provides the full posterior
 over parameter space. We prove theoretically and verify experimentally, on
 artificial data and real-world problems from medicine and astrophysics, that
 INNs are a powerful analysis tool to find multi-modalities in parameter space,
 uncover parameter correlations, and identify unrecoverable parameters. When analyzing complex physical systems, a common problem is that the system parameters of interest cannot be measured directly. For many of these systems, scientists have developed sophisticated theories on how measurable quantities y arise from the hidden parameters x. We will call such mappings the forward process. However, the inverse process is required to infer the hidden states of a system from measurements. Unfortunately, the inverse is often both intractable and ill-posed, since crucial information is lost in the forward process.To fully assess the diversity of possible inverse solutions for a given measurement, an inverse solver should be able to estimate the complete posterior of the parameters, conditioned on an observation. This makes it possible to quantify uncertainty, reveal multi-modal distributions, and identify degenerate and unrecoverable parameters -all highly relevant for applications in natural science. In this paper, we ask if invertible neural networks (INNs) are a suitable model class for this task. INNs are characterized by three properties:(i ) The mapping from inputs to outputs is bijective, i.e. its inverse exists, ( ii) both forward and inverse mapping are efficiently computable, and (iii) both mappings have a tractable Jacobian, which allows explicit computation of posterior probabilities.Networks that are invertible by construction offer a unique opportunity: We can train them on the well-understood forward process x → y and get the inverse y → x for free by The standard direct approach requires a discriminative, supervised loss (SL) term between predicted and true x, causing problems when y → x is ambiguous. Our network uses a supervised loss only for the well-defined forward process x → y. Generated x are required to follow the prior p(x) by an unsupervised loss (USL), while the latent variables z are made to follow a Gaussian distribution, also by an unsupervised loss. See details in Section 3.3.running them backwards at prediction time. To counteract the inherent information loss of the forward process, we introduce additional latent output variables z, which capture the information about x that is not contained in y. Thus, our INN learns to associate hidden parameter values x with unique pairs [y, z] of measurements and latent variables. Forward training optimizes the mapping [y, z] = f (x) and implicitly determines its inverse x = f −1 (y, z) = g(y, z). Additionally, we make sure that the density p(z ) of the latent variables is shaped as a Gaussian distribution. Thus , the INN represents the desired posterior p(x | y) by a deterministic function x = g(y, z) that transforms ("pushes") the known distribution p(z) to x-space, conditional on y.Compared to standard approaches (see FIG0 , left), INNs circumvent a fundamental difficulty of learning inverse problems: Defining a sensible supervised loss for direct posterior learning is problematic since it requires prior knowledge about that posterior's behavior, constituting a kind of hen-end-egg problem. If the loss does not match the possibly complicated (e.g. multimodal) shape of the posterior, learning will converge to incorrect or misleading solutions. Since the forward process is usually much simpler and better understood, forward training diminishes this difficulty. Specifically , we make the following contributions:• We show that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically in the asymptotic limit of zero loss, and practically on synthetic and real-world data from astrophysics and medicine.• The architectural restrictions imposed by invertibility do not seem to have detrimental effects on our network's representational power.• While forward training is sufficient in the asymptotic limit, we find that a combination with unsupervised backward training improves results on finite training sets.• In our applications, our approach to learning the posterior compares favourably to approximate Bayesian computation (ABC) and conditional VAEs. This enables identifying unrecoverable parameters, parameter correlations and multimodalities. We have shown that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically and practically on problems from medicine and astrophysics. We share the excitement of the application experts to develop INNs as a generic tool, helping them to better interpret their data and models, and to improve experimental setups. As a side effect, our results confirm the findings of others that the restriction to coupling layers does not noticeably reduce the expressive power of the network.In summary, we see the following fundamental advantages of our INN-based method compared to alternative approaches: Firstly, one can learn the forward process and obtain the (more complicated) inverse process 'for free', as opposed to e.g. cGANs, which focus on the inverse and learn the forward process only implicitly. Secondly, the learned posteriors are not restricted to a particular parametric form, in contrast to classical variational methods. Lastly, in comparison to ABC and related Bayesian methods, the generation of the INN posteriors is computationally very cheap. In future work, we plan to systematically analyze the properties of different invertible architectures, as well as more flexible models utilizing cycle losses, in the context of representative inverse problem. We are also interested in how our method can be scaled up to higher dimensionalities, where MMD becomes less effective. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In CVPR, pages 2223-2232, 2017a.Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pages 465-476, 2017b. DISPLAYFORM0 In Eq. 9, the Jacobians cancel out due to the inverse function theorem, i.e. the Jacobian DISPLAYFORM1 is trained as proposed, and both the supervised loss DISPLAYFORM2 and the unsupervised loss L z = D q(y, z), p(y ) p(z ) reach zero, sampling according to Eq. 1 with g = f −1 returns the true posterior p(x | y * ) for any measurement y * .Proof : We denote the chosen latent distribution as p Z (z), the distribution of observations as p Y (y), and the joint distribution of network outputs as q(y, z). As shown by BID14 , if the MMD loss converges to 0, the network outputs follow the prescribed distribution: DISPLAYFORM3 Suppose we take a posterior conditioned on a fixed y * , i.e. p(x | y * ), and transform it using the forward pass of our perfectly converged INN. From this we obtain an output distribution q * (y, z). Because L y = 0, we know that the output distribution of y (marginalized over z) must be q * (y) = δ(y − y * ). Also, because of the independence between z and y in the output, the distribution of z-outputs is still q * (z) = p Z (z) . So the joint distribution of outputs is DISPLAYFORM4 When we invert the network, and repeatedly input y * while sampling z ∼ p Z (z), this is the same as sampling [y, z] from the q * (y, z) above. Using the Lemma from above, we know that the inverted network will output samples from p(x | y * ).Corollary: If the conditions of the theorem above are fulfilled, the unsupervised reverse loss L x = D q(x), p X (x) between the marginalized outputs of the inverted network, q(x), and the prior data distribution, p X (x), will also be 0. This justifies using the loss on the prior to speed up convergence, without altering the final results.Proof: Due to the theorem, the estimated posteriors generated by the INN are correct, i.e. q(x | y * ) = p(x | y * ). If they are marginalized over observations y * from the training data, then q(x) will be equal to p X (x) by definition. As shown by BID14 , this is equivalent to L x = 0.
To analyze inverse problems with Invertible Neural Networks The author proposes to use invertible networks to solve ambiguous inverse problems and suggest to not only train the forward model, but also the inverse model with an MMD critic. The research paper proposes an invertible network with observations for posterior probability of complex input distributions with a theoretical valid bidirectional training scheme.
