A key problem in neuroscience and life sciences more generally is that the data generation process is often best thought of as a hierarchy of dynamic systems. One example of this is in-vivo calcium imaging data, where observed calcium transients are driven by a combination of electro-chemical kinetics where hypothesized trajectories around manifolds determining the frequency of these transients. A recent approach using sequential variational auto-encoders demonstrated it was possible to learn the latent dynamic structure of reaching behaviour from spiking data modelled as a Poisson process. Here we extend this approach using a ladder method to infer the spiking events driving calcium transients along with the deeper latent dynamic system. We show strong performance of this approach on a benchmark synthetic dataset against a number of alternatives. In-vivo two-photon calcium imaging provides systems neuroscientists with the ability to observe the activity of hundreds of neurons simultaneously during behavioural experiments. Such highdimensional data is ripe for techniques identifying low-dimensional latent factors driving neural dynamics. The most common methods, such as principal components analysis, ignore non-linearity and temporal dynamics in brain activity. Pandarinath et al. (2018) [1] developed a new technique using deep, recurrent, variational auto-encoders which they named Latent Factor Analysis via Dynamical Systems (LFADS). Using LFADS they found non-linear, dynamic latent variables describing highdimensional activity in the motor cortex that can decode reaching behaviour with much higher fidelity than other methods. However, LFADS was designed for application to spiking data recorded from extracellular electrodes, not for two-photon calcium imaging data. Two-photon calcium imaging poses the additional problem of identifying latent spike trains in fluorescence traces. If we continue to model the frequency of events as being generated by a Poisson process, this can be seen as hierarchy of dynamic systems (Fig 1A) , in which low dimensional dynamics generate spike train probabilities that drive fluctuations in biophysical dynamics of calcium activity (Fig 1B. Here we propose a method that extends LFADS to accommodate calcium activity using this hierarchical dynamic systems approach.
We propose an extension to LFADS capable of inferring spike trains to reconstruct calcium fluorescence traces using hierarchical VAEs.