 the number of trials can be more important for determining coding accuracy than the number of neurons \cite{stringer2019high}.

Generative models of neuronal activity hold the promise of alleviating the above problem by enabling the synthesis of an unlimited number of realistic samples for assessing advanced analysis methods. Recently, the use of deep generative models on neuronal population spike train data has become increasingly popular. Latent Factor Analysis via Dynamical Systems (LFADS, \citet{pandarinath2018inferring}) uses the Variational Autoencoders framework to learn the population dynamics in latent representation (using recurrent neural networks) and extract 'denoised' single-trial firing rates from neural spiking data. Spike-GAN~\cite{molano-mazon2018synthesizing} demonstrated that GAN can model neural spikes that accurately match the statistics of real recorded spiking behaviour from a small number of neurons. Moreover, the discriminator in Spike-GAN is able to learn to detect which population activity pattern is the relevant feature, and this can provide insights into how a population of neurons encodes information. \citet{ramesh2019adversarial} trained a conditional GAN~\cite{mirza2014conditional}, conditioned on the stimulus, to generate multivariate binary spike trains. They fitted the generative model with recorded data in the V1 area of macaque visual cortex, and GAN generated spike trains were able capture the firing rate and pairwise correlation statistics better than the dichotomized Gaussian model and a deep supervised model.

All of the aforementioned generative models operate on population spike trains. Spike trains are discrete in nature meaning that they cannot be subject to any continuous increment or decrement. Hence it remains a difficult task to optimize deep generative models for discrete data with back-propagation, which is key for training deep neural networks.

Population spike trains can be obtained from different recording techniques, each having advantages and weaknesses. Electrophysiological recordings have high temporal resolution. However, this method is not without flaws~\cite{harris2016improving}. For instance, a single microelectrode can only detect activity from few neurons in close proximity, and extensive pre-processing is required to infer single-unit activity from a multi-unit signal. Disentangling circuit computations in neuronal populations of a large scale remains a difficult task, hence resulting in recordings with low spatial resolution but high temporal resolution~\cite{rey2015past}.
On the other hand, calcium imaging recordings have high spatial resolution and low temporal resolution~\cite{wei2019comparison}. This technique, which assesses changes in intracellular calcium concentration as a proxy for neuronal spiking activity, has become a powerful imaging technique to monitor large neuronal populations activity in \textit{vivo}. The continuous nature of calcium fluorescence signals makes optimization via back-propagation a much more straightforward task as compared to spike-train data. Thus, calcium imaging datasets are more attractive candidates for training generative models.

In this work, we explore the feasibility of using the Generative Adversarial Network (GAN) framework to synthesize calcium imaging data, as a method to scale-up or augment the amount of neuronal population activity data. We validate the method on artificial data with known ground-truth and we synthesize data mimicking real two-photon calcium (Ca$^{2+}$) imaging data as recorded from the primary visual cortex of a behaving mouse \cite{henschke2020reward, pakan2018impact}.

\section{Methods}

\subsection{Network architecture} \label{network_architecture}

The original generative adversarial network framework, introduced by \citet{NIPS2014_5423}, plays a min-max game where the generator $G$ attempts to generate convincing samples from the latent space $Z$, and the discriminator $D$ learns to distinguish between generated samples and real samples $X$.
The GAN framework has shown promising results across various domains, such as image and audio generation, unsupervised translation and many more \cite{karras2017progressive, donahue2018adversarial, zhu2017unpaired}. However, the originally proposed optimization objective was difficult to train, and was prone to mode collapse. Instead of the original objective of minimizing the Jensen-Shannon divergence between the original data distribution and generated data distribution, \citet{arjovsky2017wasserstein} proposed Wasserstein GAN (WGAN) which minimizes the smoother Earth-Mover's (1st Wasserstein) distance of the two data distributions. In WGAN, the weights of the discriminator (critic) were restricted within a predefined range (weight clipping) in order to enforce the 1-Lipschitz condition and to compute the Wasserstein distance. \citet{gulrajani2017improved} further improved the objective function with gradient penalty (WGAN-GP), instead of gradient clipping, in order to enforce the Lipschitz condition more effectively. In this work, we use the WGAN-GP formulation of the loss function without the need of incorporating any information of the neural activities into the objective function:
\begin{align*}
    \mathcal{L}_D = \underset{z \sim P_Z}{\mathbb{E}}[D(G(z))] - \underset{x \sim P_X}{\mathbb{E}}[D(x)] + \lambda \underset{\hat{x} \sim P_{\hat{X}}}{\mathbb{E}}[(\parallel \nabla_{\hat{x}}D(\hat{x}) \parallel_2 - 1)^2]
\end{align*}
where $\lambda$ denotes the gradient penalty coefficient and $\hat{X}$ are samples taken between the true data and generator's data distribution.

For the network architecture, we adapted the WaveGAN architecture by \citet{donahue2018adversarial}, which has shown promising results in audio signal generation. The generator uses 1-dimensional transpose convolution layers to up-sample the spatial dimension. Each transpose convolution layer was followed by batch normalization and Leaky ReLU activation. We added a dense layer with sigmoid activation as output layer in the generator. We also replaced Batch Normalization~\cite{ioffe2015batch} with Layer Normalization~\cite{ba2016layer} in order to make the operation compatible with the WGAN framework. Samples generated using transpose convolution often exhibit the "checkerboard" artifacts described by \citet{odena2016deconvolution}. In the context of signal generation, the discrimination could exploit the periodic artifacts pattern and learn a naive policy to reject generated samples. \citet{donahue2018adversarial} proposed the Phase Shuffle mechanism in the discriminator to address the aforementioned issue. The Phase Shift layer randomly shifts the activated units after each convolution layer by $-n$ to $n$ in the time domain, in order to distort the periodic pattern. Hence, the resulting samples constitute a more challenging task for the discriminator. In our network, we incorporated the Phase Shift operation, as well as using a kernel size that is divisible by the stride size, as suggested in \citet{odena2016deconvolution}. This led to a noticeable improvement in the generated samples. The discriminator is largely a mirror of the generator's architecture, except for layer normalization which we do not use in the discriminator. Instead, we apply the Phase Shuffle operation after each convolution layer.

To improve the model learning performance and stability, the calcium signals were scaled to the range between 0 and 1 by normalizing with the maximum training set calcium signal. Correspondingly, we chose sigmoid activation in the output layer of the generator and then re-scale the signals to its original range before inferring their spike trains. The model architecture can be found in Table~\ref{table:models_architecture}.

\subsection{CalciumGAN Training}

We trained both the generator and discriminator with the WGAN framework, with 5 discriminator update steps for each generator update step. We then used Adam optimizer \cite{kingma2014adam} to optimize both networks, with a learning rate of $\lambda=10^{-4}$, $\beta_{1}=0.9$ and $\beta_{2}=0.9999$. To speed up the training process, we implemented mixed precision training in our codebase. As a result, we are able to train our model with a batch size of 128 on a single NVIDIA RTX~2080~TI GPU. The exact hyper-parameters being used in this work can be found in Table~\ref{table:hyperparamteres}.

\subsection{Spike Train Analysis} \label{spike_train_anaylsis}

\begin{figure}[ht]
    \centering
    \includegraphics[width=1\linewidth]{calciumgan-pipeline.pdf}
    \caption{Pipeline diagram of a CalciumGAN analysis. White boxes illustrate data in different processing stages. Blue boxes illustrate analysis steps and techniques.}
    \label{fig:pipeline}
\end{figure}

We devised a consistent model analysis pipeline to evaluate the quality of samples generated by the model, as well as its ability to generalize, in the context of neuronal population spiking activities. The complete model analysis pipeline is shown in Figure~\ref{fig:pipeline}. 

Since we evaluate our model performance in terms of spike activities, we needed a deconvolution algorithm to infer the spike trains from calcium signals. In this work, we used the Online Active Set method to Infer Spikes (OASIS) deconvolution algorithm \cite{friedrich2017fast} for its fast online deconvolution performance.

We apply the Electrophysiology Analysis Toolkit (Elephant, \citet{elephant}) to