Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses. As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies. A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images. To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention. By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels. Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods. Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics. To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis. We introduced the idea of applying image synthesis to understanding and exploring cell-cell interactions in various and different contexts. To do so we use a protein attention based GAN, CCIGAN, which can provide accurate characterizations of cellular protein localization phenomena from conditioned counterfactual cell-cell scenarios. Additionally, the architecture of the attention module we propose can be generalized to other multiplexed datasets that require real world priors. Furthermore, CCIGAN outperforms a variety of current methods in biological modeling. We demonstrate this through biological consistency where CCIGAN recapitulates, discovers, and quantifies meaningful cellular interactions through 3 different experiments in a tumor environment unrecognized by other models. This highlights the potential for CCIGAN to identify cellular protein interactions which account for variation in patient responses to cancer therapy, providing a framework for biological hypotheses which explain clinical outcomes on a cellular level. where ResBlk is the residual block with skip connection used in ResNet (He et al., 2016) , and SPADE is the spatially-adaptive normalization layer. The detailed architecture of our discriminator is shown on Table 7 . For all baseline models, we use the architecture based on the their original implementation. Due to the size of the cell patch is (64, 64), we reduce the size of hidden layers to fit our dataset. For fair comparison, we use the same reduction of hidden layers and the same discriminator architecture for SPADE, pix2pixHD, and CCIGAN.
We explore cell-cell interactions across tumor environment contexts observed in highly multiplexed images, by image synthesis using a novel attention GAN architecture. A new method to model the data generated by multiplexed ion beam imaging by time-of-flight (MIBI-TOF) by learning the many-to-many mapping between cell types and protein markers' expression levels.