Stretched sinograms for limited-angle tomographic reconstruction with
  neural networks
We present a direct method for limited angle tomographic reconstruction using
convolutional networks. The key to our method is to first stretch every tilt
view in the direction perpendicular to the tilt axis by the secant of the tilt
angle. These stretched views are then fed into a 2-D U-Net which directly
outputs the 3-D reconstruction. We train our networks by minimizing the mean
squared error between the network's generated reconstruction and a ground truth
3-D volume. To demonstrate and evaluate our method, we synthesize tilt views
from a 3-D image of fly brain tissue acquired with Focused Ion Beam Scanning
Electron Microscopy. We compare our method to using a U-Net to directly
reconstruct the unstretched tilt views and show that this simple stretching
procedure leads to significantly better reconstructions. We also compare to
using a network to clean up reconstructions generated by backprojection and
filtered backprojection, and find that this simple stretching procedure also
gives lower mean squared error on previously unseen images.
Kyle Luther, Sebastian Seung
# Stretched sinograms for limited-angle tomographic reconstruction with neural networks

###### Abstract

We present a direct method for limited angle tomographic reconstruction using convolutional networks. The key to our method is to first stretch every tilt view in the direction perpendicular to the tilt axis by the secant of the tilt angle. These stretched views are then fed into a 2-D U-Net which directly outputs the 3-D reconstruction. We train our networks by minimizing the mean squared error between the network's generated reconstruction and a ground truth 3-D volume. To demonstrate and evaluate our method, we synthesize tilt views from a 3-D image of fly brain tissue acquired with Focused Ion Beam Scanning Electron Microscopy. We compare our method to using a U-Net to directly reconstruct the unstretched tilt views and show that this simple stretching procedure leads to significantly better reconstructions. We also compare to using a network to clean up reconstructions generated by backprojection and filtered backprojection, and find that this simple stretching procedure also gives lower mean squared error on previously unseen images.

## 1 Introduction

Electron tomography is an imaging technique that uses transmission electron microscope images acquired from multiple viewpoints to reconstruct the 3-D structure of an object [5]. Linear reconstruction techniques like filtered backprojection (FBP) or iterative reconstruction methods (SART,SIRT) are widely used but display strong artifacts in the presence of limited tilt angles, highly noisy inputs, and complex nonlinear misalignments, all of which are commonplace in electron tomography workflows [16, 33, 15].

**Problem setting** Our goal is to use deep learning to perform reconstruction of electron tomography tilt series. Ideally the implicit prior contained in a neural network will enable higher quality reconstructions using fewer tilts than would be required by classical reconstruction methods like filtered backprojection.

**Related methods** The related fields of X-Ray and positron emission tomography have seen a proliferation of deep learning methods to perform tomographic reconstruction. [28] categorize and review several distinct ways that neural networks have been used to aid in tomographic reconstruction of X-Ray CT data and we review the most relevant methods below.
_Domain transform_ methods use a neural network \(f\) to map sinogram data \(\mathbf{y}\) to a reconstruction \(\hat{\mathbf{x}}=f(\mathbf{y})\). Perhaps the simplest approach from this category is the work of [7] which uses a convolutional network to directly map sinograms to reconstructions. A related approach was used by [29] which used hybrid fully connected/convolutional networks. [34] used fully connected layers and added an additional manifold encoding-decoding strategy.

Notably these approaches have not shown the same level of interest as competing methods which we review below and [31] argue that the generalization performance of domain transform methods has been lackluster in addition to them typically having rather large computational and memory requirements.

_Image domain_ methods apply neural networks to the output of a classic reconstruction method, typically either backprojection or filtered backprojection. These methods can be thought of as neural post-processing. [10, 2, 20] demonstrate that deep networks can improve the quality of reconstructions that have been generated from low-dose (noisy) views. [9] show that artifacts caused by reconstruction from sparse viewpoints can be removed, and [1] show that artifacts caused by a limited-angle set of views can be removed using deep networks to post-process reconstructions. [32] show that post-processing can be even used to remove metal artifacts. A hybrid method was used in [31, 12] where the first step was to map a sinogram of size nview \(\times\) width to a sequence of individual backprojections of size nview \(\times\) height \(\times\) width.

_Sensor domain_ methods instead apply neural networks to the raw sinograms and are typically only used as a pre-processing step so that classical reconstruction methods are applied to the network outputs. Networks have been shown to remove artifacts [3, 6]. Networks have also been used to inpaint missing views from limited-angle sinograms [21].

_Dual domain_ methods combine both _sensor_ and _image_ domain approaches and apply networks both before and after backprojection [13, 27, 26].

_Dictionary-based reconstruction of affine-aligned tilt views_ On a seemingly unrelated front, dictionary based reconstruction with sparsity priors was applied to electron tomographic reconstruction [8, 25]. Critically, these works first performed an _affine_ alignment to tilt views before reconstruction. Using an affine alignment is non-standard as it stretches every view instead of just translating them. However, both works showed impressive reconstructions when combining the affine-stretched views and a translationally invariant dictionary.

**What is missing?** With the exception of [7], prior neural methods either rely on fully connected architectures [29, 34] or backprojection to actually perform the reconstruction. Our goal is generate quality reconstructions with the widely used U-Net architecture [18] and avoid the use of backprojection.

**Our contributions** We propose a simple backprojection-free pre-processing scheme that significantly improves the ability of a convolutional network to perform tomographic reconstructions. We simply stretch each view in the sinogram by \(\sec\theta\) along the direction perpendicular to the tilt axis. Our method is applicable to parallel beam, limited angle geometries that are typical in electron tomography.
## 2 Method: neural reconstruction of stretched sinograms

Our training and reconstruction procedures are shown in Fig. 1.

### Stretched sinograms

**Notation and assumptions** We assume we are working with a parallel beam geometry and that we have \(n_{\mathrm{view}}\) tilt views of size \(n_{h}\times n_{w}\) over a limited range of angles \((-\theta,+\theta)\) where \(\theta<90^{\circ}\). We denote the 3-D tensor of 2-D tilt views by \(\mathbf{y}\in\mathbb{R}^{n_{\mathrm{view}}\times n_{h}\times n_{w}}\). We assume these views have been log-normalized so that they are related to the object density \(\mathbf{x}\in\mathbb{R}^{n_{d}\times n_{h}\times n_{w}}\) via the Radon transform:

\[\mathbf{y}=P\mathbf{x} \tag{1}\]

**Generating stretched views** To generate the stretched sinogram, we simply stretch every tilt view by \(\sec\theta\) in the direction perpendicular to the tilt axis. This stretching is performed so that the image size is preserved, meaning the stretched sinograms are also \(n_{h}\times n_{w}\) in size. Bilinear interpolation is used to perform the stretching and the stretching is done treating the center of each image as the origin.

If the tilt axis is parallel to the \(\hat{y}\) axis, this means we stretch along the \(\hat{x}\)-axis of each 2D tilt view. We now write the formula for tilt stretching in this case.

Figure 1: Method overview **Inference**: we first stretch every tilt view by \(\sec\theta\) in the direction perpendicular to the tilt axis, then perform reconstruction with a U-Net **Training**: We train our networks using simulated tilt views and mean squared error (MSE) between network outputs and ground truth volumes
Let \(i,j\) index the rows,columns of a tilt image. We use natural coordinates so \((-1,-1)\) refers to the upper left and \((+1,+1)\) refers to the bottom right corner of the image. Bilinear interpolation is used to evaluate at fractional pixel locations.

\[y^{\rm stretched}(\theta,i,j)=y\left(\theta,i,j\sec\theta\right) \tag{2}\]

This simple stretching procedure can be extended to stacks with arbitrary tilt axes (e.g. dual tilt setups [14]). The formula in Eq. 2 would need change to stretch along a linear combination of \(i,j\). We can write the stretching procedure in matrix notation, treating bilinear interpolation as a sparse linear operator \(S\):

\[{\bf y}^{\rm stretched}=S{\bf y} \tag{3}\]

This operation maps an \(n_{\rm view}\times n_{h}\times n_{w}\) tensor of raw views to an \(n_{\rm view}\times n_{h}\times n_{w}\) tensor of stretched views.

**Visualizing the stretched views** In Fig. 2 we compare a simulated sinogram and its corresponding stretched sinogram. We simulate the sinogram by applying the Radon transform to a \(45\times 512\times 512\) volume of FIB-SEM data of fly brain tissue [30; 19] which is further described in Section 3. Specifically we compute 45 projections uniformly spaced over the angles \((-45^{\circ},+45^{\circ})\) with a tilt axis in the \(\hat{y}\) direction. We show the \(x\theta\) view for \(y=1\) and the \(xy\) view for \(\theta=-45^{\circ}\) for both the sinogram and stretched sinogram.

Figure 2: Sinogram vs stretched sinogram representations. The sinogram is generated with a vertical tilt axis and with tilt angles spaced at \(2^{\circ}\) increments between \(-45^{\circ}\) and \(+45^{\circ}\). The stretched sinogram is by stretching each \(xy\) view of the sinogram by \(\sec\theta\) (Eq. 2). In the top row we show an \(x\theta\)-slice of the raw/stretched sinograms. In the bottom row we show \(xy\) slices corresponding to \(\theta=-45^{\circ}\).
### 3-D Reconstruction with a 2-D U-Net

For simplicity we stick closely to the original 2-D U-Net architecture proposed in [18]. More details are provided in the Appendix. We treat the stretched sinogram with \(K\) tilt views as a 2-D image with \(K\) input channels. We generate a 3-D reconstruction from the 2-D U-Net by treating the output channel dimension as the depth dimension of the output volume. To improve training speed we use Instance Normalization layers before every ReLU layer [22]. We found that the more popular Batch Normalization operation gave significantly higher test time error, likely due to the fact that we train our U-Nets with a batch size of 1.

### Supervised network training

We assume we have paired examples of tilts \(\mathbf{y}\) and volumes \(\mathbf{x}\). We train our networks using stochastic gradient descent applied to the mean squared error (MSE) objective between these volumes and network reconstructions. Specifically, we generate the network reconstruction \(\hat{\mathbf{x}}=f_{\omega}(S\mathbf{y})\) using a U-Net applied to the stretched tilt views. We then compute and backpropagate through the MSE:

\[l(\omega)=\frac{1}{n_{oixel}}\left\|\mathbf{x}-f_{\omega}(S\mathbf{y})\right\| ^{2} \tag{4}\]

where \(n_{pixel}=n_{d}\times n_{h}\times n_{w}\) is the number of pixels in the output reconstructions.

## 3 Experiments with simulated data

### Data and simulated tilt views

We download three \(1k\times 1k\times 1k\) voxel volumes to use as the train, validation, and test sets from the 3D isotropic dataset imaged at \(8\times 8\times 8\) nm\({}^{3}\) resolution by [30]. This dataset contains images of a fruit-fly brain imaged with a focused ion

Figure 3: \(128\times 128\) sized crop from an \(xy\) view of a test set reconstruction. Directly using a U-Net to perform reconstruction or applying a U-Net to clean up FBP causes a neuron boundary to disappear.
beam milling and scanning electron microscope imaging (FIB-SEM) technique. Slices from this dataset are shown in the Appendix.

We sample patches of size \(32\times 512\times 512\) voxels at arbitrary positions from the \(1k\times 1k\times 1k\) voxel training volume. These patches are normalized to be zero-mean, unit-variance. From these patches, simulated tilt series consisting of 8 tilt views ranging uniformly over \((-60^{\circ},+60^{\circ})\) are generated using the ASTRA toolbox [24, 23]. The input shape to our networks is therefore \(8\times 512\times 512\).

For data augmentation we add Gaussian noise with standard deviation of 0.3 times the standard deviation of the projections and randomly shift a chosen number \(n\) of the 8 tilt views in both \(x\) and \(y\) directions by an integer number of pixels between \([-3,+3]\) to simulate misalignments that can occur in real tomographic tilt series. We compare results for when we misalign \(n=0,2,4,8\) of the sections. Each tilt view is then normalized to be unit mean and zero variance after augmentation.

### Comparisons

We compare our method representation to 5 other methods: (1) using the raw sinograms as input to a U-Net, (2) using backprojection as input to a U-Net, (3) using filtered backprojection as input to a U-Net (the approach of [9]), (4) using backprojection without U-Net postprocessing, (5) using filtered backprojection without U-Net postprocessing. We use the ASTRA Toolbox to implement both the backprojection and filtered backprojection operations. For filtered backprojection, we use the default _ram-lak_ filter which implements a ramp-function in frequency space. In the case of backprojection and filtered-backprojection, the depth dimension is treated as the channel index to the 2D U-Nets and the channel dimensionality is 32 instead of 8.

### Training details

The forward and backward projections were implemented on a CPU using the ASTRA toolbox and these were distributed across 64 cpus to speed up inference time. We use a single ASUS RTX3090 Turbo 24GB GPU for inference and backpropagation. Networks were trained using PyTorch [17]. We use the Adam optimizer to train our U-Nets [11]. We trained all networks for up to 72 hours with a step-size of 0.001. Learning curves for all configurations are shown in the Appendix. We used early stopping computed on the validation set so that we perform test-set evaluation using the network with lowest validation set error.

### Results

In Fig. 3 we show a qualitative comparison between various methods for one \(xy\) slice of the reconstruction on the test set (in the setting where 4/8 of the tilt views were misaligned). We evaluate our networks on a \(992\times 1000\times 1000\) test set volume extracted from a neighboring location of FIB-SEM images of fly brain
tissue. This volume is broken into 124 (almost) non-overlapping patches3 of size \(32\times 512\times 512\) and tilt views are simulated and subsequently reconstructed.

Footnote 3: There is a small sliver of width 24 pixels where the patches overlap due to the fact that we are extracting \(512\times 512\) patches from \(1000\times 1000\) sections

**Vary level of noise** In this setup we use networks trained on non-misaligned data and do not misalign tilts at inference time either. Here we vary the level of noise at inference time. We observe that as we add noise, the MSE of plain filtered backprojection dramatically increases. All neural methods improve over both classical methods. We observe that not stretching the inputs before inference with a U-Net gives nearly 50% higher mean squared error than our method. We observe that stretching provides the lowest MSE with _bp+U-Net_ closesly following behind. The results are shown in Tab. 1.

**Vary level of misalignments**

In this setup we train and test networks with various number of misaligned tilts. All neural methods improve over both classical methods. We observe that stretching the tilts before inference gives much lower MSE than not stretching. This time, we observe a larger gap between _stretch+U-Net_ compared to _bp+U-Net_ or _fbp+U-Net_ as we increase the number of misaligned sections. The results are shown in Tab. 2.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline noise & direct U-Net & stretch+U-Net & fbp+U-Net & bp+U-Net & fbp & bp \\ \hline
0.0 & \(0.145\pm 0.001\) & \(\mathbf{0.092\pm 0.001}\) & \(0.120\pm 0.001\) & \(0.097\pm 0.001\) & \(0.389\pm 0.003\) & \(0.451\pm 0.005\) \\
0.1 & \(0.146\pm 0.002\) & \(\mathbf{0.093\pm 0.001}\) & \(0.119\pm 0.001\) & \(0.098\pm 0.001\) & \(0.637\pm 0.003\) & \(0.452\pm 0.005\) \\
0.2 & \(0.150\pm 0.002\) & \(\mathbf{0.098\pm 0.001}\) & \(0.122\pm 0.001\) & \(0.103\pm 0.001\) & \(0.996\pm 0.004\) & \(0.454\pm 0.005\) \\
0.3 & \(0.157\pm 0.002\) & \(\mathbf{0.104\pm 0.001}\) & \(0.129\pm 0.001\) & \(0.109\pm 0.001\) & \(1.244\pm 0.004\) & \(0.458\pm 0.005\) \\
0.4 & \(0.166\pm 0.002\) & \(\mathbf{0.114\pm 0.001}\) & \(0.140\pm 0.001\) & \(0.119\pm 0.001\) & \(1.403\pm 0.004\) & \(0.462\pm 0.005\) \\ \hline \end{tabular}
\end{table}
Table 1: Test set MSE as we vary the level of noise added to tilts at inference time. In each setting we reconstruct \(32\times 512\times 512\) volumes from 8 tilt views of size \(512\times 512\) with tilt angles uniformly spaced between \([-60^{\circ},+60^{\circ}]\).

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline n shifts & direct U-Net & stretch+U-Net & fbp+U-Net & bp+U-Net & fbp & bp \\ \hline
0/8 & \(0.157\pm 0.002\) & \(\mathbf{0.104\pm 0.001}\) & \(0.129\pm 0.001\) & \(0.109\pm 0.001\) & \(1.244\pm 0.004\) & \(0.458\pm 0.005\) \\
2/8 & \(0.217\pm 0.002\) & \(\mathbf{0.118\pm 0.001}\) & \(0.157\pm 0.001\) & \(0.130\pm 0.001\) & \(1.274\pm 0.004\) & \(0.479\pm 0.001\) \\
4/8 & \(0.226\pm 0.003\) & \(\mathbf{0.121\pm 0.001}\) & \(0.166\pm 0.002\) & \(0.135\pm 0.001\) & \(1.304\pm 0.004\) & \(0.481\pm 0.005\) \\
8/8 & \(0.235\pm 0.002\) & \(\mathbf{0.126\pm 0.001}\) & \(0.180\pm 0.001\) & \(0.164\pm 0.002\) & \(1.367\pm 0.004\) & \(0.508\pm 0.006\) \\ \hline \end{tabular}
\end{table}
Table 2: Test set MSE as we vary the number of misaligned tilt views (both at train and test time). In each setting we reconstruct \(32\times 512\times 512\) volumes from 8 tilt views of size \(512\times 512\) with tilt angles uniformly spaced between \([-60^{\circ},+60^{\circ}]\).
## 4 Discussion

Why should using the stretched sinogram as network input lead to lower reconstruction error than backprojection or filtered backprojection? Both BP and FBP are reconstruction algorithms so intuitively it might seem like cleaning up a reconstruction would be easier to generating one from scratch like networks do in our method. Our experiments suggest however that this intuition is wrong. We speculate that it is because backprojection _attenuates_ high frequency modes while filtered backprojection _amplifies_ in the input. Stretching on the other hand, may suppress less information in the input which may be helpful for data-driven learning.

**Limitations** In this proof of concept, our tilt views were synthesized from a 3-D image, which could serve as ground truth for the desired reconstruction. In the real world, where would the ground truth for supervised training come from? One possible scenario is that conventional tomography would be used to generate the ground truth for convolutional net tomography. One would first acquire a high quality, densely sampled, low noise, well-aligned set of tilt views. These views given to classic reconstruction methods like FBP or SIRT to generate 3D reconstructions. Neural nets would then be trained to perform tomography from subsets of the tilt views (possibly with additional noise and misalignment augmentations). Once trained, the nets could be applied in large scale tomographic imaging pipelines, where there is a strong motivation to reduce the number of tilt views and/or acquisition time of each image.

We have relied on mean squared error as the primary metric for evaluating and comparing methods. In cell biology, the goal is to learn something new about the structure of the object being reconstructed. In other fields like connectomics, a central goal is identify boundaries between cells [4]. Neither of these goals is directly linked to MSE. We have attempted to validate our reconstruction via qualitative inspections and have indeed seen cases where membranes (in particular membranes that lie in the imaging plane) are washed out. This suggests that downstream segmentation of neurites may indeed benefit via stretched sinogram representations. However this point must be remembered when interpreting the results of this study.

**Extensions** As discussed in the limitations section a central goal is to use tomography as part of a broader pipeline. Many pipelines in the literature already use deep neural networks to generate segmentations of electron microscope images. It seems reasonable to do away with explicit reconstruction altogether and instead directly map tilt views to segmentation. This has the bonus of avoiding any questions regarding MSE, since the network would be directly optimizing the quantity of interest: namely the segmentation performance.
