# Accelerated deep self-supervised ptycho-laminography for three-dimensional
nanoscale imaging of integrated circuits

Iksung Kang  1,8 Yi Jiang  2 Mirko Holler  3 Manuel Guizar-Sicairos  3,4 A. F.
J. Levi  5 Jeffrey Klug  2 Stefan Vogt  2,9 and George Barbastathis6,7,10
<EMAIL_ADDRESS><EMAIL_ADDRESS>1Department of Electrical Engineering and Computer
Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
2Argonne National Laboratory, Lemont, IL 60439, USA
3Paul Scherrer Institut, Forschungsstrasse 111, 5232 Villigen PSI, Switzerland
4Institute of Physics (IPHYS), Ecole Polytechnique Fédérale de Lausanne, Rte
Cantonale, Lausanne 1015, Switzerland
5Department of Electrical and Computer Engineering, University of Southern
California, Los Angeles, CA 90007, USA
6Department of Mechanical Engineering, Massachusetts Institute of Technology,
Cambridge, MA 02139, USA
7Singapore-MIT Alliance for Research and Technology (SMART) Centre, 1 Create
Way, Singapore 117543, Singapore
8Present address: Department of Molecular and Cell Biology, University of
California, Berkeley, California 94720, USA
9 10

###### Abstract

Three-dimensional inspection of nanostructures such as integrated circuits is
important for security and reliability assurance. Two scanning operations are
required: ptychographic to recover the complex transmissivity of the specimen;
and rotation of the specimen to acquire multiple projections covering the 3D
spatial frequency domain. Two types of rotational scanning are possible:
tomographic and laminographic. For flat, extended samples, for which the full
180 degree coverage is not possible, the latter is preferable because it
provides better coverage of the 3D spatial frequency domain compared to
limited-angle tomography. It is also because the amount of attenuation through
the sample is approximately the same for all projections. However, both
techniques are time consuming because of extensive acquisition and computation
time. Here, we demonstrate the acceleration of ptycho-laminographic
reconstruction of integrated circuits with $16$-times fewer angular samples
and $4.67$-times faster computation by using a physics-regularized deep self-
supervised learning architecture. We check the fidelity of our reconstruction
against a densely sampled reconstruction that uses full scanning and no
learning. As already reported elsewhere [Zhou and Horstmeyer, Opt. Express,
28(9), pp. 12872-12896], we observe improvement of reconstruction quality even
over the densely sampled reconstruction, due to the ability of the self-
supervised learning kernel to fill the missing cone.

††journal: opticajournal††articletype: Research Article

##  1 Introduction

Hard X-rays offer non-destructive visualization and metrology of nanoscopic
details inside complex structures, such as Integrated Circuits (ICs). Short
wavelength and long penetration depth of X-rays enable probing into the
volumetric interiors of ICs. X-ray imaging instruments often incorporate the
object rotation with respect to the X-ray illumination to improve the depth
resolution of reconstructions. It is desirable to select the rotation geometry
by carefully considering the objects’ geometrical properties. For instance,
for flat extended nanostructures like ICs, the object’s rotation axis can be
oblique to the direction of synchrotron X-rays, i.e. laminographic imaging [1,
2]. For flat, extended samples, the oblique geometry keeps the amount of X-ray
absorption and scattering by the structures approximately the same regardless
of the object rotation, so that the volumetric interiors of ICs can be more
reliably reconstructed. This makes a clear distinction from existing
tomographic imaging methods [3, 4, 5, 6], where the strengths of absorption
would vary across different rotation angles.

On the other hand, translational scanning of the object enables a larger field
of view reconstruction. Originally proposed for Scanning Transmission Electron
Microscopy (STEM), ptychography leverages lateral movement of either the
object or the illumination to acquire several diffraction patterns from
different lateral locations in order to add robustness to phase retrieval [7,
8, 9, 10] and to reconstruct a larger field-of-view object [11, 12, 13].
Objects are computationally retrieved from the ptychographic measurements by
some well-established algorithms, such as ptychographic iterative engine (PIE)
[11], difference map (DM) [14], least-square maximum likelihood (LSQ-ML) [15],
etc. Alternatively, ptychography may also be conducted in the Fourier domain
by replacing the illumination with a set of plane waves incident at different
angles [16, 17, 18].

Figure 1: Accelerated deep self-supervised ptycho-laminography. (A) Ptycho-
laminographic imaging geometry. Synchrotron X-rays illuminate a sample of
integrated circuits in the ptycho-laminography geometry, with the sample
rotating around the oblique laminographic axis and scanned over a few thousand
angles. For each ptychography scan, the sample is laterally scanned at several
hundred different locations. (B) Equivalent imaging geometry. Forward
operators $H_{n}\>(n=1,2,\cdots,N)$ are defined according to each
laminographic rotation. (C) Proposed physics-informed machine learning
framework. Our pre-processor translates experimental ptycho-laminographic
measurement from the detector plane to the sample domain with minimal
processing using a ptychographic reconstruction algorithm. ADePt generates a
three-dimensional image of integrated circuit morphology from the pre-
processed projections throughout the optimization process. (D) Deep neural
network architecture for self-supervised learning. The proposed architecture
is essentially an encoder-decoder convolutional neural network with skip
connections and receiving random noise as input. The output is the image. Code
is publicly available at https://github.com/iksungk/ADePt.

As ICs have a flat geometry and an extended field of view, ptychography
becomes synergistic with laminography in imaging such nanostructures. Holler
et al demonstrated ptychographic X-ray laminography on ICs fabricated with
$16$-nm technology with $18.9$-nm resolution [19] using Laminographic Nano-
Imaging instrument (LamNI) [20], whose reconstruction quality easily surpassed
that of X-ray ptychographic tomography [6]. In a typical X-ray ptycho-
laminographic imaging apparatus as depicted in Fig. 1A, synchrotron X-rays
obliquely illuminate ICs with the angle of $\theta$ between the direction of
X-ray propagation and the rotation axis. ICs are scanned from a few thousand
laminographic angular views, where angular sampling depends on the sample
thickness, the resolution, and the laminographic angle [19], and for each
ptychography scan, X-rays laterally scan the ICs at several-hundred different
lateral positions.

Three-dimensional information of nanostructures is retrieved using a two-step
iterative update process based on our ptycho-laminographic measurements, i.e.
the reconstruction from the densely-sampled dataset (or densely-sampled
reconstruction): (1) a thousand iterations of a ptychographic reconstruction
algorithm processes hundreds of diffraction patterns to get a projection for
each of the $2000$ ptychograms (More details about the reconstruction in Sec.
2.2.); and (2) a volumetric reconstruction is formed by the laminographic
synthesis of all projections. The missing cone in the Fourier domain is filled
in during the post-processing step. The densely sampled reconstruction,
however, results in a long data acquisition and computation time due to the
strict angular sampling requirement [19] of ptychograms and the iterative
reconstruction of projections.

Here, we demonstrate up to an aggregate $9.57$-fold time savings in X-ray
ptycho-laminographic reconstruction by using a physics-regularized deep self-
supervised learning architecture, called Accelerated Deep self-supervised
Ptycho-laminography (ADePt). We achieved this acceleration by reducing the
number of angular samples by $16$ times and the computation time by $4.67$
times. Considering the memory limitations in GPU, ADePt yields the
reconstruction of $4.36\times 4.36\times 3.92\mbox{-}\mu\mathrm{m}^{3}$ ICs
within two hours, see Table S1 for more timing details. We provide both
quantitative and qualitative comparisons on the performance of ADePt with the
densely sampled reconstruction using Bit-Error Ratio (BER) and three-
dimensional power spectral density. Finally, we observe that the self-
supervised learning kernel fills missing cones from much fewer number of
projections compared to the densely sampled reconstruction.

ADePt consists of two components: a lightweight pre-processor and a deep
neural network as illustrated in Fig. 1C. The pre-processor consists of a few
iterations of maximum likelihood ptychography, projection matching alignment,
and center cropping, and works as an approximate inverse operator on
diffraction patterns. The network parameterizes a three-dimensional structure
of ICs by considering both network-structure and physics priors. The proposed
learning approach makes significant advances from the baseline in terms of the
following: (1) ADePt does not use the ground truth structure of ICs for its
learning process, which sets it apart from any supervised machine learning
approaches and saves significant amount of time and resources from the ground
truth preparation, and only operates on a single set of experimental
measurements; and (2) ADePt explicitly leverages the physical forward model of
the X-ray ptycho-laminography geometry, which tightly regularizes the solution
space and guides the algorithm to generate a physically-feasible
reconstruction even when the number of projections is largely reduced.

##  2 Methods

###  2.1 X-ray ptycho-laminography experiment

Ptycho-laminography measurements were carried out at the coherent small-angle
X-ray scattering (cSAXS) beamline at the Swiss Light Source at the Paul
Scherrer Institut (PSI), Switzerland. An integrated circuit produced with
$16\mbox{-}\mathrm{nm}$ technology was scanned with synchrotron X-rays of
$6.2\>\mathrm{keV}$ using laminographic nano-imaging instrument (LamNI) [20].
The laminographic angle between the rotation axis and the beam propagation
axis was fixed to be $61^{\circ}$, and the angular step of projections was
$0.18^{\circ}$, making $2000$ scans in total. Diffraction patterns were
recorded by a step scan at around $200$ lateral locations with an in-vacuum
Eiger 1.5 M detector (pixel size: $75\>\mu\mathrm{m}$, sample-to-detector
distance: $5.23\>\mathrm{m}$, exposure time: $0.1\>\mathrm{s}$) [21].

###  2.2 Baseline method and densely sampled reconstruction preparation

Densely sampled reconstruction preparation takes experimental ptycho-
laminographic measurements from $2000$ ptychography scans and uses a three-
step baseline method: (1) For every ptychography scan, projections are
retrieved by $1000$ iterations of the least-square likelihood ptychographic
algorithm [15] as implemented in PtychoShelves [22]. This ptychographic
reconstruction step takes far-field X-ray diffraction patterns ($512\times
512\>\mathrm{px}^{2}$, pixel size: $75\>\mu\mathrm{m}$) recorded at $200$
different lateral locations. (2) The projections are precisely aligned to each
other using the projection matching alignment (PMA) algorithm [23]. (3) The
aligned projections are synthesized to reconstruct a volumetric structure
using the standard Fourier Backprojection (FBP) method, followed by the
recovery of missing cone information. Data processing exactly follows the
steps in Ref. [19]. These steps result in the densely sampled reconstruction
with the voxel size of $27.2\>\mathrm{nm}$, which is larger than the
resolution of $19\>\mathrm{nm}$ reported in Ref. [19] due to fewer number of
projections acquired and a larger step size for each scan.

Despite the intensive densely-sampled reconstruction, the densely sampled
reconstruction remains ambiguous especially for longitudinal layer features as
a missing cone exists in the $k$-space due to the oblique X-ray illumination
in the ptycho-laminographic imaging [19].

###  2.3 Architecture

The network design is based on a modified U-net architecture with a random
input noise, following the implementation of deep image prior [24], which has
been widely used for many computational imaging applications, including image
dehazing [25], super-resolution [24, 26], phase retrieval [27, 28], tomography
[29, 30], and magnetic resonance imaging (MRI) [31]. This implementation
shares a similarity with coordinate-based learning approaches [32, 33, 34,
35]. The deep neural network is implemented as an encoder-decoder architecture
with skip connections [36] as shown in Fig. 1D. Network weights are
initialized with Xavier uniform distribution [37] (gain: $0.2$). Following the
convention of deep image prior [24], a random input noise $z$ is given to the
network, sampled from a uniform distribution $z\sim U_{M\times N\times
L}\left[0,\>0.1\right]$. The encoder reduces the lateral dimensions by a
factor of $4$, and the decoder restores the dimensions back to the original.
The architecture is chosen to have skip connections to relay the encoder
features to the decoder according to our finding that an hourglass
architecture does not reliably render high-frequency details. Finally, a
sigmoid-like activation function sets the range of output values of the ICs
within $\left[-0.03,\>0.03\right]$.

###  2.4 Pre-processing and network optimization

Figure 1C shows our pre-processor into three steps: (1) $10$-times fewer
iterations, i.e. $100$, of the least-square maximum likelihood ptychographic
algorithm are run to obtain intermediate projections; (2) the projection
matching alignment algorithm aligns the intermediate projections to match with
each other, which is applied to both densely-sampled and reduced datasets; and
(3) $256\times 256$ center pixels are cropped from the aligned projections to
be used for the reconstruction process with ADePt. A discussion of the
sensitivity to the alignment of projections is summarized in Fig. S1 in
Supplementary Materials.

ADePt iteratively updates the randomly initialized weights in the deep neural
network according to the loss functional

$\hat{w}=\text{argmin}_{w}\left[\frac{1}{2}\sum_{n=1}^{N}\left\lVert
H_{n}x-\mathrm{HPF}\left(y_{n}\right)\right\rVert_{\mathbf{2}}^{2}+\lambda\mathrm{TV}\left(x\right)\right],\>\>\>\mathrm{where}\>\>x=T_{w}\left(x\right).$
(1)

$T_{w}\left(\cdot\right)$ is a deep neural network parameterized with $w$,
$H_{n}$ a physical forward model corresponding to the $n$-th ptychography
scan, $y_{n}$ a pre-processed projection of the $n$-th ptychography scan,
$\mathrm{HPF}\left(\cdot\right)$ a high-pass filter on $y_{n}$’s, and
$\mathrm{TV}\left(\cdot\right)$ a total-variation regularization operator
acting upon the $3$D IC structure $x$. The TV regularization parameter
$\lambda$ is set differently across the $z$-axis, i.e. $\lambda=3\times
10^{-6}$ for $z<2.75\>\mu\mathrm{m}$ and $\lambda=3\times 10^{-8}$ otherwise.
Please see Fig. S2 in Supplementary Material for more details.

The reconstruction process is run for $1500$ iterations using Adam optimizer
[38] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and the initial learning rate of
$2\times 10^{-4}$ which is halved after $1000$ iterations.

##  3 Results

###  3.1 Qualitative performance comparison

Figure 2: Comparison between FBP & ADePt’s reconstructions and the densely
sampled reconstruction. (A) We exploit physics-informed machine learning to
reliably reconstruct integrated circuits with a reduced number of projections,
i.e. $125$ and $250$ out of $2000$, and qualitatively compare the FBP &
ADePt’s reconstructions with the densely-sampled reconstruction. Please note
that the colormaps used in some reconstructions are selected differently.
Specifically, for (1) the densely-sampled reconstruction, the colormaps of
Layers 2-5 are set to its 2.5th and 80th percentiles, and for (2) the FBP
reconstructions with both 125 and 250 angles, the colormaps of Layers 2-5 are
fixed to its 12.5th and 80th percentiles. For all other figures, the colormaps
are fixed to the minimum and maximum values of their respective
reconstructions. (B) We qualitatively compare the reconstructions within their
respective zoomed-in areas for better evaluation. The colormap conventions
followed in (A) are also applied. Figure 3: Qualitative comparison among $yz$
cross-sections of different reconstructions. Although the densely sampled
reconstruction yields the best contrast among coarser features, missing cone
artifacts are not completely addressed (see red arrows). Some finer features
can be displayed more clearly with ADePt than with the FBP reconstructions and
the densely-sampled reconstruction, at the cost of quantitativeness (see
yellow boxes).

We prepare the densely sampled reconstruction for the performance comparison
using the two-step iterative method with $2000$ projections sampled every
$0.18^{\circ}$ angular increment $\Delta\varphi$. Ptycho-laminographic
measurements are processed with $1000$ iterations of the least-square maximum
likelihood algorithm for each ptychography scan, followed by the projection
matching alignment, the filtered backprojection (FBP), and the recovery of
missing cone information [19].

In this study, we assess ADePt’s three-dimensional physically-feasible
rendering performance under sparsely sampled conditions by making a comparison
between the densely sampled reconstruction and the ADePt’s reconstructions.
Here, we increase $\Delta\varphi$ from $0.18^{\circ}$ to $1.44^{\circ}$ and
$2.88^{\circ}$, limiting the number of available projections to $250$ and
$125$ from $2000$. In Fig. 2, we visualize reconstructed IC features at five
different depths, i.e. $z$-axis locations, using the two-step baseline and
ADePt with $250$ and $125$ projections to qualitatively compare them with the
densely sampled reconstruction. Visually comparing the two, ADePt’s
reconstructions are more closely aligned to the densely sampled reconstruction
for both cases – even better than the densely sampled reconstruction with
greater feature contrast to identify high spatial-frequency details that comes
at the cost of quantitativeness – although the FBP reconstructions show much
lower spatial resolution due to accrued artifacts from the missing cones and
sparse sampling. We support the argument with the aid of the cross-section
profiles. Fig. S3 in Supplementary Materials visualizes the reconstructions
from different axial views. Moreover, ADePt benefits from the sparse sampling
scheme, resulting in $\times 6.58$ and $\times 9.57$ aggregate reduction in
computation time with $250$ and $125$ projections, respectively, compared to
the densely sampled reconstruction. For more details on time breakdown, please
see Table 1.

|  Densely sampled reconstruction (2000 angles) |  ADePt (250 angles) |  ADePt (125 angles)
---|---|---|---
Data acquisition∗ | $11.1$ | $1.38\>\>(\times 8\>\>\mathrm{faster})$ | $0.694\>\>(\times 16\>\>\mathrm{faster})$
Data computation† | $4.27$ | $0.958\>\>(\times 4.46\>\>\mathrm{faster})$ | $0.914\>\>(\times 4.67\>\>\mathrm{faster})$
(1) Iterative update (maximum likelihood) | $4.05^{\S}$ | $0.0890$ | $0.0445$
(2) Reconstruction - Data I/O, alignment, synthesis | $0.218^{\P}$ | $0.869$ | $0.869$
Total time (hr) | $15.4$ | $2.34\>\>(\times 6.58\>\>\mathrm{faster})$ | $1.61\>\>(\times 9.57\>\>\mathrm{faster})$
Table 1: Data preparation and computation time breakdown. ∗Data acquisition
uses LamNI [20] to acquire ptycho-laminographic measurements over the
integrated circuit sample ($26.2\times 38.2\times 3.92\>\mu\text{m}^{3}$).
†Data computation includes an iterative ptychographic update step, data I/O,
projection matching alignment [23], and laminographic synthesis. Densely
sampled reconstruction uses $1000$ iterations of the LSQ-ML algorithm [15,
22], and ADePt’s reconstructions use $100$ iterations of the same algorithm.
The data computation time breakdown comparison is made on the reconstructed
sample ($4.36\times 4.36\times 3.92\>\mu\text{m}^{3}$). §Iterative
ptychographic update of the densely sampled reconstruction is performed on
$10$ GTX 1080 GPUs. ¶Data reconstruction process of the densely sampled
reconstruction is based on $1$ V$100$ GPU. All other computations use $2$
V$100$ GPUs.

###  3.2 Power spectral density representation

It is easier to describe the aforementioned artifacts in a different domain as
both the terms sparse sampling and missing cones are defined based on the
$k$-space. We visualize both the densely sampled reconstruction and FBP &
ADePt reconstructions as $k$-space representations by means of power spectral
density, which is the Fourier transform of the autocorrelation function, i.e.
the Wiener-Khinchin theorem.

Figure 4: Power spectral density analysis for qualitative comparison. We
visualize FBP & ADePt’s reconstructions and the densely sampled reconstruction
in $k$-space to visualize artifacts due to missing cone and sparse sampling.
Cuts are made along the $k_{y}\mbox{-}k_{z}$ and $k_{x}\mbox{-}k_{y}$ planes
(red arrows). We demonstrate that ADePt provides reconstructions with fewer
artifacts in $k$-space, considering that the FBP reconstructions display
artifacts due to the angular subsampling and missing cone, and that the
densely-sampled reconstruction shows artifacts due to imperfect missing cone
filling (black arrows).

In Fig. 4, we provide two-dimensional power spectral density profiles of the
baseline and ADePt’s reconstructions using $250$ and $125$ projections along
with the densely sampled reconstruction. Both FBP reconstructions show their
missing cones rooted in an oblique rotation axis in the ptycho-laminography
geometry. Sparse-sampling artifacts are also easily noticed in the
reconstructions, which provides explanations on lack of spatial resolution as
shown in Fig. 2.

Our framework, however, learns from experimental ptycho-laminographic
measurements how the missing parts should be filled to represent physically
feasible IC solutions guided by the physical and network-structure priors.
Moreover, comparing the densely sampled reconstruction and ADePt’s
reconstructions in $k$-space, we notice that ADePt renders ICs even better –
the missing cone of the densely sampled reconstruction remain unintentionally
accentuated, as shown with black arrows in Fig. 4, whereas the cone filled and
spectrum recovered with ADePt is more continuous. As already reported
elsewhere [39], it is plausible that ADePt could outperform the baseline
method even with more limited amount of data given as the proposed framework
takes a self-supervised approach, thus completely agnostic to the type of
specimen. However, this does not preclude future use of priors to boost
performance.

###  3.3 Quantitative evaluation

Figure 5: Quantitative analysis of reconstructions and ablation study. (A) We
use bit-error rate (BER) and Pearson correlation coefficient (PCC) to compare
ADePt’s reconstructions with the baseline for different scales of features in
the integrated circuits. (B) Ablation study. We assess relative contribution
of each design element, i.e. high-pass filtering (HPF) and total-variation
(TV) regularization, to the final reconstruction by removing one at a time
from the complete model. We incorporate HPF to enforce a high-frequency
content bias to our deep neural network and TV regularization to suppress
spurious high-frequency artifacts in the background. Fig. S4 in Supplementary
Materials illustrates (A) and (B) further using another quantitative metric.
(C) The self-supervised learning algorithm behaves unfavorably when each
component is ablated. Total variation (TV) regularization suppresses high-
frequency artifacts, and high-pass filtering (HPF) improves the spatial
resolution of features recovered by the algorithm (red arrows).

We use Bit-Error Rate (BER) as a metric to quantify the ratio of erroneous
occupancy in the reconstructions with reference to the densely sampled
reconstruction as the ground truth [40]. Binary objects such as integrated
circuits are particularly suited to this metric for quality assessment.
Although many printing materials comprise ICs, including copper, tungsten, and
aluminum, here we just treat the ICs as binary with regard to the occupancy,
irrespective of material. Using the Expectation-Maximization (EM) algorithm in
the context of Gaussian mixture models, we first binarize the densely-sampled
reconstruction to get the ground truth. However, since the densely-sampled
reconstruction may still be ambiguous especially for longitudinal features due
to missing cone in the Fourier domain (see Fig. 4), the layers with ambiguous
features will not be accurately binarized. Thus, we exclude the layers with
binarization errors from our quantitative analyses in Section 3.3. More
details can be found in Fig. S5 in Supplementary Materials.

In Fig. 5A, we compare ADePt’s performance with the baseline method’s using
BER and Pearson correlation coefficient (PCC). We demonstrate that ADePt
outperforms the baseline method for both conditions with fewer projections,
but the difference between two methods becomes more statistically significant
when only $125$ projections are considered during the reconstruction process,
which suggests that ADePt is more tolerant to sparse sampling. The layers
located at $z>2.75\>\mu\mathrm{m}$ are considered to have finer circuit
features, according to typical design rules of integrated circuits (Fig. S2 in
Supplementary Materials illustrates this further). This implies that higher-
frequency details are better reconstructed with the physics-informed self-
supervised machine learning, even at regions with fine transverse details.

###  3.4 Ablation study

As illustrated in the loss function in Fig. 1C, we employ two key design
elements to the implementation: (1) high-pass filtering on the intermediate
projections to improve feature contrast and to enforce high-frequency bias to
the network; and (2) total-variation regularization in the loss functional to
suppress residual artifacts. We assess the relative contribution of each
design element in our proposed framework by ablating each one of the following
elements in succession: high-pass filter on projections and total-variation
regularization.

The results are shown in detail in Fig. 5. One salient observation is that in
the region of all features ($z$ between $0$ and $3.92~{}\mu\text{m}$)
performance degrades by equal amounts if the high-pass filter or TV term are
ablated. On the other hand, in the fine feature regime ($z$ between $2.75$ and
$3.92~{}\mu\text{m}$) the significance of the high-pass filter is higher, as
evidenced in Fig. 5B by the catastrophic drop in performance when it is
ablated. Visual inspection of the ablated reconstructions in Fig. 5C confirms
these trends.

##  4 Discussion

ADePt provides a three-dimensional estimate of integrated circuits from
experimental ptycho-laminographic measurements using deep self-supervised
learning. The proposed framework makes explicit use of a physical forward
model to obtain the image iteratively, and regularizes through the chosen
sparsity-enforcing neural network kernel and an additional total variation
penalty in the loss function. This is a significant departure from supervised
learning approaches that typically require ground truth to prepare a paired
dataset for training, which is expensive.

Supervised methods’ performance is also limited in the absence of accurate
ground truth. For IC imaging this is almost always the case since obtaining
the true shape is challenging. In earlier supervised work [40], we used
reconstructions from dense angular sampling as ground truth. Here, we observe
that ADePt fills the missing cone more effectively than the densely sampled
reconstructions, as Fig. 4 clearly illustrated. That the integrated circuit
geometry is very compatible with our chosen auxiliary total variation
regularizer strengthens this claim. Further investigation of the ADePt
scheme’s performance in different types of specimens is a good topic for
future work.

In this work, we treat all ptycho-laminographic projections as equal. However,
in previous work [41] we have demonstrated that there is benefit to weighing
each differently, according to an attentional scheme [42, 43]. Also,
coordinate-based learning methods [33, 32, 34, 35] may be beneficial to
further increase the reconstruction volume as the methods generally take fewer
trainable parameters than convolutional neural network architectures. We leave
this for future work as well.

## Backmatter

Funding U.S. Department of Energy (DE-AC02-06CH11357); Korea Foundation for
Advanced Studies; Intelligence Advanced Research Projects Activity
(FA8650-17-C-9113).

## Acknowledgement

We are grateful to William Harrod, Ed Cole, Lee Oesterling, Antonio Orozco,
and Yudong Yao for helpful discussions and comments, and acknowledge Los
Alamos National Laboratory (LANL)’s contributions. Funding from the
Intelligence Advanced Research Projects Activity, Office of the Director of
National Intelligence (IARPA-ODNI), contract FA8650-17-C-9113 is gratefully
acknowledged. The MIT SuperCloud and Lincoln Laboratory Supercomputing Center
provided resources (high performance computing, database, consultation) that
have contributed to the research results reported within this paper. I. Kang
also acknowledges support from Korea Foundation for Advanced Studies (KFAS).
This research used resources of the Advanced Photon Source, a U.S. Department
of Energy (DOE) Office of Science User Facility, operated for the DOE Office
of Science by Argonne National Laboratory under Contract No. DE-
AC02-06CH11357. The views and conclusions contained herein are those of the
authors and should not be interpreted as necessarily representing the official
policies or endorsements, either expressed or implied, of the ODNI, IARPA or
the US Government. The measurements were performed at the cSAXS beamline of
the Swiss Light Source at the Paul Scherrer Institut, Switzerland. Samples
were prepared at the University of Southern California.

Disclosures The authors declare no competing interests.

Data Availability Statement Codes are publicly available at
https://github.com/iksungk/ADePt.

## References

  * [1] L. Helfen, T. Baumbach, P. Mikulik, D. Kiel, P. Pernot, P. Cloetens, and J. Baruchel, “High-resolution three-dimensional imaging of flat objects by synchrotron-radiation computed laminography,” Applied Physics Letters 86, 071915 (2005).
  * [2] L. Helfen, A. Myagotin, P. Mikulík, P. Pernot, A. Voropaev, M. Elyyan, M. Di Michiel, J. Baruchel, and T. Baumbach, “On the implementation of computed laminography using synchrotron radiation,” Review of Scientific Instruments 82, 063702 (2011).
  * [3] Z. H. Levine, A. R. Kalukin, S. P. Frigo, I. McNulty, and M. Kuhn, “Tomographic reconstruction of an integrated circuit interconnect,” Applied physics letters 74, 150–152 (1999).
  * [4] A. Tkachuk, M. Feser, H. Cui, F. Duewer, H. Chang, and W. Yun, “High-resolution X-ray tomography using laboratory sources,” in _Developments in X-ray Tomography V,_ vol. 6318 (SPIE, 2006), pp. 431–438.
  * [5] M. Holler, A. Diaz, M. Guizar-Sicairos, P. Karvinen, E. Färm, E. Härkönen, M. Ritala, A. Menzel, J. Raabe, and O. Bunk, “X-ray ptychographic computed tomography at 16 nm isotropic 3D resolution,” Scientific Reports 4, 1–5 (2014).
  * [6] M. Holler, M. Guizar-Sicairos, E. H. Tsai, R. Dinapoli, E. Müller, O. Bunk, J. Raabe, and G. Aeppli, “High-resolution non-destructive three-dimensional imaging of integrated circuits,” Nature 543, 402–406 (2017).
  * [7] W. Hoppe, “Beugung im inhomogenen primärstrahlwellenfeld. i. prinzip einer phasenmessung von elektronenbeungungsinterferenzen,” Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography 25, 495–501 (1969).
  * [8] R. Hegerl and W. Hoppe, “Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Primärstrahlwellenfeld,” Berichte der Bunsengesellschaft für physikalische Chemie 74, 1148–1154 (1970).
  * [9] J. Rodenburg and R. Bates, “The theory of super-resolution electron microscopy via wigner-distribution deconvolution,” Philosophical Transactions of the Royal Society of London. Series A: Physical and Engineering Sciences 339, 521–553 (1992).
  * [10] M. Guizar-Sicairos and P. Thibault, “Ptychography: A solution to the phase problem,” Physics Today 74, 42–48 (2021).
  * [11] J. M. Rodenburg and H. M. Faulkner, “A phase retrieval algorithm for shifting illumination,” Applied Physics Letters 85, 4795–4797 (2004).
  * [12] J. Rodenburg, A. Hurst, and A. Cullis, “Transmission microscopy without lenses for objects of unlimited size,” Ultramicroscopy 107, 227–231 (2007).
  * [13] H. M. L. Faulkner and J. Rodenburg, “Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm,” Physical review letters 93, 023903 (2004).
  * [14] P. Thibault, M. Dierolf, O. Bunk, A. Menzel, and F. Pfeiffer, “Probe retrieval in ptychographic coherent diffractive imaging,” Ultramicroscopy 109, 338–343 (2009).
  * [15] M. Odstrčil, A. Menzel, and M. Guizar-Sicairos, “Iterative least-squares solver for generalized maximum-likelihood ptychography,” Optics Express 26, 3108–3123 (2018).
  * [16] P. C. Konda, L. Loetgering, K. C. Zhou, S. Xu, A. R. Harvey, and R. Horstmeyer, “Fourier ptychography: current applications and future promises,” Optics Express 28, 9603–9630 (2020).
  * [17] G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nature Photonics 7, 739–745 (2013).
  * [18] L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for fourier ptychography with an led array microscope,” Biomedical optics Express 5, 2376–2389 (2014).
  * [19] M. Holler, M. Odstrcil, M. Guizar-Sicairos, M. Lebugle, E. Müller, S. Finizio, G. Tinti, C. David, J. Zusman, W. Unglaub _et al._ , “Three-dimensional imaging of integrated circuits with macro-to nanoscale zoom,” Nature Electronics 2, 464–470 (2019).
  * [20] M. Holler, M. Odstrčil, M. Guizar-Sicairos, M. Lebugle, U. Frommherz, T. Lachat, O. Bunk, J. Raabe, and G. Aeppli, “Lamni–an instrument for X-ray scanning microscopy in laminography geometry,” Journal of synchrotron radiation 27, 730–736 (2020).
  * [21] M. Guizar-Sicairos, I. Johnson, A. Diaz, M. Holler, P. Karvinen, H.-C. Stadler, R. Dinapoli, O. Bunk, and A. Menzel, “High-throughput ptychography using Eiger: scanning X-ray nano-imaging of extended regions,” Optics Express 22, 14859–14870 (2014).
  * [22] K. Wakonig, H.-C. Stadler, M. Odstrčil, E. H. Tsai, A. Diaz, M. Holler, I. Usov, J. Raabe, A. Menzel, and M. Guizar-Sicairos, “Ptychoshelves, a versatile high-level framework for high-performance analysis of ptychographic data,” Journal of Applied Crystallography 53 (2020).
  * [23] M. Odstrčil, M. Holler, J. Raabe, and M. Guizar-Sicairos, “Alignment methods for nanotomography with deep subpixel accuracy,” Optics Express 27, 36637–36652 (2019).
  * [24] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _Proceedings of the IEEE conference on computer vision and pattern recognition,_ (2018), pp. 9446–9454.
  * [25] Y. Gandelsman, A. Shocher, and M. Irani, “" double-dip": Unsupervised image decomposition via coupled deep-image-priors,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,_ (2019), pp. 11026–11035.
  * [26] G. Mataev, P. Milanfar, and M. Elad, “Deepred: Deep image prior powered by red,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops,_ (2019), pp. 0–0.
  * [27] F. Wang, Y. Bian, H. Wang, M. Lyu, G. Pedrini, W. Osten, G. Barbastathis, and G. Situ, “Phase imaging with an untrained neural network,” Light: Science & Applications 9, 1–7 (2020).
  * [28] E. Bostan, R. Heckel, M. Chen, M. Kellman, and L. Waller, “Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network,” Optica 7, 559–562 (2020).
  * [29] K. Gong, C. Catana, J. Qi, and Q. Li, “Pet image reconstruction using deep image prior,” IEEE transactions on medical imaging 38, 1655–1665 (2018).
  * [30] D. O. Baguer, J. Leuschner, and M. Schmidt, “Computed tomography reconstruction using deep image prior and learned reconstruction methods,” Inverse Problems 36, 094004 (2020).
  * [31] J. Liu, Y. Sun, C. Eldeniz, W. Gan, H. An, and U. S. Kamilov, “Rare: Image reconstruction using deep priors learned without groundtruth,” IEEE Journal of Selected Topics in Signal Processing 14, 1088–1099 (2020).
  * [32] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in Neural Information Processing Systems 33, 7462–7473 (2020).
  * [33] Y. Sun, J. Liu, M. Xie, B. Wohlberg, and U. S. Kamilov, “Coil: Coordinate-based internal learning for tomographic imaging,” IEEE Transactions on Computational Imaging 7, 1400–1412 (2021).
  * [34] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM 65, 99–106 (2021).
  * [35] R. Liu, Y. Sun, J. Zhu, L. Tian, and U. S. Kamilov, “Recovery of continuous 3d refractive index maps from discrete intensity-only measurements using neural fields,” Nature Machine Intelligence 4, 781–791 (2022).
  * [36] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention,_ (Springer, 2015), pp. 234–241.
  * [37] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics,_ (JMLR Workshop and Conference Proceedings, 2010), pp. 249–256.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).
  * [39] K. C. Zhou and R. Horstmeyer, “Diffraction tomography with a deep image prior,” Optics Express 28, 12872–12896 (2020).
  * [40] I. Kang, Z. Wu, Y. Jiang, Y. Yao, J. Deng, J. Klug, S. Vogt, and G. Barbastathis, “Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time,” arXiv preprint arXiv:2212.00014 (2022).
  * [41] I. Kang, A. Goy, and G. Barbastathis, “Dynamical machine learning volumetric reconstruction of objects’ interiors from limited angular views,” Light: Science & Applications 10, 1–21 (2021).
  * [42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762 (2017).
  * [43] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen, “Axial-deeplab: Stand-alone axial-attention for panoptic segmentation,” in _European Conference on Computer Vision,_ (Springer, 2020), pp. 108–126.