0 Research please specify Deng Luo (), Zainab Alsuwaykit (), Dawar Khan (), Ondřej Strnad, and Ivan Viola are with the Visual Computing Center at King Abdullah University of Science and Technology (KAUST), Saudi Arabia. E-mail: {deng.luo | zainab.alsuwaykit | dawar.khan | ondrej.strnad |<EMAIL_ADDRESS>Tobias Isenberg is with Université Paris-Saclay, CNRS, Inria, LISN, France. E-mail<EMAIL_ADDRESS>
# DiffFit: Visually-Guided Differentiable Fitting
of Molecule Structures to Cryo-EM Map

Deng Luo 0000-0003-4610-8730  Zainab Alsuwaykit0009-0009-7444-445X  Dawar
Khan0000-0001-5864-1888  Ondřej Strnad0000-0002-8077-4692  Tobias
Isenberg0000-0001-7953-8644  and Ivan Viola0000-0003-4248-6574

###### Abstract

We introduce DiffFit, a differentiable algorithm for fitting protein atomistic
structures into experimental reconstructed Cryo-em volume map. This process is
essential in structural biology to semi-automatically reconstruct large meso-
scale models of complex protein assemblies and complete cellular structures
that are based on measured Cryo-em data. Current approaches require manual
fitting in 3D that already results in approximately aligned structures
followed by an automated fine-tuning of the alignment. With our DiffFit
approach, we enable domain scientists to automatically fit new structures and
visualize the fitting results for inspection and interactive revision. Our
fitting begins with differentiable 3D rigid transformations of the protein
atom coordinates, followed by sampling the density values at its atom
coordinates from the target Cryo-em volume. To ensure a meaningful correlation
between the sampled densities and the protein structure, we propose a novel
loss function based on a multi-resolution volume-array approach and the
exploitation of the negative space. Such loss function serves as a critical
metric for assessing the fitting quality, ensuring both fitting accuracy and
improved visualization of the results. We assessed the placement quality of
DiffFit with several large, realistic datasets and found its quality to be
superior to that of previous methods. We further evaluated our method in two
use cases. First, we demonstrate its use in the process of automating the
integration of known composite structures into larger protein complexes.
Second, we show that it facilitates the fitting of predicted protein domains
into volume densities to aid researchers in the identification of unknown
proteins. We implemented our algorithm as an open-source plugin
(github.com/nanovis/DiffFitViewer) in ChimeraX, a leading visualization
software in the field. All supplemental materials are available at
osf.io/5tx4q.

###### keywords:

Scalar field data, algorithms, application-motivated visualization,
process/workflow design, life sciences, health, medicine, biology, structural
biology, bioinformatics, genomics, cryo-EM.

Compositing a protein (PDB 8SMK [53]) from its three unique chains. Top row
from left to right: three input chains, input target volume, the best fits in
the first fitting round, the remaining voxels after _zeroing-out_ , and the
fitted chains. Bottom row from left to right: two remaining input chains,
remaining region of interest in the target volume from the first round, the
best fits in the second round, the remaining voxels after _zeroing-out_ , the
fitted chains. Right: the final composited structure overlaid on the original
target volume. The involved computation takes 30 seconds in total, and the
human-in-the-loop interaction takes $\approx$ 5 minutes. cvt short=CVT,
long=Centroidal Voronoi tessellation, Cryo-em short=cryo-EM, long=Cryo-
Electron Microscopy , Cryo-et short=cryo-ET, long=Cryo-Electron Tomography,
pca short=PCA, long=principal component analysis,

Introduction

As humans we have been striving for centuries and even millennia to
understand, as Faust says, “was die Welt im Innersten zusammenhält [what binds
the world, and guides its course]” [46]. This quest applies, among other
directions of scientific inquiry, to the inner working of the biological
world. In particular, we are trying to understand how the biological processes
at tiny scales work and how they, ultimately, keep us alive. In the field of
structural biology, traditionally we have relied on techniques such as X-ray
crystallography or nuclear magnetic resonance spectroscopy to understand the
actual molecular compositions of cells and organelles—yet with the limitation
that these could only provide (still impressive and highly useful) estimates
or manually constructed models of the structure of actual biological samples
(e.g., [37, 45, 22, 42, 30, 5]). In the recent few years, however, the Cryo-em
approach [29] has opened the doors to visualizing the biomolecules within
_actual samples_ at near-atomic resolution. In addition, the Protein Data Bank
(PDB) initiative has over decades collected thousands of molecular models of
the building blocks of the cells or organelles studied in structural biology.
This new situation means that we are now at the brink of reconstructing the
molecular composition of actual samples at a ground-truth level.

Yet to achieve this type of reconstruction we not only need to interactively
visualize molecular data, for which tools [41] exist, but need to be able to
faithfully place 3D models of known molecular building blocks, such as from
PDB data and the AlphaFold predicted library [17], to the captured Cryo-em
datasets. So far, the fitting process involves a substantial time commitment
and numerous manual interventions by the domain experts—rendering this process
not very effective. The sheer complexity and size of the involved molecules,
combined with the variability and noise inherent in Cryo-em data, pose
substantial obstacles. A fully automatic process, in contrast, is also not the
ideal solution because the existence of local minima (wrong placement of the
compositing proteins) requires the domain experts to verify each placement
where they utilize their knowledge and experience. Fully automated methods are
currently far from feasible. Instead, we need the optimal balance between user
interaction and automation.

For this purpose, we developed DiffFit, an automated differentiable fitting
algorithm coupled with visual inspection and decision making, designed to
optimize alignment between protein structures and experimental reconstructions
of volumes (i. e., Cryo-em maps). Our technique works in both one-to-one
fitting scenarios and many-to-one scenarios, in which multiple protein subunit
structures are precisely aligned with a single, large, experimentally
reconstructed volume. The DiffFit method is iterative and gradually brings the
source protein structures into the target volumes to stepwise reconstruct the
molecular-subunits alignment. By leveraging advanced strategies such as volume
filtering, multi-resolution volumes, and negative space utilization, we
construct a loss function that allows us to quantify the fitting accuracy both
during the iterations and for the final decision making. In this way we
iteratively reduce the differences between the two representations—volumetric
and atomistic—until we achieve the desired fit. Through this visually-guided
fitting procedure we thus relieve the domain experts from manually placing
structures as they assemble the reconstruction of the Cryo-em map, thus
significantly speeding up the process into a manageable interactive procedure,
that yields results at sufficient precision for the visualization and analysis
of complex, real-world protein structures, ultimately facilitating large-scale
structural modeling initiatives. In summary, we contribute:

  * •

a differentiable fitting algorithm designed to fit multiple molecular subunits
to a single reconstructed Cryo-em volume;

  * •

human-in-the-loop strategy providing visual inspection and decision-making
within an iterative structure-discovery cycle;

  * •

a novel loss function and data processing that calculates new updates in each
iteration to expedite algorithm convergence and quantify fitting accuracy; and

  * •

three use-case scenarios of fitting either one or multiple known subunits or
identifying yet unknown subunits being part of the molecular assembly.

##  1 Background and related work

We rely on Cryo-em data, so below we first briefly provide essential relevant
background. Then we describe past work on image registration and model fitting
and show how both relate to our own research.

###  1.1 Brief background

Structural biology employs various techniques to understand how atoms are
arranged in macromolecular complexes, i. e., samples in the range of 60 kDa
(i. e., 4,472 atoms; PDB 6NBD [14]) to 50,000 kDa (3,163,608 atoms; PDB 8J07
[47]) that are essential for scientists to study processes in living
cells—Cryo-em being a particularly powerful one [2, 25, 23]. With Cryo-em,
bioscientists capture images using an electron microscope of flash-frozen
biological specimens, preserving their natural structure without any staining
or fixing [3], which would otherwise interfere with the sample. These images
are then used to construct Cryo-em 3D volumes or maps using the so-called
_single particle method_ that aligns thousands of projections from
structurally identical molecular instances into a single map (using the
Fourier slice-projection theorem) representing the electron density of the
sample, which can be used to infer the atom positions within the molecule.

Subsequently, the bioscientists need to build accurate atomistic or molecular
models that match the electron density map obtained from the Cryo-em process
to gain insight into molecular function and interactions. This process
involves mapping or fitting known sub-molecules into their corresponding
positions within the map. The objective is to achieve an optimal
correspondence between the model and the experimental or simulated volume,
revealing the organization of molecules in 3D space, including single
molecules, complexes, and the placement of small molecules and ligands into
binding sites. Molecular models are available in the Protein Data Bank (PDB,
rcsb.org), accessible in various formats such as PDB, Crystallographic
Information File (CIF), and mmCIF (macromolecular CIF). The fitting itself is
today typically achieved through manual placement, alignment, and comparison
with the density maps. The manual character of this process makes it extremely
time-consuming and tedious, and means that only expert biologists can complete
it. To address this challenge, numerous studies have aimed to automate the
fitting process, focusing on image registration as the foundation and
exploring methods to streamline 3D model construction as we review below.

###  1.2 Image registration and geometric fitting

The fitting of 3D structures into captured or simulated volumes relates to the
problem of image registration in image processing. It entails aligning two
images within a shared reference frame, regardless of whether they originate
from the same or different modalities [15, 8]. This process involves feature
extraction, determining transformations, and assessing accuracy through
metrics. Scale-invariant features from images [24], for example, can
facilitate matching across diverse views, despite significant distortions or
variations. This process involves detecting invariant keypoints using the
difference-of-Gaussian function, determining locations and scales, assigning
directions based on local gradients, and measuring gradients within selected
scales around each keypoint. Extracted features are stored in a database and
can then be matched with new images using fast nearest-neighbor algorithms,
with applications including object recognition. Among many applications of the
process, physicians rely on various imaging modalities to diagnose patients,
each capturing images with differing orientation. Image registration addresses
this variability by aligning images within a unified frame by optimizing
parameters like orientation and translation. Medical image registration is an
active research area which encompasses diverse methods, including techniques
based on cross-correlation [4, 26] and those based on mutual information [33,
28, 44, 20]. Shang et al. [40], e.g., introduced a method for medical image
registration using pca neural networks to extract feature images and compute
rotation angles and translation parameters by aligning the first principal
directions and centroids in a simple and efficient way. For complex spatial
transformations, another recent approach [1] uses Kernel pca and Teaching-
Learning-based optimization (TLBO) for multi-modal image registration. Similar
to these methods, transformations and alignments need to be determined to fit
the atomistic model into a volumetric map. We can thus also use optimization
techniques in Cryo-em map fitting to refine the fit and optimize parameters
such as orientation and translation—which is what we show with our work. The
major difference to image registration is that, in our workflow, we fit two
different data representations and one is a sub-part of the whole that is
potentially present at multiple locations in the target volume.

The model-to-data fitting that we need for Cryo-em data is a problem that has
also been investigated in depth in computer graphics and pattern recognition
[35, 18]. It has applications not only in structural biology but is also
needed in architectural geometry, virtual and augmented reality, robotics, and
various other fields [6, 38, 21, 52]. The key challenges in geometric fitting
include accuracy, efficiency, robustness, and usability of the fitting module
[38, 21]. Structural biology, in contrast, has special challenges such as
noisy data, non-geometric shapes, and large data sizes so that geometric
fitting methods are not directly applicable.

Yet our DiffFit algorithm still relates to techniques from computer graphics
and pattern analysis. The differentiable compositing technique proposed by
Reddy et al.[35], in particular, offers valuable insights into addressing
fitting challenges as well as manipulating and understanding image patterns.
With _differentiable compositing_ we can handle patterns effectively,
outperforming state-of-the-art alternatives in pattern manipulation [54, 39].
Reddy et al.’s method [35] discovers complex patterns by aligning elements
with their own position and rotation, and facilitates refinement based on
similarity to the target for precise adjustment. In addition, Reddy et al.use
a multi-resolution pyramid—relevant for handling the multi-resolution
volumetric data in our domain. Reddy et al.’s method [35], however, is
restricted to certain pattern types, requires manual element marking, and may
not always find the best solution, leading to orientation errors and missed
elements. Nevertheless, we build our solution on top of their differentiable
compositing.

Another approach, spline surface fitting [38], enhances the smoothness in
aircraft engine geometry reconstruction by concurrently approximating point
and normal data, ensuring boundary smoothness and optimal convergence, while
exploring norm-like functions’ effects on error measurement. A further
recently proposed adaptive spline surface fitting method [21] employs surface
meshes for high-precision CAD applications, supported by empirical evidence.
The reliance on control meshes of this approach, however, limits its
applicability to irregular topologies and compromise the preservation of sharp
features. All these methods have common objectives and tasks such as
similarity measures, pattern matching, fitting, and geometric transformations
and, thus, can serve as a motivation and starting point toward our goals in
structural biology. Since structural biology data often consists of large,
complex structures without regular shapes like CAD models or easy
representations in geometric meshes with smooth surfaces, however, the
aforementioned methods are not directly applicable to our data.

###  1.3 Fitting in structural biology

Existing fitting methods for structural biology can roughly be categorized
into manual, semi-automated, and automated approaches, each with its own
advantages and challenges when used for aligning molecular models with Cryo-em
density maps. Manual or semi-automated methods naturally involve human
intervention, yet they provide control and precision—which is particularly
beneficial in the structural analysis of complex datasets or when specific
adjustments are needed for accuracy. For example, a popular tool for molecular
manipulation and visualization, UCSF ChimeraX [31], includes the _fitmap_
technique [11]. It suggests multiple possible placements of the atomistic
model on the density map and then asks the user to make the final decision.
The fitting process alternates between rigid-body rotation and gradient
descent translation. In this way it maximizes the alignment between the atomic
model and the density data by optimizing the sum of density values.
Nonetheless, all of these manual or semi-automatic approaches are time-
consuming and require a significant level of expertise.

To tackle this challenge and to automate the fitting process, researchers have
developed methods that rely on deep learning (DL) [51, 43, 50]. $A^{2}$-Net by
Xu et al.[51], for example, uses DL to accurately determine amino acids within
a 3D Cryo-em density volume. It employs a sequence-guided Monte Carlo Tree
Search (MCTS) to traverse candidate amino acids, considering the sequential
nature of amino acids in a protein. Here, the authors divide the problem of
molecular structure determination into three subproblems: amino acid detection
in the density volumes, assignment of atomic coordinates to determine the
position of each amino acid, and main chain threading to resolve the
sequential order of amino acids that form each protein chain. Xu et al.’s
method also demonstrates a remarkable speed improvement, being 100 times
faster in finding solutions at runtime than existing methods [7, 49], while
achieving an excellent accuracy of $89.8\%$. In addition, they introduced the
$A^{2}$ dataset with 250,000 amino acids in 1,713 Cryo-em density volumes,
with a resolution of 3 $A^{\circ}$, pioneering automated molecular structure
determination training benchmarks.

Another recent method by Mallet et al.[27] for finding antibodies in cryo-EM
densities CrAI uses machine learning (ML). They formulate the objective as an
object detection problem, using the structural properties of Fabs (Fragment
antigen-binding) and VHHs (single-domain antibodies). Furthermore, DeepTracer
[32] is a fully automated DL-based method designed to determine the all-atom
structure of a protein complex using its high-resolution Cryo-em map and amino
acid sequence. It employs a customized deep convolutional neural network
primarily for precise prediction of the protein structure, including the
locations of amino acids, backbone, secondary structure positions, and amino
acid types. The reprocessed Cryo-em maps are the input to the neural network,
which transforms the output into a protein structure. Despite yielding
accurate outcomes, the resulting atomstic structures may exhibit geometric
issues, local fit to map discrepancies, misplaced side chains, or errors in
tracing and/or connectivity. All DL-based techniques, moreover, not only
require a substantial amount of time for the training (as opposed to the
runtime) but also rely on large training datasets of Cryo-em volumes and
manually fitted sub-molecules—which is why we do not resort to DL approaches.

An alternative to DL is map-to-map alignment, which is used to accurately
align two-dimensional or three-dimensional maps to facilitate comparison and
analysis of spatial structures or features within the maps. CryoAlign [13] is
a Cryo-em density map alignment method that achieves fast, accurate, and
robust comparison of two density maps based on local spatial feature
descriptors. This approach involves sampling the density map to generate a
point cloud representation and extracting key points by clustering based on
local properties. CryoAlign then calculates local feature descriptors to
capture structural characteristics, reducing the number of points considered
and improving efficiency. By employing a mutual feature-matching strategy,
CryoAlign establishes correspondences between keypoints in different maps and
uses iterative refinement to enhance alignment. A combination of fast
rotational matching search based on spherical harmonics and translational
scans [10] yields accurate fitting results in seconds to a few minutes. This
ADP_EM approach is reliable, in particular, for fitting X-ray crystal
structures to low-resolution density maps, with reduced docking times and
while maintaining a thorough 6D exploration with fine rotational sampling
steps to find valid docking solutions.

In our work, we design a differentiable optimization method for fitting
atomistic structures into volumetric data with the goal of precise fitting
with fast-enough computation to be applicable in semi-automatic fitting in the
standard tool ChimeraX. For this purpose we make use of the PyTorch
capabilities for GPU parallel computing, trilinear interpolation sampling in
volumetric data, and auto differentiation.

##  2 Method

Figure 1: DiffFit workflow. The target Cryo-em volume and the structures to be
fit on the very left serve as inputs, which are passed into the novel volume
processing, followed by the differentiable fitting algorithm. The fitting
results are then clustered and inspected by the expert. The expert may zero
out voxels corresponding to the placed structures and feed the map back
iteratively as input for a new fitting, round until the compositing is done.

We begin to describe our approach by explaining our process of differentiable
structure fitting, before we show how it can be used for visually-guided
fitting. After discussing these conceptual aspects, we also briefly discuss
implementation details.

###  2.1 Differentiable structure fitting

Given a Cryo-em map, the domain practitioners—bioscientists—do not know the
precise location and orientation parameters that govern where and how a
protein’s sub-structures fit together. For some regions in the map, the
bioscientists may not even know which protein subunits are supposed to be
present. Our goal is to develop a new approach that addresses both of these
domain tasks. For certain protein subunits, bioscientist are highly confident
about their presence in the map. In such cases, our technique will aid in the
identification of their respective placement parameters. Second, for the
regions with unknown protein subunits, the task of our technique is to
identify potential protein subunit candidates from a large database that best
fit the Cryo-em map region.

####  2.1.1 Inspiration and approach overview

We base our approach on the previously mentioned 2D differentiable compositing
approach by Reddy et al.[35], which discovers pattern structure from
wallpaper-like textures containing repetitive patterns made out of elementary
patches. We first review their approach, before we describe how we build up
our solution on top of their technique. In their case, given a 2D image, which
is a composite of multiple small element patches, the task is to identify the
number of occurrences of each patch and the placement parameters for each
existing patch. The parameters include the type of patch out of several known
patches as well as position, orientation, and depth. Their solution
distributes tens to hundreds of patches in the image and uses the
differentiable optimization methodology to translate and reorient patters such
that they correspond to the appearance of the patterns in the input wallpaper
image. Each single instance $E_{i}$ (out of total number of $\omega$
instances) of a pattern is stored in a layer $J_{i}$ for each patch instance
by sampling from that patch, with the translation, rotation, and patch-pattern
type probability taken into account:

$J_{i}(\mathbf{x})=f_{t}\left(\mathbf{x},E_{i}\right)=\frac{1}{\sum_{k=1}^{o}e^{t_{i}^{k}}}\sum_{j=1}^{o}e^{t_{i}^{j}}h_{j}\left(R_{\theta_{i}}^{-1}\left(\mathbf{x}-\mathbf{c}_{i}\right)\right)$
(1)

where $f_{t}\left(\mathbf{x},E_{i}\right)$ is a differentiable function using
the expected value over patch-pattern type probabilities stored in a tuple
$\mathbf{t}$ representing all $o$ patch patterns; softmax
$e^{t_{i}^{k}}/\sum_{k=1}^{m}e^{t_{i}^{k}}$ over type logits define the patch-
pattern type probabilities; $h_{j}$ is the image patch sampler function;
$\mathbf{x}$ is the image location; $\mathbf{c}_{i}$ is center location of
patch element $E_{i}$; and $R_{\theta_{i}}^{-1}$ is the inverse of a $2\times
2$ matrix rotation with angle $\theta_{i}$.

The solution image results from compositing of all instances together using
$f_{c}$ compositing function so that each known patch-pattern type is present
multiple times with various positional parameters. Patches can overlap other
patches, which leads to partial or full occlusion of a certain pattern. This
is characterized by $v_{i}(\mathbf{x}),v\in\\{0,1\\}$, which is the visibility
of layer $i$ at image location $\mathbf{x}$:

$I(\mathbf{x})=f_{c}\left(\left\\{J_{i}(\mathbf{x})\right\\}_{i}\right)=\sum_{i=0}^{\omega}J_{i}(\mathbf{x})v_{i}(\mathbf{x})$
(2)

This solution image is compared with the input image in an optimization, where
the parameters of all patch instances are updated in every iteration. The
solution image then becomes increasingly similar to the input image. Reddy et
al.define the $L^{2}$ distance loss $L_{d}$ for the optimizer as:

$L_{d}(A,I)=\frac{1}{P}\sum_{p=1}^{P}\left\|A\left(\mathbf{x}_{p}\right)-I\left(\mathbf{x}_{p}\right)\right\|_{2}^{2}$
(3)

where the sum is over the number of all pixels $P$ in the image. $A$ is the
input image and $I$ is the composited solution image. The optimal elements
$\mathcal{E}^{*}$ are then found by minimizing loss $L_{d}$ over the entire
set of elements $\mathcal{E}=\\{E_{0},..,E_{\omega}\\}$ :

$\mathcal{E}^{*}=\underset{\mathcal{E}}{\arg\min}\leavevmode\nobreak\
L_{d}(A,f_{c}(\mathcal{E})).$ (4)

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Clustering and filtering: (a) target volume, (b) atom coordinates of
the source structure, (c) positions of 1000 fit results (the dots are
clustered, hiding the number of results), (d) 1000 instances of the structure,
(e) positions of 1000 fit results with a transparency level set based on an
exponential scaling of the correlation metric (two clusters stand out), and
(f) instances of the structure at those two clusters.

Reddy et al.’s patch-pattern fitting problem is similar to ours in the sense
that in both cases we are compositing element instances into a scene. But
Reddy et al.’s differentiable compositing approach cannot be directly applied
to the structural biology domain to solve the protein fitting problem for the
following reasons:

  1. 1.

the pattern image and the element patches are defined in 2D with layers, while
the Cryo-em map and protein subunits are defined in 3D;

  2. 2.

the pattern image and the element patches are of the same representation,
i.e., 2D grid data, while the Cryo-em map and protein subunits are of
different representations—one is a 3D volume while the other is a set of atom
coordinates that can be regarded as a point cloud;

  3. 3.

the instance patches in differentiable compositing are all of the same size,
while the protein subunits differ in numbers of atoms;

  4. 4.

differentiable compositing expects the patches to overlap, while protein
subunits do not spatially overlap; and

  5. 5.

forming 1000 layers of 2D images is possible to fit into the current GPU
memory, while forming 1000 3D volumes is prohibitive with the currently
available GPU memory.

Initially, we attempted to align our problem better with differentiable
compositing by first _simulating_ a Cryo-em map from the atomistic point cloud
of the protein model and then fitting the simulated map to a target map. That
way the representational discrepancy (see reason (2) above) is eliminated.
That approach, however, was only successful for trivial cases, while for real-
world scenarii it frequently fell into local minima. Interested reader will
find exemplary volume-only based molecular fitting videos in supplementary
material at osf.io/5tx4q.

Driven by the successful fitting cases from many experiments, we gradually
built several novel strategies on top of differentiable compositing that
effectively address the problem of molecular structure fitting. Most notably,
we address the substantially higher complexity of our scenario based on the
knowledge, experience, and deep insight of the target audience: the
bioscientists. Our solution is thus based on a human-in-the-loop strategy and
we propose a fast and robust visual analytics approach, DiffFit, with two main
steps: (1) an automated excessive molecular fitting and (2) a visual
inspection and filtering of the fitted results by bioscientists. These two
consecutive steps are building blocks of a visual analytics feedback loop, in
which multiple proteins are iteratively composited to fit the underlying Cryo-
em volume.

We schematically present our DiffFit workflow in Figure 1. First, we seed an
excess amount of all compositing molecules in the volume scene. If fitting all
molecules simultaneously exceeds the available GPU memory, we sort the
molecules by atom count and partition them into batches. Then, we fit the
batches of molecules within several iterations in descending atom-count order.
For this fitting we rely on a novel loss function that calculates the average
density value from densities at each atom. The differentiable property of our
fitting scenario allows us to use optimization based on gradient-descent. In
addition, we associate each fit with a numerical value that characterizes the
quality of fit. For this purpose, we create a simulated Cryo-em map of each
fitted molecule and calculate the correlation of the simulated densities with
the real Cryo-em map densities. Then, we collect the fitting results and
cluster them based on positional and orientational parameters. For each
cluster, we select one representative fit—the fit with the highest correlation
value. We then sort the clusters by their representatives’ correlation and
interactively visualize them in ChimeraX to allow the bioscientists to inspect
the solutions. Once a given molecular placement is verified, we disable
molecule placements in the respective regions in the following iterations by
setting the those voxels in the Cryo-em map volume to zero that are covered by
the molecular fit. We thus gradually erase the successful placements from the
map, forcing following placements to search for a fit in non-zero volume
locations. Once the feedback update in the map is completed, we perform the
next iteration of fitting with another molecular structure. We repeat this
workflow pattern until the map has nearly all voxels zeroed out and the entire
sub-unit placement of the complex molecular structure is complete. Below we
introduce the details of DiffFit, in the following order: (1) sample one
coordinate; (2) fit one placement of one molecule; (3) fit multiple placements
of one molecule; and (4) fit multiple placements of multiple molecules.

####  2.1.2 Sampling of one coordinate

Because our task is to find the optimal alignment of an atomistic molecular
structure to the reconstructed Cryo-em volume map, we are searching for the
optimal fit characterized by two rigid-body transformation parameters: a
translational offset $\mathbf{p}$ and a rotation. The rotation is represented
by a quaternion $\mathbf{q}$ or its corresponding rotation matrix
$M_{\mathbf{q}}$. The position $\mathbf{x}_{i}$ corresponds to the center
point of an atom $i$ within the molecular subunit for which we are finding the
fit. For the calculation of the fit, we transform every atom position in one
subunit according to the rotation and translational offset:

$T(\mathbf{x}_{i})=M_{\mathbf{q}}\cdot\mathbf{x}_{i}+\mathbf{p}$

We sample a density value $D$ of the atom to be placed at position
$T(\mathbf{x}_{i})$ from a scalar volume $V$ by trilinear interpolation as:

$D(T(\mathbf{x}_{i}))=S(T(\mathbf{x}_{i}),V)$

Figure 3: Our visual browser based on ChimeraX. The target volume in the left
is overlaid with a fitted molecule corresponding to the selected fit result in
the table on the right (clustered fits, each row being the representative
placement with the highest correlation from that cluster). After inspection,
users can save the placement and then select “Simulate volume” and “Zero
density” to zero out the corresponding voxels from the target volume.

####  2.1.3 Placement of one molecule (or one subunit)

To find the best $\mathbf{p}$ and $\mathbf{q}$ parameters we formulate an
initial loss function $L$ that gives us the minimum negative average density
per atom for a molecular subunit with $N$ atoms that form the set
$\mathbf{X}_{m}$ of all atom center points
T($\mathbf{x}_{i})\in\mathbf{X}_{m}$ for a particular molecular subunit $m$:

$L(\mathbf{p},\mathbf{q},\mathbf{X}_{m},V)=-\left(\frac{1}{N}\sum_{i}^{N}D(T(\mathbf{x}_{i}))\right)=-\frac{1}{N}\sum_{i}^{N}S(M_{\mathbf{q}}\cdot\mathbf{x}_{i}+\mathbf{p},V).$
(5)

For the optimization we rely on the calculation of the gradient of our
differentiable formulation and use it with the Adam optimizer [19]. Despite
the Adam optimizer being known for robustness with respect to local minima,
our initial loss function formulation frequently led to such a local minimum,
i.e., a place that is not an optimal placement of the molecule in the map but
from which the optimizer cannot find better solution in the parameter-space
neighborhood. Such local minima are a common and severe problem that is also
manifested in the functionality of most commonly used tools for molecular
subunit fitting (e.g., ChimeraX’s fit-in-map feature). We thus introduced
several strategies to form a novel loss function and, ultimately, make DiffFit
more robust.

A first strategy that we found to substantially contribute to a good fitting
performance is filtering of the input Cryo-em map volume $V$. For this purpose
we clamp the volume values based on a user-specified minimal threshold and a
minimum size of connected-voxels that form a cluster. We also detect all
voxels with a density value smaller than a given threshold and set them to
zero. The size of the connected voxel cluster after thresholding has to be
bigger than the cluster size hyperparameter. Otherwise, we set all the voxels
in that cluster to zero. This step leads to the filtered volume $V_{F}$ and
ensures that only relevant volume regions are considered for fitting,
improving the algorithm’s focus and efficiency. We then normalize the filtered
values to $[0,1]$—a typical practice in learning and optimization approaches—,
which leads to a volume $\hat{V}_{F}$ and which turns out to be essential for
controlling the magnitude of the calculations that lead to the loss function
and hence the settings of the hyperparameters in the workflow.

To accommodate the inherent noise and variability within biological datasets,
we also applied a series of convolution iterations to the target volume, and
capture each smoothing as a separate volume. This iterative convolutional
smoothing leads to an array of volumes and we use each of these volumes in the
fitting process. This multi-resolution approach enhances the robustness of the
fitting process by mitigating the impact of noise and data irregularities. We
empirically found that a 10-element array of increasingly smoothed volumes,
iteratively filtered with a Gaussian smoothing kernel, performs well. We
expose this array’s size as a hyperparameter to allow users to control it. We
experimented with Laplacian smoothing as well, which led to unsatisfactory
performance. We denote the non-smoothed volume as $\hat{V}_{F}^{G_{0}}$ and
express the recurrent formulation of the iterative convolution smoothing as:

$\hat{V}_{F}^{G_{n}}=\hat{V}_{F}^{G_{n-1}}*G_{n}$

A third adaptation we apply to the initial fitting process is a stricter
penalization of a mismatch. If an atom center is placed within the Cryo-em map
volume but outside the extent of the molecular target structure, i.e., outside
of the target _footprint_ , where normally the target density would be zero.
To discourage such misalignment even more, we assign these regions a negative
value. After smoothing, therefore, for those voxels with a density value of
zero, we replace the zero with a negative value. We experimented with varying
the negative values or creating a smooth gradient of negative values but found
a constant value of $-0.5$ outside the molecular footprint within the map
performs well. We expose this value, however, as a tunable hyperparameter. We
denote the resulting volume as $\hat{V}_{F_{-c}}^{G_{n}}$ with $-c$ being the
negative constant value. Finally, we update the loss function formulation with
a volume smoothed after $j$ iterations as:

$L(\mathbf{p},\mathbf{q},\mathbf{X}_{m},\hat{V}_{F_{-c}}^{G_{j}})$

We weigh each fit with a multi-resolution volume array element $w_{j}$ for a
total number of resolutions $n$, and sum up all the multi-resolution
components to form the final loss function for one $\mathbf{p}$ and
$\mathbf{q}$ pair:

$L_{m}([p,q])=\sum_{j=1}^{n}w_{j}\cdot
L(\mathbf{p},\mathbf{q},\mathbf{X}_{m},\hat{V}_{F_{-c}}^{G_{j}})$ (6)

To start the optimization we need to initialize the position offset
$\mathbf{p}$ and the rotation quaternion $\mathbf{q}$. We uniformly sample
$N_{\mathbf{q}}$ points on a unit sphere and then convert these into
quaternions to be applied for each offseted position. Instead of uniformly
sampling positions from the volume’s bounding box (as in ChimeraX), we
uniformly sample $N_{\mathbf{p}}$ positions from the positive voxels in the
filtered and normalized volume $\hat{V}_{F_{-c}}$. This data-adaptive
initialization increases our success rate by a factor of two by searching from
$N_{\mathbf{q}}\cdot N_{\mathbf{p}}$ initial placements, as compared to
ChimeraX’ traditional initialization.

####  2.1.4 Fitting multiple placements of one molecule

To look for fits for multiple copies of a single molecule $m$, we then take
advantage of GPU parallelization and optimize all $N_{\mathbf{q}}\cdot
N_{\mathbf{p}}$ pairs of $[\mathbf{p},\mathbf{q}]$ of the molecule with atoms
$\mathbf{X}_{m}$ altogether in one single loss function:

$L_{par}(m)=\sum_{k=1}^{N_{\mathbf{q}}\cdot
N_{\mathbf{p}}}L_{m}([\mathbf{p}_{k},\mathbf{q}_{k}]).$ (7)

####  2.1.5 Fitting multiple placements of multiple molecules

Finally, as all subunit molecules have different numbers of atoms, it is not
easy to parallelize the treatment of multiple molecules without overhead on
the array padding of zeros. And because, usually, the $N_{\mathbf{q}}\cdot
N_{\mathbf{p}}$ initial placements of $\mathbf{X}_{m}$ atoms would result in a
total number of sampling operations higher than the total number of GPU
threads, we process different subunits molecules sequentially in a for loop
and form an overall loss function for $M$ molecules as

$L_{all}=\sum_{l=1}^{M}L_{par}(l)$ (8)

####  2.1.6 Quantify the fit quality

By sampling in the simulated volume from the molecule, we can get a weight for
each atom coordinate as $W(\mathbf{x})=S(\mathbf{x},V_{sim})$. Then, for all
the atoms in a molecule, we can form two vectors, one as sampled density
vector
$\mathbf{D}=[D(\mathbf{x}_{1}),D(\mathbf{x}_{2}),...,D(\mathbf{x}_{N})]$ from
the target volume, the other as weight vector
$\mathbf{W}=[W(\mathbf{x}_{1}),W(\mathbf{x}_{2}),...,W(\mathbf{x}_{N})]$ from
the simulated volume. We can then calculate three alignment metrics, mean
overlap $\mu$, correlation $\rho$, and correlation about the mean $\rho_{\mu}$
as

$\mu=\frac{\mathbf{D}\cdot\mathbf{W}}{N}$

$\rho=\frac{\mathbf{D}\cdot\mathbf{W}}{|\mathbf{D}||\mathbf{W}|}$

$\rho_{\mu}=\frac{(\mathbf{D}-D_{\mu})\cdot(\mathbf{W}-W_{\mu})}{|\mathbf{D}-D_{\mu}||\mathbf{W}-W_{\mu}|}$

where the subtraction operator represents subtracting the scalar average
densities $D_{\mu}$ and $W_{\mu}$ from each component of the sampled density
vectors. We use these quality metrics during the interactive assessment by the
bioscientist in ChimeraX that we describe next.

###  2.2 Visually-guided fitting

A critical aspect of the post-processing of DiffFit involves the clustering
and sorting of fitting results to facilitate user-guided selection and
refinement. After the optimization phase, the algorithm generates a vast array
of potential fits, characterized by their translation and rotation parameters.
To manage this abundance of data and facilitate efficient result exploration,
we apply a clustering algorithm to group the fitting results based on their
spatial and orientational similarity (Fig. 2(c), (e)).

Each cluster represents a set of fits that are closely related, suggesting a
consensus among them regarding the position and orientation of the fitted
structure within the target volume. We sort these clusters based on a defined
metric, such as the overall density overlap or correlation coefficient we just
discussed, ensuring that the most promising fits are prioritized for user
review. This hierarchical organization allows researchers to quickly identify
the most accurate and relevant fitting results, streamlining the analysis
process.

To further assist in the exploration of fitting results, we created an
interactive visual browser as a comprehensive visualization tool that presents
the sorted clusters in a user-friendly format (Figure 3). This browser
displays key metrics for each cluster, including the average correlation
coefficient, density overlap, and a consensus error measure, providing a quick
overview of each cluster’s quality and relevance. The browser also allows
users to select a cluster and visually inspect the fitting results within the
3D context of the target volume. This interactive exploration is crucial for
assessing the fit quality in complex Cryo-em map regions, where subtle
differences in position or orientation can substantially impact the biological
interpretation of the results.

An innovative feature of our approach is the ability to iteratively refine the
fitting process through the selective exclusion of already placed molecules’
densities. Once a cluster is selected and its fit (or multiple fits) is
verified as accurate by the bioscientist, the corresponding density within the
target volume can be _zeroed-out_ , effectively removing the respective volume
region from further consideration in the following part of the fitting
process. This step is crucial for complex volumes containing multiple closely
situated structures, as it prevents the algorithm from repeatedly fitting
structures to the same volume region and reduces false positives when fitting
the remaining region.

Thus by iteratively fitting and zeroing out densities, users can progressively
shrink the target volume, isolating and identifying individual structures
within a dense or complex dataset. This iterative refinement approach ensures
that the fitting process is not only guided by the algorithm’s optimization
but also by the user’s expert knowledge and visual assessment, ultimately
leading to more accurate and biologically meaningful results.

Our resulting visually-guided fitting framework enhances the DiffFit algorithm
by integrating clustering, sorting, and interactive exploration tools. These
features enable users to efficiently filter through large datasets of fitting
results, identify the most promising fits, and iteratively refine the fitting
process based on visual assessment. The combination of automated optimization
with user-guided inspection and filtering addresses the challenge of
accurately fitting molecular subunit structures within volumetric data.

##  3 Implementation

We utilized PyTorch for the DiffFit algorithm implementation, capitalizing on
its dynamic computational graph, automatic differentiation, and GPU
acceleration to estimate positional offset and rotational quaternion
parameters efficiently. Specifically, we employed torch.nn.Conv3d for Gaussian
smoothing and torch.nn.functional.grid_sample for trilinear interpolation with
padding mode set to “border.” This approach enables rapid processing of large
volumes and multiple structures. We leveraged tensor operations and the Adam
optimization algorithm for accurate optimization results. We also integrated
additional functions from SciPy, Bio.PDB, mrcfile, and Numpy libraries to
enhance the algorithm’s functionality. In addition, DiffFit was seamlessly
integrated into ChimeraX (Figure 3), based on its bundle development
environment.

##  4 Use case scenarios

We designed DiffFit, with its advanced fitting algorithms and integration with
visualization tools, to tackle a range of challenges in structural biology. We
thus now explore three key scenarios where DiffFit can be particularly
impactful, demonstrating its versatility and potential.

###  4.1 Scenario 1: Fit a single structure

(a) Source structure.

(b) Target map.

(c) Fit result.

Figure 4: Fitting a single structure for 8JGF [12].

The most straightforward application of DiffFit is the fitting of a single
atomistic structure into a target volumetric map, for which we show an example
in Figure 4. This scenario is common in cases where an already resolved
protein structure or a predicted structure needs to be placed within a newly
reconstructed volume captured through Cryo-em for further refinement.

In this scenario, DiffFit’s optimization efficiently determines the optimal
position and orientation of the protein (Fig. 4(a)) within the volume (Fig.
4(b)). The whole fitting process is automatic, removing the prerequisite of
manually placing the structure at an approximate orientation close to the
final optimal. The interactive visually-guided inspection process allows
researchers to verify the fit (Fig. 4(c)), leveraging their expertise to
ensure biological relevance and accuracy. We report the successful hit rate
and computation times in Table 1 and compare them with the current solution in
ChimeraX for four recently published structures, for which we fitted each
structure 1000 times to obtain reliable results. The reported performance
values are based on a workstation that uses a Nvidia RTX 4090 GPU for the
DiffFit part, and single thread on an AMD Ryzen Threadripper PRO 3995WX 2.70
GHz for the ChimeraX fitmap command.

Table 1: Performance results of fitting a single structure.

Structure | C Hit | D Hit | Gain | C Time | D Time | Gain
---|---|---|---|---|---|---
8JGF [12] | 11 | 380 | 35× | 0:38:57 | 0:00:12 | 195×
8X79 [48] | 1 | 292 | 292× | 0:30:33 | 0:00:14 | 131×
8GAM [16] | 3 | 396 | 132× | 1:31:37 | 0:00:14 | 393×

C stands for ChimeraX, D stands for DiffFit; Gain = D/C for Hit and C/D for
Time.

###  4.2 Scenario 2: Composite multiple structures

A more complex use case involves the fitting of multiple structures into a
single, large, often complex, volumetric dataset, such as assembling a viral
capsid from individual protein units or constructing a ribosomal complex from
its constituent proteins and RNA molecules. DiffFit can handle such composite
fitting tasks by iteratively optimizing the placement of each component, from
biggest to smallest, while considering the spatial relationships and
interactions among them to prevent overlaps and ensure a coherent assembly.

DiffFit: Visually-Guided Differentiable Fitting of Molecule Structures to
Cryo-EM Map shows an example of compositing multiple structures into a Cryo-em
volume map, for the PDB-protein 8SMK [53]. In the first row we show how the
middle and bottom parts are fitted in a first round of computation, with the
remaining top part of the protein being fitted in a second round of our
interactive process.

The ability to _zero-out_ densities, once a fit is confirmed, allows DiffFit
to sequentially fit multiple structures without interference from previously
placed components, as we demonstrate in DiffFit: Visually-Guided
Differentiable Fitting of Molecule Structures to Cryo-EM Map. This iterative
approach is particularly useful for densely packed molecular complexes where
individual components may be difficult to distinguish in the volumetric data.
This scenario is critical for understanding the functional context of proteins
within larger biomolecular assemblies or cellular environments.

###  4.3 Scenario 3: Identify unknown densities

DiffFit also offers bioscientists the capability to identify and characterize
unknown densities within volumetric datasets. In cases where a volume contains
unassigned or ambiguous regions, possibly indicating the presence of
previously unidentified molecules or molecular complexes, DiffFit can be used
to screen a library of known structures and predicted structures for potential
matches.

By fitting the structures from a library into the unidentified densities and
evaluating the fit quality, researchers can hypothesize the identity of the
unknown components. This scenario is invaluable for discovery-based research,
where identifying novel components within complex molecular assemblies can
lead to significant biological insights.

(a) Library of structures to search against (subset).

(b) Unknown density identification; comparison of three potential fits which
are overlaid on top of the target volumes.

Figure 5: Unknown density identification, where dozens to hundreds of
molecular structures can be evaluated for potential fit.

We performed the search with the demo dataset from a recent automated domain-
level protein identification technique, DomainFit [9]. The task is to
identify, from a library of 359 protein domains (Fig. 5(a) shows a subset),
which one(s) best fits into the given volume (Fig. 5(b)). DomainFit identified
two candidates (I7MLV6_D3, and I7M317_D1) after first fitting all domains
using ChimeraX’s fitmap command and then performing statistical analyses to
remove false positives. With our DiffFit we identified these same two
candidates with our fitting technique followed by visual inspection, where the
two candidates were the first to be inspected. For the comparison of both
approaches we report the hit times the same way as before in Table 2, showing
a $\approx$ 2× gain on the hit rate. In addition, DomainFit takes approx. 12
hours to finish the task, while with DiffFit it takes $\approx$ 10 minutes—a
72× gain in computation time.

Table 2: Performance results of identifying unknown structures.

Structure | C Hit | D Hit | Gain
---|---|---|---
I7MLV6_D3 | 108 | 280 | 2.6×
I7M317_D1 | 127 | 163 | 1.3×

C stands for ChimeraX, D stands for DiffFit; Gain = D/C for Hit.

##  5 Feedback

In addition to quantitative metrics reported in the use-cases above,
qualitative feedback from users plays a crucial role in evaluating DiffFit’s
practical utility and user experience. We thus solicited feedback from a
diverse group of users, including Ph.D. students, structural biologists, and
computational scientists, through surveys, interviews, and hands-on testing
sessions. In total, 7 people responded to our request, and we report key
themes from their qualitative feedback as follows.

Usability. Our participants appreciated DiffFit’s intuitive interface and
workflow (“quite intuitive and easy to use”), which significantly lowers the
barrier to entry for new users, while still providing advanced features for
experienced researchers. The visually-guided fitting process, in particular,
was highlighted as a powerful feature for refining fitting results based on
expert judgment. Suggestions for further improving this interaction step were
to show several visual clusters at a time, to be able to visually compare the
fits, and some tabbing-based way to check the various clusters. This would
make it easier to observe how each cluster fits relative to the Cryo-em volume
and thus researchers would be better able to focus on a particular chain of
interest.

Impact on Research. The participants reported that DiffFit has the potential
to have a tangible impact on their research, enabling them to tackle complex
fitting challenges that were previously out of reach, such as the ability to
rapidly sample a large database of candidate structures for regions with
unassigned protein density. The new ability to rapidly and accurately fit
structures into volumetric data opens up new avenues of investigation and has
the potential to accelerate discoveries in structural biology and one
respondent stated that our automatic fitting and visual inspection approach
“could be a key feature in ChimeraX that [could become] a standard in many
pipelines” as well as “a key implementation [for] a standard modeling
workflow.”

The respondents also added suggestions for future development, such as a
better structuring of the handling of the associated files and the addition of
workflows that would rely on automatically creating model subdivisions and
doing fits on those, before integrating the results into the reference volume.
They also highlighted further potential such as using our approach for the
analysis of density maps that are generated through X-ray crystallography.

##  6 Limitations and future work

While we report substantial improvements over the state-of-the-art techniques,
our technique also has its limitations. One issue arises in Scenario 2, due to
the gradual _zeroing-out_ the target map, when we composite multiple
structures into a single map. In this case the volume removal can potentially
remove some voxels that are also part of adjacent interacting molecular
subunits. The next to-be-fitted subunit might then have difficulties to find
its correct location as these few zeroed voxels penalize its correct position.
We are in the process of investigating this limitation in the future and plan
to experiment with heuristics that remove only those voxels that are fully
covered by the molecular structure or to slightly shrink the to-be-_zeroed-
out_ region before its removal from the map. Another related limitation is
that our fitting process does not check for collisions within adjacent
subunits, as previously investigated in visualization work (e.g., [36]). While
the map shrinking process should address this issue, in some cases subunits
may still end up overlapping. Resolving this issue would be rather
straightforward by removing those subunit poses which overlap with previously
confirmed subunit placements or perform a minimal pose modification to resolve
the structural collision.

Our technique is also not entirely parameter-free, which would be the ideal
scenario. The threshold value for removal of voxels with small density values,
for example, and the threshold value for the connected-voxel minimal cluster
size need to be set manually. Manipulating these parameters requires prior
domain experience with Cryo-em data from the bioscientists. We plan to
automate identification of these thresholds or, at least, define suitable
default values and provide guidance about reasonable parameter ranges.

The visualization design in our current version of the tool also offers only
the most essential visual encoding types, with a clear potential for further
improvement. We plan to adapt comparative visualization techniques for
intersecting surfaces and smart visibility techniques for combined rendering
of volumetric density representation with molecular surfaces or cartoon
representations. Furthermore, visualization design for analyzing the
structural poses within a cluster suffers from high spatial occlusion among
the subunits’ various poses (see Fig. 2(d)). This is a common domain problem
and a dedicated research effort will be needed to combat the occlusion of this
magnitude.

Driven by the feedback we received, additional possible future directions
include extending DiffFit to deformable transformations, take stoichiometry
and symmetry into consideration, and speeding up it by integration with CUDA.

##  7 Conclusion

In conclusion, DiffFit offers a novel and efficient solution to the challenges
of atom-to-map fitting that has recently arisen in structural biology. We
address these challenges with our differentiable fitting algorithm along with
a set of essential strategies that make the algorithm robust for the domain
problem, adding a human-in-the-loop visual analytics approach to the workflow,
and providing our approach as an open-source package that is designed to work
as part of a standard software tool of the domain (ChimeraX). A key ingredient
to our approach was the change from pixel-to-pixel (or the equivalent voxel-
to-voxel) fitting to points-to-volume fitting, which enabled us to deal with
the specific constraints of the fitting problem in structural biology. In
hindsight, as we witness the success of using atom coordinates to sample the
volume instead of using all the voxels, DiffFit suggests that, in Reddy’s [35]
original technique that inspired our work, maybe not all the pixels in a patch
are needed for the optimization, which means that the image compositing
optimization could be more efficient and could handle larger examples with
faster processing.

The quantitative performance metrics we reported collectively demonstrate
DiffFit’s substantial improvement over traditional approaches, highlighting
its potential to transform the field of structural biology. This potential is
particularly large because the workflows of Assembline, a protocol published
in 2022 [34], and DomainFit [9], so far, rely on ChimeraX’s fitmap command as
their first step. This stage can now be replaced by our more effective
DiffFit, laying a new foundation for these and other workflows. By
demonstrating the effectiveness of our technique across three scenarios, from
fitting individual structures to assembling complex molecular architectures
and identifying unknown components, we demonstrated that DiffFit overcomes the
limitations of manually placing individual molecules before fitting, makes the
compositing faster, more accurate, and intuitive, and opens the possibility of
scanning the whole set of known and predicted molecules with the current
computational resources. Ultimately, we thus escape the Faustian bargain [46]
and instead are now free ourselves to explore the inner workings of the
biological world.

###### Acknowledgements.

This research was supported by King Abdullah University of Science and
Technology (KAUST) (BAS/1/1680-01-01).

## Supplemental material pointers

We provide the open-source repository of our tool at
github.com/nanovis/DiffFitViewer. Other supplemental material is available at
osf.io/5tx4q.

## Images license and copyright

We as authors state that all of our figures are and remain under our own
personal copyright, with the permission to be used here. We also make them
available under the Creative Commons Attribution 4.0 International ( CC BY
4.0) license and share them at osf.io/5tx4q.

## References

  * [1] P. Arora, R. Mehta, and R. Ahuja.  An integration of meta-heuristic approach utilizing kernel principal component analysis for multimodal medical image registration.  Cluster Comput, 24 pages, 2024.  To appear. doi: 10/gtmxvd
  * [2] X.-c. Bai, G. McMullan, and S. H. W. Scheres.  How cryo-EM is revolutionizing structural biology.  Trends Biochem Sci, 40(1):49–57, 2015. doi: 10/f6wq7v
  * [3] W. Chen, X. Wang, and Y. Wang.  FFF: Fragment-guided flexible fitting for building complete protein structures.  In Proc. CVPR, pp. 19776–19785. IEEE CS, Los Alamitos, 2023. doi: 10/gtmxvf
  * [4] J. P. Didon and F. Langevin.  Registration of MR images: From 2D to 3D, using a projection based cross correlation method.  In Proc. EMBC, vol. 1, pp. 489–490. IEEE, Piscataway, 1995. doi: 10/bgp7mt
  * [5] D. Duran, P. Hermosilla, T. Ropinski, B. Kozlíková, Á. Vinacua, and P.-P. Vázquez.  Visualization of large molecular trajectories.  IEEE Trans Vis Comput Graph, 25(1):987–996, 2019. doi: 10/gjbdnk
  * [6] M. Falk, V. Tobiasson, A. Bock, C. Hansen, and A. Ynnerman.  A visual environment for data driven protein modeling and validation.  IEEE Trans Vis Comput Graph, 11 pages, 2024.  To appear. doi: 10/gsc3c9
  * [7] B. Frenz, A. C. Walls, E. H. Egelman, D. Veesler, and F. DiMaio.  RosettaES: A sampling strategy enabling automated interpretation of difficult cryo-EM maps.  Nat. Methods, 14(8):797–800, 2017. doi: 10/gf5cmq
  * [8] Y. Fu, Y. Lei, T. Wang, W. J. Curran, T. Liu, and X. Yang.  Deep learning in medical image registration: A review.  Phys Med Biol, 65(20), article no. 20TR01, 27 pages, 2020. doi: 10/ghn45h
  * [9] J. Gao, M. Tong, C. Lee, J. Gaertig, T. Legal, and K. H. Bui.  DomainFit: Identification of protein domains in cryo-EM maps at intermediate resolution using AlphaFold2-predicted models.  bioRxiv preprint 2023.11.28.569001, 2023. doi: 10/gs63f2
  * [10] J. I. Garzón, J. Kovacs, R. Abagyan, and P. Chacón.  ADP_EM: Fast exhaustive multi-resolution docking for high-throughput coverage.  Bioinf, 23(4):427–433, 2006. doi: 10/c9xkmg
  * [11] T. D. Goddard, C. C. Huang, and T. E. Ferrin.  Visualizing density maps with UCSF Chimera.  J Struct Biol, 157(1):281–287, 2007.  Software tools for macromolecular microscopy. doi: 10/ft849s
  * [12] L. Guo, Y. Zhang, G. Fang, L. Tie, Y. Zhuang, C. Xue, Q. Liu, M. Zhang, K. Zhu, C. You, P. Xu, Q. Yuan, C. Zhang, L. Liu, N. Rong, S. Peng, Y. Liu, C. Wang, X. Luo, Z. Lv, D. Kang, X. Yu, C. Zhang, Y. Jiang, X. Dong, J. Zhou, Z. Liu, F. Yang, H. E. Xu, and J.-P. Sun.  Ligand recognition and G protein coupling of the human itch receptor MRGPRX1.  Nat Commun, 14, article no. 5004, 12 pages, 2023. doi: 10/mpt4
  * [13] B. He, F. Zhang, C. Feng, J. Yang, X. Gao, and R. Han.  Accurate global and local 3D alignment of cryo-EM density maps using local spatial structural features.  Nat Commun, 15, article no. 1593, 15 pages, 2024. doi: 10/gtmxvg
  * [14] M. A. Herzik, M. Wu, and G. C. Lander.  High-resolution structure determination of sub-100 kDa complexes using conventional cryo-EM.  Nat Commun, 10, article no. 1032, 9 pages, 2019. doi: 10/gg7zp4
  * [15] D. L. G. Hill, P. G. Batchelor, M. Holden, and D. J. Hawkes.  Medical image registration.  Phys Med Biol, 46(3):R1–R45, 2001. doi: 10/fgbhgj
  * [16] C. Hu, M. T. Myers, X. Zhou, Z. Hou, M. L. Lozen, K. H. Nam, Y. Zhang, and A. Ke.  Exploiting activation and inactivation mechanisms in type I-C CRISPR-Cas3 for genome-editing applications.  Mol Cell, 84(3):463–475, 2024. doi: 10/mpt7
  * [17] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis.  Highly accurate protein structure prediction with AlphaFold.  Nature, 596:583–589, 2021. doi: 10/gk7nfp
  * [18] M. Kesäniemi and K. Virtanen.  Direct least square fitting of hyperellipsoids.  IEEE Trans Pattern Anal Mach Intell, 40(1):63–76, 2018. doi: 10/gcqjrb
  * [19] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint 1412.6980, 2017. doi: 10/hnkr
  * [20] S. Klein, J. P. W. Pluim, M. Staring, and M. A. Viergever.  Adaptive stochastic gradient descent optimisation for image registration.  Int J Comput Vis, 81(3):227–239, 2009. doi: 10/dghjcp
  * [21] Y.-B. Kou, Y.-F. Feng, L.-Y. Shen, X. Li, and C.-M. Yuan.  Adaptive spline surface fitting with arbitrary topological control mesh.  IEEE Trans Vis Comput Graph, 13 pages, 2024.  To appear. doi: 10/gtmxw4
  * [22] D. Kuťák, M. N. Selzer, J. Byška, M. L. Ganuza, I. Barišić, B. Kozlíková, and H. Miao.  Vivern—A virtual environment for multiscale visualization and modeling of DNA nanostructures.  IEEE Trans Vis Comput Graph, 28(12):4825–4838, 2022. doi: 10/mpwt
  * [23] K. Lasker, O. Dror, M. Shatsky, R. Nussinov, and H. J. Wolfson.  EMatch: Discovery of high resolution structural homologues of protein domains in intermediate resolution cryo-EM maps.  IEEE/ACM Trans Comput Biol Bioinf, 4(1):28–39, 2007. doi: 10/ccbr4m
  * [24] D. Lowe.  Distinctive image features from scale-invariant keypoints.  Int J Comput Vision, 60(2):91–110, 2004. doi: 10/bqrmsp
  * [25] S. Malhotra, S. Träger, M. Dal Peraro, and M. Topf.  Modelling structures in cryo-EM maps.  Curr Opin Struct Biol, 58:105–114, 2019. doi: 10/gtmxvc
  * [26] M. Malinsky, R. Peter, E. Hodneland, A. J. Lundervold, A. Lundervold, and J. Jan.  Registration of FA and T1-weighted MRI data of healthy human brain based on template matching and normalized cross-correlation.  J Digit Imaging, 26(4):774–785, 2013. doi: 10/gtmxw5
  * [27] V. Mallet, C. Rapisarda, H. Minoux, and M. Ovsjanikov.  Finding antibodies in cryo-EM densities with CrAI.  bioRxiv preprint 2023.09.27.559736, 2023. doi: 10/gtn9kj
  * [28] D. Mattes, D. R. Haynor, H. Vesselle, T. K. Lewellen, and W. Eubank.  PET-CT image registration in the chest using free-form deformations.  IEEE Trans Med Imaging, 22(1):120–128, 2003. doi: 10/cs4pch
  * [29] T. Nakane, A. Kotecha, A. Sente, G. J. McMullan, S. Masiulis, P. M. G. E. Brown, I. Grigoras, L. Malinauskaite, T. Malinauskas, J. Miehling, L. Yu, D. Karia, E. V. Pechnikova, E. de Jong, J. Keizer, M. Bischoff, J. McCormack, P. Tiemeijer, S. W. Hardwick, D. Y. Chirgadze, G. N. Murshudov, A. R. Aricescu, and S. H. W. Scheres.  Single-particle cryo-EM at atomic resolution.  Nature, 587(7832):152–156, 2020. doi: 10/gjwcfp
  * [30] N. Nguyen, O. Strnad, T. Klein, D. Luo, R. Alharbi, P. Wonka, M. Maritan, P. Mindek, L. Autin, D. S. Goodsell, and I. Viola.  Modeling in the time of COVID-19: Statistical and rule-based mesoscale models.  IEEE Trans Vis Comput Graph, 27(2):722–732, 2021. doi: 10/k8sh
  * [31] E. F. Pettersen, T. D. Goddard, C. C. Huang, G. S. Couch, D. M. Greenblatt, E. C. Meng, and T. E. Ferrin.  UCSF Chimera—A visualization system for exploratory research and analysis.  J Comput Chem, 25(13):1605–1612, 2004. doi: 10/b4bq4c
  * [32] J. Pfab, P. Minh Nhut, and D. Si.  DeepTracer for fast de novo cryo-EM protein structure modeling and special studies on CoV-related complexes.  PNAS, 118(2), article no. e2017525118, 12 pages, 2021. doi: 10/gjnkwt
  * [33] J. P. W. Pluim, J. B. A. Maintz, and M. A. Viergever.  Mutual-information-based registration of medical images: A survey.  IEEE Trans. Med. Imaging, 22(8):986–1004, 2003. doi: 10/czrq8h
  * [34] V. Rantos, K. Karius, and J. Kosinski.  Integrative structural modeling of macromolecular complexes using Assembline.  Nat Protoc, 17:152–176, 2022. doi: 10/gq586q
  * [35] P. Reddy, P. Guerrero, M. Fisher, W. Li, and N. J. Mitra.  Discovering pattern structure using differentiable compositing.  ACM Trans Graph, 39(6), article no. 262, 15 pages, 2020. doi: 10/gtmxvb
  * [36] J. Schmidt, R. Preiner, T. Auzinger, M. Wimmer, M. E. Gröller, and S. Bruckner.  YMCA – Your mesh comparison application.  In Proc. VAST, pp. 153–162. IEEE CS, Los Alamitos, 2014. doi: 10/f3sttr
  * [37] M. Schäfer, N. Brich, J. Byška, S. M. Marques, D. Bednář, P. Thiel, B. Kozlíková, and M. Krone.  InVADo: Interactive visual analysis of molecular docking data.  IEEE Trans Vis Comput Graph, 30(4):1984–1997, 2024. doi: 10/gtnn2n
  * [38] A. Seiler, D. Großmann, and B. Jüttler.  Spline surface fitting using normal data and norm-like functions.  Comput Aided Geom Des, 64:37–49, 2018. doi: 10/gd5gt4
  * [39] T. R. Shaham, T. Dekel, and T. Michaeli.  SinGAN: Learning a generative model from a single natural image.  In Proc. ICCV, pp. 4569–4579. IEEE CS, Los Alamitos, 2019. doi: 10/gg8fc3
  * [40] L. Shang, J. Lv, and Z. Yi.  Rigid medical image registration using PCA neural network.  Neurocomput, 69(13–15):1717–1722, 2006. doi: 10/fsrrb2
  * [41] R. Skånberg, I. Hotz, A. Ynnerman, and M. Linares.  VIAMD: A software for visual interactive analysis of molecular dynamics.  J Chem Inf Model, 63(23):7382–7391, 2023. doi: 10/gs7tzh
  * [42] R. Skånberg, P.-P. Vázquez, V. Guallar, and T. Ropinski.  Real-time molecular visualization supporting diffuse interreflections and ambient occlusion.  IEEE Trans Vis Comput Graph, 22(1):718–727, 2016. doi: 10/mpwv
  * [43] T. Terwilliger, P. Adams, P. Afonine, and O. Sobolev.  A fully automatic method yielding initial models from high-resolution cryo-electron microscopy maps.  Nat Methods, 15(11):905–908, 2018. doi: 10/gfnb8d
  * [44] P. Thevenaz and M. Unser.  Optimization of mutual information for multiresolution image registration.  IEEE Trans Image Process, 9(12):2083–2099, 2000. doi: 10/fs4j7h
  * [45] P. Ulbrich, M. Waldner, K. Furmanová, S. M. Marques, D. Bednář, B. Kozlíková, and J. Byška.  sMolBoxes: Dataflow model for molecular dynamics exploration.  IEEE Trans Vis Comput Graph, 29(1):581–590, 2023. doi: 10/kvjv
  * [46] J. W. von Goethe.  Faust. Eine Tragödie [Faust: A Tragedy].  J. G. Cotta, Tübingen, 1808.  Online: gutenberg.org/ebooks/21000; English translation: gutenberg.org/ebooks/3023.
  * [47] T. Walton, M. Gui, S. Velkova, M. R. Fassad, R. A. Hirst, E. Haarman, C. O’Callaghan, M. Bottier, T. Burgoyne, H. M. Mitchison, and A. Brown.  Axonemal structures reveal mechanoregulatory and disease mechanisms.  Nature, 618(7965):625–633, 2023. doi: 10/gscb95
  * [48] J. J. Wang, S. Jin, H. Zhang, Y. Xu, W. Hu, Y. Jiang, C. Chen, D. W. Wang, H. E. Xu, and C. Wu.  Molecular recognition and activation of the prostacyclin receptor by anti-pulmonary arterial hypertension drugs.  Sci Adv, 10(6), article no. eadk5184, 11 pages, 2024. doi: 10/mpt5
  * [49] R. Y.-R. Wang, M. Kudryashev, X. Li, E. H. Egelman, M. Basler, Y. Cheng, D. Baker, and F. DiMaio.  De novo protein structure determination from near-atomic-resolution cryo-EM maps.  Nat Methods, 12(4):335–338, 2015. doi: 10/gf5cn8
  * [50] X. Wang, E. Alnabati, T. W. Aderinwale, S. R. M. V. Subramaniya, G. Terashi, and D. Kihara.  Detecting protein and DNA/RNA structures in cryo-EM maps of intermediate resolution using deep learning.  Nat Commun, 12, article no. 2302, 9 pages, 2020. doi: 10/gmz55b
  * [51] K. Xu, Z. Wang, J. Shi, H. Li, and Q. C. Zhang.  A2-Net: Molecular structure estimation from cryo-EM density volumes.  Proc AAAI Conf Artif Intell, 33(1):1230–1237, 2019. doi: 10/ghkjcf
  * [52] I.-C. Yeh, C.-H. Lin, O. Sorkine, and T.-Y. Lee.  Template-based 3D model fitting using dual-domain relaxation.  IEEE Trans Vis Comput Graph, 17(8):1178–1190, 2011. doi: 10/dkbjmd
  * [53] X. Zhou, S. Kong, A. Maker, S. G. Remesh, K. K. Leung, K. A. Verba, and J. A. Wells.  Antibody discovery identifies regulatory mechanisms of protein arginine deiminase 4.  Nat Chem Biol, 2024.  To appear. doi: 10/mpwf
  * [54] Y. Zhou, Z. Zhu, X. Bai, D. Lischinski, D. Cohen-Or, and H. Huang.  Non-stationary texture synthesis by adversarial expansion.  ACM Trans Graph, 37(4), article no. 49, 13 pages, 2018. doi: 10/gd52tv