# Fair Latency-Aware Metric for real-time video segmentation networks

Evann Courdier  Francois Fleuret

###### Abstract

As supervised semantic segmentation is reaching satisfying results, many
recent papers focused on making segmentation network architectures faster,
smaller and more efficient. In particular, studies often aim to reach the
stage to which they can claim to be “real-time”. Achieving this goal is
especially relevant in the context of real-time video operations for
autonomous vehicles and robots, or medical imaging during surgery.

The common metric used for assessing these methods is so far the same as the
ones used for image segmentation without time constraint: mean Intersection
over Union (mIoU). In this paper, we argue that this metric is not relevant
enough for real-time video as it does not take into account the processing
time (latency) of the network. We propose a similar but more relevant metric
called FLAME for video-segmentation networks, that compares the output
segmentation of the network with the ground truth segmentation of the current
video frame at the time when the network finishes the processing.

We perform experiments to compare a few networks using this metric and propose
a simple addition to network training to enhance results according to that
metric.

Machine Learning, ICML

##  1 Introduction

Recent image segmentation networks are becoming good at producing visually
accurate results and more and more focus is now given to designing
architectures that are faster and can run on smaller hardware with less memory
and computing power. In particular, enabling real-time segmentation is
critical for applications in robotics, autonomous driving or medical imaging
during surgery.

The main metric currently used to assess segmentation performance is the mean
Intersection over Union (mIoU). For networks aiming at speed, researchers also
estimate efficiency with the Frame Per Second (FPS) metric, or its inverse the
Second Per Frame metric also called latency.

To know if a given network will fit one’s needs, one usually separately
assesses these two metrics, while they are intrinsically correlated. Moreover,
a network claiming a good mIoU with a relatively long latency will usually
produce a good segmentation of a scene that is no longer up-to-date. To the
best of our knowledge, there are currently no metric that aims at assessing
the usefulness of a network in a real-time setting. This confirms the
relevance of a latency-aware metric capturing the real-time performance and
“usefulness” of a network.

In the case of videos, we propose a novel Fair Latency-Aware MEtric (FLAME)
that simply comes down to a different way of computing the mIoU, and is an
attempt to combine both metrics by incorporating into the mIoU some
information about the latency. We hope this metric can (1) help clarify what
the objective of real-time networks should be and (2) represent a simple value
practitioners can easily understand and check when searching for a network
usable in real-time.

Our contributions are as follows:

  * •

we propose a simple and relevant metric to assess real-time performance of
video segmentation networks,

  * •

we review what this metric depends on, and what new information it brings,

  * •

we experimentally compare different networks with this metric and confirm its
relevance,

  * •

we consider different additions to the training process in order to optimize
performance according to this metric.

We will make our code publicly available at the time of the conference if the
paper is accepted.

##  2 Related Work

###  2.1 Image Semantic Segmentation

Most popular approaches for tackling Semantic Segmentation use a variant of
powerful deep classification networks that have been made fully
convolutionnal, with all final fully connected layers replaced by
convolutions. That idea is at the core of the FCN paper (Long et al., 2015).

The main issue coming with this technique is that it significantly reduces the
image resolution to retrieve semantic information. Subsequent models for
semantic segmentation are built on this “fully convolutional network” idea and
attempt to cope with the dimension reductions, while increasing the Receptive
Field.

One commonly used techniques is to use a decoder network plugged after the FCN
to upsample the segmentation map using transposed convolution, as first did
Ronneberger et al. (2015) and Badrinarayanan et al. (2017) with SegNet and
U-Net. This setup allows to merge spatially rich shallow layers into
semantically rich deeper layer.

Another successful technique is to avoid downsampling by using dilated
convolutions (Yu et al., 2017) as was popularized for segmentation in DeepLab
v2 (Chen et al., 2017). This allows to process a large field of view without
having to reduce the image size, although it comes with the drawback of a
larger computational complexity.

Finally, another successful idea is to add a so-called “Spatial Pyramidal
Pooling” module (He et al., 2015b) initially introduced for segmentation by
Zhao et al. (2016). SPP pools the image simultaneously at different
resolutions over a grid, which enlarges the Receptive Field. This allows to
incorporate a larger context and to take into account higher-level semantic.

###  2.2 Real-time Semantic Segmentation

Reducing the computational cost and the memory cost of deep segmentation
systems is critical for many applications that need to run real-time on slow
hardware. One of the first to have been working toward this goal was ICNet
(Zhao et al., 2017), which is a fast network that uses multi-scale processing
with a special fuse block to merge those multi-scale information.

Among the different ways to design an efficient architecture, the first
possibility is to use optimized blocks, for instance by factorizing kernels
$k\times k$ into $1\times k$ and $k\times 1$ kernels as is done by ERFNet
(Romera et al., 2017), or by using group convolutions. In this case, there are
different methods such as ShuffleNet (Zhang et al., 2018) to create
connections between groups.

It is also possible to use depthwise separable convolution (DSC), which are
the combination of depthwise and pointwise convolutions. These DSC are used to
lower the number of parameters and makes the inference faster, at the cost of
accuracy. They are used broadly in MobileNets (Howard et al., 2017; Sandler et
al., 2018).

Another main idea from these network is to to quickly downsample images in
order to perform most of the processing at smaller resolution and never do
full resolution processing. This idea is present in ENet (Paszke et al.,
2016).

Finally, separating the localization problem from the semantic extraction
problem and merging them appropriately as done in BiSeNet (Yu et al., 2018) is
also a working technique.

A recent paper has used neural architecture search to discover neural
architectures adapted to fast semantic segmentation, and in particular a
network they named FasterSeg (Chen et al., 2019).

In this category, Swifnet (Orsic et al., 2019) is another recent and relevant
work. It is a lightweight Resnet followed by a simple decoder using lateral
connections similarly to U-Net. It stands a bit aside insofar as its main
strength and difference is actually to be pretrained on ImageNet. For our
work, we choose SwiftNet as one of our base networks for its simplicity and
its speed.

###  2.3 Video segmentation networks

Finally, another part of the literature focuses on designing video
segmentation systems. More specifically, these works try to leverage the
temporal correlation of consecutive frames in a video to improve the next-
frame prediction and reduce computation and latency. However, most works in
this domain are more focused on improving segmentation accuracy than reducing
the latency.

The Clockwork net by Shelhamer et al. (2016) is a model that leverages
temporal correlation by running different parts of the network at each time-
step conditionally to how much the video has changed from the previous frame.
This technique has the disadvantage of not providing a fixed frame-rate.

Another direction to address the problem is to try propagating previous
features to consecutive frames to avoid recomputing very similar features for
following frames, as do Zhu et al. (2016), though their design still keeps it
too slow for real-time.

Li et al. (2018) built on these two previous ideas. Their network decides at
each frame whether to propagate previous features or to recompute the entire
segmentation map. They improved the clockwork design to reduce the maximum
latency but did not reach real-time.

Other works use predictive learning, that is predicting future frames or flow
motion using past frames and segmentations to help current segmentation (Jin
et al., 2017a, b).

Gadde et al. (2017) are using video temporal coherence to produce better
output segmentation maps through representation warping, and are not focused
on time efficiency.

Finally, a recent paper by Zhang et al. (2019) is proposing a fast
segmentation network using a Dynamic Targeting Network and a Mask Refinement
Network that respectively locates the segmentation target and predicts the
segmentation.

##  3 A latency-aware metric

###  3.1 The mean Intersection over Union

The Intersection over Union metric is surely one of, if not the most commonly
used method to assess the correctness of a segmentation mask. It is by
definition stronger than both precision and recall, which makes it a good
candidate for strong metric. It is defined as follow:

$\text{IoU}=\frac{TP}{TP+FP+FN}=\frac{|GT\cap Pred|}{|GT\cup Pred|}$ (1)

With TP: True Positive, FP: False Positive, FN: False Negative, GT: Ground
Truth.

The last equality is where the metric takes its name: we divide the
intersection of the ground truth and the predicted areas by the union of these
same areas.

As there usually are a few dozens a classes on a image, we take the mean of
all the IoU of each class to get the mean IoU: mIoU.

Finally, note that in the case of the mIoU of multiple images, one do not
average again single mIoUs to get another value. Rather, for a given class, we
consider every image when computing the union and intersection. The formula
for $K$ classes and $N$ images is then:

$mIoU=\frac{1}{K}\sum_{k=1}^{K}\frac{\sum_{i=1}^{N}|GT_{i,k}\cap
Pred_{i,k}|}{\sum_{i=1}^{N}|GT_{i,k}\cup Pred_{i,k}|}$ (2)

###  3.2 A latency-aware mean Intersection over Union

Figure 1: Left: How mIoU is computed now ; the output of the network is
compared to the ground truth segmentation of the image in input of the
network. Right: Our proposed way to measure mIoU ; the output of the network
is compared to the ground truth segmentation of the current image when network
finishes processing.

In order to give more meaning to the mIoU value when it is used for real-time
video applications, we propose to make it ”latency aware”. The ”latency-aware”
mIoU, named FLAME, is the mIoU between the segmentation mask in output of the
network and the ground truth segmentation of the current image at the time the
network finishes its computation.

Let us consider a video sequence and let $I_{t}$ and $S_{t}$ denote
respectively the frame at time $t$ and its ground truth segmentation. Let $F$
denote the operation of a semantic segmentation network that processes one
image in $l_{F}$ milliseconds. The common way to assess performance is to
compute:

$mIoU\left(F(I_{t}),S_{t}\right)$ (3)

while our metric proposes to consider:

$mIoU\left(F(I_{t}),S_{t+l_{F}}\right)$ (4)

The only change is the addition of $l_{F}$. Instead of predicting the
segmentation of the current frame, our FLAME metric expects systems to predict
the segmentation of a future frame, thus acknowledging the prediction time.

We claim this metric is particularly relevant for real-time applications in
which we are usually interested in what is currently happening, and not what
was a few instant back. It is indeed relevant to compare the information we
get at a given time using a network $\left(F(I_{t})\right)$ with the
information we ideally would like to get at that time
$\left(S_{t+l_{F}}\right)$.

In practice, a video sequence does not have a continuous set of frames. In
this case, we pick the frame appearing next after the model has output a
segmentation.

More precisely, let’s assume without loss of generality that $t=0$ when frame
of index $0$ enters the network and consider a video sequence with a delay $d$
between two frames (fps $=1/d$). Then, the index of the segmentation that the
metric would use as ground truth is:

$k_{F}=\lceil l_{F}/d\rceil$ (5)

In the following, when we refer to $t+k_{F}\times d$, we will abuse notation
and write $t+k_{F}$.

Figure 2: Definition of the frame used in the FLAME metric

###  3.3 Information brought by the metric

This new FLAME metric indicates the match of the current scene with what our
network outputs at that time. The longer the network takes in processing, the
less accurate it will be at predicting the segmentation of the frame coming
after that processing.

More importantly, the metric gives directly an insight of how good and useful
a network is in a real-time setting, without having to juggle with the static
mIoU and the FPS.

It does not give information about whether the network is actually fast enough
to work in real-time, but rather about how good it would be if we suppose it
is. Note that a network does not need to be ”real-time” strictly speaking for
the metric to make sense. For a network reaching a few frames per second, the
metric would still provide useful information.

This metric also brings to light the fact that a real-time video segmentation
network should be trained to predict a future segmentation, and in particular
the segmentation of the frame which will be current when it finishes the
processing. Additionally, these networks should be ranked according to how
well they anticipate the future and take into account their own latency.

###  3.4 What the metric depends on

Due to its latency-aware property, the FLAME metric is also more sensitive to
the setting in which it is computed than a usual metric.

The hardware \- what kind of GPU device is it running on - has a great
influence on the latency. Therefore, it also has a similar influence on our
metric. A given network will have a different value of the latency-aware
metric per hardware. This is actually a good feature as it can help choose the
right hardware depending on one’s precision need.

The dataset, and notably its sampling frequency, has a strong impact on the
metric. In particular, the higher the frame-rate of the video dataset, the
more precise and useful the metric will be. Indeed, if the time between two
frames is too long, it won’t allow fine grained difference between networks
whose latency are close. In the worst case, the time between two frames is
bigger than the two networks processing times, in which case this metric will
be less informative.

We see here the importance of having a good quality dataset with high sampling
frequency for the latency-aware metric to provide us with useful measurements.

##  4 Dataset and experimental setup

###  4.1 Dataset

As video sequence dataset, we chose the CityScapes111https://www.cityscapes-
dataset.com/ dataset (Cordts et al., 2016) to conduct our experiments. This
dataset contains 2,975 training, 500 validation and 1,525 testing video
sequences. Each sequence contains 30 frames, the 20th of which is annotated
with fine pixel-level class labels for 19 object categories. A sequence is
1.8s long, which gives a framerate of approximately 16.6 fps and there are
around 60ms between each frame.

We chose Cityscapes for the presence of video sequences and its widespread use
as a segmentation benchmark. However, Cityscapes sequences have a little low
framerate, which means the time between two images is a little long, which
reduces the accuracy of our metric.

As Cityscapes contains only the ground truth segmentation for one image per
sequence, we have to process as follows:

  1. 1.

We time the latency $l_{F}$ of the network

  2. 2.

We determine how many frames of offset $k_{F}$ this time corresponds to:
$k_{F}=\lceil l_{F}/0.06\rceil$ (0.06 = 60ms)

  3. 3.

We use as input of the network the frame of index $20-k_{F}$ since we always
have only the 20th frame’s ground truth segmentation

###  4.2 Networks

For our experiments, we chose 2 network architectures. We picked SwiftNet
(Orsic et al., 2019) and DeepLab-V3+ (Chen et al., 2018) with 2 different
encoders : Resnet-101 and MobileNet v2 (Sandler et al., 2018).

####  4.2.1 SwiftNet

SwiftNet is a state of the art network in real-time segmentation. For our
experiments, we have built this network as it is described in the original
paper. For convenience, we briefly describe it below. It is a network with an
encoder-decoder structure:

  * •

The encoder backbone is a classical Resnet-18 whose fully connected layers
have been removed to make it fully convolutionnal.

  * •

A Spatial Pyramidal Pooling Module with 4 different pooling layers of grid
size (1,2,4,8) is plugged in output of the encoder to increase its receptive
field.

  * •

Finally, a decoder with 3 upsampling modules recovers original image
resolution. An upsampling module upsamples the previous layer’s output and
then merges it with a skip connection coming from the encoder.

Figure 3: SwiftNet architecture

We will refer to it as SwiftNet-R18. It has approximately 12M parameters. On
Cityscapes, it reaches 75% mIoU and runs at about 40 fps on a GTX 1080 Ti. On
this hardware, SwiftNet has a latency of 26 ms. This means we have to use
$k_{F}=1$ frame offset to compute the FLAME performance.

We will also consider a variant of this network, since we noticed
experimentally that expending the number of channels after the initial
convolution helped reach higher performance.

More specifically, we replace the first layer of the original SwiftNet:

$conv(3,64,7\times 7,s=2)$

with the following block of four layers:

$\begin{array}[]{l}conv(3,130,7\times 7,s=2)\\\ BN(130)\\\ ReLU\\\
conv(130,64,3\times 3,s=1).\end{array}$

The newly created convolutions were initialized using He’s initialization (He
et al., 2015a). We will refer to our updated SwiftNet version as
SwiftNet-R18-X for eXtended.

Table 1: Results of the 3 experiments for each of the 4 networks. The first
line gives the static mIoU, the next three lines reports value of the FLAME
metric for different training configurations. The last three lines gives
information about latency, frame offset used for the network as explained in
4.1, and fps of these networks. Note that the $k$ temporal offset depends on
the network, hardware and dataset framerate, and is greater and leads to
poorer performance for slow processing.

Input Target (train) Target (test) DeepLab-R101 DeepLab-MN SwiftNet-R18
SwiftNet-R18-X $I_{t}$ $S_{t}$ $S_{t}$ 0.77 0.72 0.75 0.735 FLAME {320pt
$I_{t}$ $S_{t}$ $S_{t+k}$ 0.495 0.56 0.64 0.63 $I_{t}$ $S_{t+k}$ $S_{t+k}$
0.53 0.565 0.65 0.635 $I_{t-1},I_{t}$ $S_{t+k}$ $S_{t+k}$ 0.60 0.58 0.67 0.685
Frame offset ($k$) 4 2 1 1 Latency (ms) 195 76 26 38 FPS 5 13 38 26

####  4.2.2 DeepLab v3+

DeepLab v3+ is a state of the art network in image segmentation. We have
followed original paper instructions for the code of these networks. We will
briefly describe it below as well. It has an encoder-decoder architecture very
similar to that of SwiftNet:

  * •

We used two different encoder backbones :

    * –

One backbone is a dilated Resnet-101 network stripped of its fully connected
layers. We use an output stride of 16, so the last two Resnet blocks are using
dilated convolutions to enlarge the receptive field.

    * –

The other backbone is a MobileNet-V2 network as described by Sandler et al.
(2018). It uses depthwise separable convolutions inside “inverted” residuals
blocks separated by linear bottlenecks.

  * •

An Atrous Spatial Pyramidal Pooling module is plugged after this encoder. It
convolves the encoder output with 4 atrous convolutions using different
dilation rates: (1, 6, 12, 18).

  * •

Finally, a small decoder upsamples the ASPP output and concatenates it with
low-level features from the encoder. The decoder blends them with a
convolution and upsamples the output to the original image size.

When using a Resnet-101 as backbone, the whole network has approximately 60M
parameters and reaches 77% mIoU on Cityscapes but runs at 5 fps only. We will
refer to it as DeepLab-R101. On our hardware, it has a latency of 195 ms,
which means we have to use $k_{F}=4$ frame offsets to compute the FLAME
performance.

When using a MobileNet backbone, the model has about 5.5M parameters. It
reaches 72% mIoU on Cityscapes and runs at 13 fps. We will refer to it as
DeepLab-MN. It has a latency of 76 ms, so we have to use $k_{F}=2$ frame
offsets to compute the FLAME performance.

###  4.3 Training

We coded our experiments using the PyTorch framework. All encoders in our
network were pretrained on the ImageNet-1k dataset. We used pretrained network
weights provided in the library.

Data augmentation We used image crops of $768\times 768$. We did standard
image augmentation with random horizontal flip, random scaling from 0.75 to
1.5 and random gaussian blur.

SwiftNet For SwiftNet, we used a batch size of 12 and trained using Adam
optimiser with default parameters. We used a learning rate of $5e^{-4}$ and a
weight decay of $1e^{-4}$. We also set a smaller learning rate of $1e^{-4}$
for the part that was ImageNet-pretrained. We trained the network for 200
epochs and used a cosine annealing schedule with $\eta_{min}=1e^{-6}$.

DeepLab v3+ For DeepLab, we used a batch size of 10 and trained using SGD
optimiser with momentum of 0,9. We used a learning rate of $5e^{-2}$ and a
weight decay of $5e^{-4}$. We similarly set a smaller learning rate of
$5e^{-3}$ for the part that was ImageNet-pretrained. We trained the network
for 220 epochs and used a poly schedule with a power of 3.

Hardware We train all networks on a Tesla P40 GPU device. We run all our
timing experiments on a GTX 1080 Ti GPU device.

##  5 Experiments and Experimental Results

We perform 3 different successive experiments. For each experiments, the only
parameters that changes are the inputs and targets used for training and
testing.

(a) Overlay of $I_{t}$ and $F(I_{t})$

(b) Overlay of $I_{t+1}$ and $F(I_{t})$

(c) Overlay of $I_{t+1}$ and $S_{t+1}$ (G.T.)

Figure 4: Output segmentation of SwiftNet-R18 trained to predict $S_{t+1}$
from $I_{t}$. We observe anisotropic blur as the network is able to infer some
objects probable movement directions from their orientation.

###  5.1 First experiment

In the first experiment, we simply evaluates the 4 networks with the FLAME
metric previously defined. The networks are trained “as usual”: input is
$I_{t}$ and target is $S_{t}$.

The results are reported in second line of table 1. Compared to their usual
mIoU, we can notice a significant drop from 10% to 30 %.

###  5.2 Second experiment

In the second experiment, the networks are trained to predict the segmentation
ground-truth that our FLAME metric is using: we use as input $I_{t}$ and as
target $S_{t+k}$. Here $k$ is different for each network as each has a
different latency and will therefore be tested against a different
segmentation ground-truth.

The results are reported in third line of table 1. We can see a slight but
consistent increase of the metric for all networks. We reported on figure 4
examples of the output segmentation of SwiftNet-R18 overlaid on image $I_{t}$
and $I_{t+1}$.

We can notice the segmentation mask is slightly blurry, as could be expected.
However, in multiple cases, we can note the interesting fact that the blur is
anisotropic, meaning it is not a blur all around the object, but rather
promoting a specific direction.

It turns out the network is able to predict some objects movement based on
their orientation. For instance, people facing left in $I_{t}$ are likely to
have moved left in the next image $I_{t+1}$. Similarly, if it detects the
front of a car, it can infer that the car is coming toward the camera and thus
is likely to look bigger in the next frame.

However, the network has no way to infer the relative speed of different
instances in the images which makes its prediction relatively inaccurate.

###  5.3 Third experiment

In the third experiment, the networks are still trained to predict the FLAME
segmentation ground-truth $S_{t+k}$, but now we use as input both $I_{t-1}$
and $I_{t}$.

The results are reported in fourth line of table 1. We can see a consistent
improvement of the metric. It is clear that using an additional input in the
network is useful here for producing sharper and more accurate segmentation
maps.

Here, thanks to the two images in input, the network has a way of determining
directions and relative instant speed of the different elements in the image.
Therefore, it is able to predict a much more accurate segmentation of the next
frame. We reported on figure 5 examples of the output segmentation of
SwiftNet-R18 overlaid on image $I_{t}$ and $I_{t+1}$.

(a) Overlay of $I_{t}$ and $F(I_{t})$

(b) Overlay of $I_{t+1}$ and $F(I_{t})$

(c) Overlay of $I_{t+1}$ and $S_{t+1}$ (G.T.)

Figure 5: Output segmentation of SwiftNet-R18 trained to predict $S_{t+1}$
from $(I_{t-1},I_{t})$. We observe a more precise segmentation as the network
has a way to infer relative speeds and directions.

###  5.4 Additional experiment

We had the intuition that it is important for the network to have a big
Receptive Field when processing simultaneously images from different time-
steps $I_{t-1}$ and $I_{t}$. In order to obtain a bigger field of view without
having to use big filters, we tried to offset part of this work on the input.

The idea is to concatenate to the current inputs of the SwiftNet network
various translations of the previous image $I_{t-1}$. Particularly, we changed
the inputs $\left\\{I_{t-1},I_{t}\right\\}$ of the previous experiment to
$\left\\{T_{1}(I_{t-1}),\cdots,T_{N}(I_{t-1}),I_{t-1},I_{t}\right\\}$.
corresponding to $N$ different fixed translations $T_{i}$.

The translations offsets were chosen to span a regular grid around the origin.
We experimented with different numbers of translations in input: $N=9$,
$N=17$, $N=25$ and $N=81$.

The intuition behind the use of translations is to trade part of the
computational cost usually associated with the use of big convolutional
kernels for the memory cost of having more inputs. Using translations would
compensate the use of big kernels by allowing the model to simultaneously
attend different parts of the image that would normally be processed
separately by a normal convolution kernel.

While initial experiments seemed promising, we discovered that setting all
translation offsets to zero yielded nearly identical results. Setting all
translation offsets to zero amounts to using as inputs of the network multiple
copies of $I_{t-1}$.

In this setting, the only structural difference remaining with the original
SwiftNet model is an additional convolutional layer added at the input of the
model to account for the translations. More explicitly, in the case of $N$
translations, we had replaced:

$conv(3,64,7\times 7,s=2)$

with the following block:

$\begin{array}[]{l}conv(6+3\times N,8\times N,7\times 7,s=2)\\\ BN(8\times
N)\\\ ReLU\\\ conv(8\times N,64,3\times 3,s=1).\end{array}$

The number of output channels of the first convolution is dependant on the
number of translations used. We notice that the best results were obtained
with 17 translations and deduced that the main reason for these results was
the increase of the number of channels in output of the first convolution.
This remark eventually lead to the design of SwiftNet-R18-X described in
section 4.2, which is indeed slightly more performant according to the FLAME
metric when using $I_{t-1}$ and $I_{t}$ as inputs.

##  6 Conclusion

In this paper, we have introduced a new simple metric for real-time semantic
segmentation. Our proposed metric sums up in one value information about how
successful a network is in such a setting by assessing how the prediction
matches the ground truth, taking into account not only the computation speed,
but the resulting latency. More generally, we introduced a different way of
computing any existing metric and to make it meaningful for real-time video
processing.

In proposing such a metric, we have also emphasized that, due to non-
instantaneous computation, a real-time network actually has to make a
prediction which is in the future of the available input.

Finally, through our experiments we have seen that a real-time segmentation
network should make use of a few of the previous images to be able to estimate
speed and directions of elements in a scene, and infer future positions of
objects. We have also shown that increasing the number of channels early in
the processing of the SwiftNet network allows to improve performance in that
context.

## References

  * Badrinarayanan et al. (2017) Badrinarayanan, V., Kendall, A., and Cipolla, R.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(12):2481–2495, 2017.
  * Chen et al. (2017) Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(4):834–848, 2017.
  * Chen et al. (2018) Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (DeepLabv3+).  _arXiv:1802.02611 [cs]_ , February 2018.  URL http://arxiv.org/abs/1802.02611.  arXiv: 1802.02611.
  * Chen et al. (2019) Chen, W., Gong, X., Liu, X., Zhang, Q., Li, Y., and Wang, Z.  Fasterseg: Searching for faster real-time semantic segmentation.  _arXiv preprint arXiv:1912.10917_ , 2019.
  * Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3213–3223, 2016.
  * Gadde et al. (2017) Gadde, R., Jampani, V., and Gehler, P. V.  Semantic Video CNNs through Representation Warping.  _arXiv:1708.03088 [cs]_ , August 2017.  URL http://arxiv.org/abs/1708.03088.  arXiv: 1708.03088.
  * He et al. (2015a) He, K., Zhang, X., Ren, S., and Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1026–1034, 2015a.
  * He et al. (2015b) He, K., Zhang, X., Ren, S., and Sun, J.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 37(9):1904–1916, 2015b.
  * Howard et al. (2017) Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.  _arXiv:1704.04861 [cs]_ , April 2017.  URL http://arxiv.org/abs/1704.04861.  arXiv: 1704.04861.
  * Jin et al. (2017a) Jin, X., Li, X., Xiao, H., Shen, X., Lin, Z., Yang, J., Chen, Y., Dong, J., Liu, L., Jie, Z., et al.  Video scene parsing with predictive feature learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 5580–5588, 2017a.
  * Jin et al. (2017b) Jin, X., Xiao, H., Shen, X., Yang, J., Lin, Z., Chen, Y., Jie, Z., Feng, J., and Yan, S.  Predicting scene parsing and motion dynamics in the future.  In _Advances in Neural Information Processing Systems_ , pp. 6915–6924, 2017b.
  * Li et al. (2018) Li, Y., Shi, J., and Lin, D.  Low-latency video semantic segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5997–6005, 2018.
  * Long et al. (2015) Long, J., Shelhamer, E., and Darrell, T.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3431–3440, 2015.
  * Orsic et al. (2019) Orsic, M., Kreso, I., Bevandic, P., and Segvic, S.  In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 12607–12616, 2019.
  * Paszke et al. (2016) Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E.  Enet: A deep neural network architecture for real-time semantic segmentation.  _arXiv preprint arXiv:1606.02147_ , 2016.
  * Romera et al. (2017) Romera, E., Alvarez, J. M., Bergasa, L. M., and Arroyo, R.  Erfnet: Efficient residual factorized convnet for real-time semantic segmentation.  _IEEE Transactions on Intelligent Transportation Systems_ , 19(1):263–272, 2017.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pp. 234–241. Springer, 2015.
  * Sandler et al. (2018) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4510–4520, 2018.
  * Shelhamer et al. (2016) Shelhamer, E., Rakelly, K., Hoffman, J., and Darrell, T.  Clockwork Convnets for Video Semantic Segmentation.  _arXiv:1608.03609 [cs]_ , August 2016.  URL http://arxiv.org/abs/1608.03609.  arXiv: 1608.03609.
  * Yu et al. (2018) Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., and Sang, N.  BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation.  _arXiv:1808.00897 [cs]_ , August 2018.  URL http://arxiv.org/abs/1808.00897.  arXiv: 1808.00897.
  * Yu et al. (2017) Yu, F., Koltun, V., and Funkhouser, T.  Dilated residual networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 472–480, 2017.
  * Zhang et al. (2019) Zhang, L., Lin, Z., Zhang, J., Lu, H., and He, Y.  Fast video object segmentation via dynamic targeting network.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 5582–5591, 2019.
  * Zhang et al. (2018) Zhang, X., Zhou, X., Lin, M., and Sun, J.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 6848–6856, 2018.
  * Zhao et al. (2016) Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J.  Pyramid Scene Parsing Network (PSPNet).  _arXiv:1612.01105 [cs]_ , December 2016.  URL http://arxiv.org/abs/1612.01105.  arXiv: 1612.01105.
  * Zhao et al. (2017) Zhao, H., Qi, X., Shen, X., Shi, J., and Jia, J.  ICNet for Real-Time Semantic Segmentation on High-Resolution Images.  _arXiv:1704.08545 [cs]_ , April 2017.  URL http://arxiv.org/abs/1704.08545.  arXiv: 1704.08545.
  * Zhu et al. (2016) Zhu, X., Xiong, Y., Dai, J., Yuan, L., and Wei, Y.  Deep Feature Flow for Video Recognition.  _arXiv:1611.07715 [cs]_ , November 2016.  URL http://arxiv.org/abs/1611.07715.  arXiv: 1611.07715.


* [106] R. Heckel and M. Soltanolkotabi, “Compressive Sensing with Un-Trained Neural Networks: Gradient Descent Finds the Smoothest Approximation,” in _International Conference on Machine Learning_ , 2020.
  * [107] M. Z. Darestani and R. Heckel, “Accelerated MRI with un-trained neural networks,” _IEEE Transactions on Computational Imaging_ , vol. 7, p. 724–733, 2021.
  * [108] K. H. Jin, H. Gupta, J. Yerly, M. Stuber, and M. Unser, “Time-Dependent Deep Image Prior for Dynamic MRI,” _arXiv:1910.01684_ , 2019.
  * [109] Q. Zou, A. H. Ahmed, P. Nagpal, S. Kruger, and M. Jacob, “Dynamic imaging using a deep generative storm (gen-storm) model,” _IEEE Transactions on Medical Imaging_ , 2021.
  * [110] A. H. Ahmed, R. Zhou, Y. Yang, P. Nagpal, M. Salerno, and M. Jacob, “Free-breathing and ungated dynamic mri using navigator-less spiral storm,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 12, pp. 3933–3943, 2020.
  * [111] S. Poddar, Y. Q. Mohsin, D. Ansah, B. Thattaliyath, R. Ashwath, and M. Jacob, “Manifold recovery using kernel low-rank regularization: Application to dynamic imaging,” _IEEE Transactions on Computational Imaging_ , vol. 5, no. 3, pp. 478–491, 2019.
  * [112] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021.
  * [113] R. Arandjelović and A. Zisserman, “NeRF in Detail: Learning to Sample for View Synthesis,” _arXiv:2106.05264_ , 2021.
  * [114] F. Williams, T. Schneider, C. Silva, D. Zorin, J. Bruna, and D. Panozzo, “Deep Geometric Prior for Surface Reconstruction,” in _Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [115] W. Huang, H. B. Li, J. Pan, G. Cruz, D. Rueckert, and K. Hammernik, “Neural Implicit k-Space for Binning-Free Non-Cartesian Cardiac MR Imaging,” in _International Conference on Information Processing in Medical Imaging_. Springer, 2023, pp. 548–560.
  * [116] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 7537–7547, 2020.
  * [117] E. Dupont, A. Goliński, M. Alizadeh, Y. W. Teh, and A. Doucet, “COIN: COmpression with Implicit Neural Representations,” _arXiv:2103.03123 [cs, eess]_ , 2021.
  * [118] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit Neural Representations with Periodic Activation Functions,” _arXiv:2006.09661 [cs, eess]_ , 2020.
  * [119] W. Huang, H. Li, G. Cruz, J. Pan, D. Rueckert, and K. Hammernik, “Neural Implicit K-Space for Binning-free Non-Cartesian Cardiac MR Imaging,” 2022.
  * [120] J. F. Kunz, S. Ruschke, and R. Heckel, “Implicit Neural Networks with Fourier-Feature Inputs for Free-breathing Cardiac MRI Reconstruction,” 2023.
  * [121] B. Yaman, S. A. H. Hosseini, S. Moeller, J. Ellermann, K. Ugurbil, and M. Akcakaya, “Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data,” 2020, pp. 921–925.
  * [122] C. Millard and M. Chiew, “Simultaneous self-supervised reconstruction and denoising of sub-sampled MRI data with Noisier2Noise,” _arXiv preprint arXiv:2210.01696_ , 2022.
  * [123] Y. Chen, J. H. Holmes, C. Corum, V. Magnotta, and M. Jacob, “Deep Factor Model: A Novel Approach for Motion Compensated Multi-Dimensional MRI,” in _IEEE international Symposium on Biomedical Imaging_ , 2023.
  * [124] F. Wang, H. Qi, A. De Goyeneche, R. Heckel, M. Lustig, and E. Shimron, “K-band: Self-supervised mri reconstruction via stochastic gradient descent over k-space subsets,” _arXiv preprint arXiv:2308.02958_ , 2023.
  * [125] M. Akçakaya, B. Yaman, H. Chung, and J. C. Ye, “Unsupervised Deep Learning Methods for Biological Image Reconstruction and Enhancement: An overview from a signal processing perspective,” _IEEE Signal Processing Magazine_ , vol. 39, no. 2, pp. 28–44, 2022.
  * [126] G. Zeng, Y. Guo, J. Zhan, Z. Wang, Z. Lai, X. Du, X. Qu, and D. Guo, “A review on deep learning MRI reconstruction without fully sampled k-space,” _BMC Medical Imaging_ , vol. 21, no. 1, p. 195, 2021.
  * [127] C. M. Stein, “Estimation Of The Mean Of A Multivariate Normal Distribution,” _The annals of Statistics_ , pp. 1135–1151, 1981.
  * [128] S. Ramani, T. Blu, and M. Unser, “Monte-Carlo SURE: A Black-Box Optimization Of Regularization Parameters For General Denoising Algorithms,” vol. 17, no. 9, pp. 1540–1554, 2008.
  * [129] C. A. Metzler, A. Mousavi, R. Heckel, and R. G. Baraniuk, “Unsupervised Learning with Stein’s Unbiased Risk Estimator,” _arXiv preprint arXiv:1805.10531_ , 2018.
  * [130] M. Zhussip, S. Soltanayev, and S. Y. Chun, “Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 255–10 264.
  * [131] Y. C. Eldar, “Generalized SURE for Exponential Families: Applications to Regularization,” vol. 57, no. 2, pp. 471–481, 2008.
  * [132] D. L. Donoho and I. M. Johnstone, “Adapting To Unknown Smoothness Via Wavelet Shrinkage,” _Journal of the American Statistical Association_ , vol. 90, no. 432, pp. 1200–1224, 1995.
  * [133] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2Noise: Learning Image Restoration without Clean Data,” in _International Conference on Machine Learning_ , 2018, pp. 2971–2980.
  * [134] T. Klug, D. Atik, and R. Heckel, “Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods,” in _Neural Information Processing Systems (NeurIPS)_. arXiv, 2023.
  * [135] C. Millard and M. Chiew, “A theoretical framework for self-supervised MR image reconstruction using sub-sampling via variable density Noisier2Noise,” _IEEE transactions on computational imaging_ , 2023.
  * [136] N. Moran, D. Schmidt, Y. Zhong, and P. Coady, “Noisier2noise: Learning to denoise from unpaired noisy data,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 064–12 072.
  * [137] F. Wang, H. Qi, A. D. Goyeneche, M. Lustig, and E. Shimron, “K-band: Training self-supervised reconstruction networks using limited-resolution data,” 2023.
  * [138] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu _et al._ , “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 1, pp. 87–110, 2022.
  * [139] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 15 908–15 919, 2021.
  * [140] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [141] A. Gillioz, J. Casas, E. Mugellini, and O. Abou Khaled, “Overview of the Transformer-based Models for NLP Tasks,” in _2020 15th Conference on Computer Science and Information Systems (FedCSIS)_. IEEE, 2020, pp. 179–183.
  * [142] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [143] Y. Korkmaz, M. Yurt, S. U. H. Dar, M. Özbey, and T. Cukur, “Deep MRI Reconstruction with Generative Vision Transformers,” _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , vol. 12964 LNCS, pp. 54–64, 2021.
  * [144] C.-M. Feng, Y. Yan, H. Fu, L. Chen, and Y. Xu, “Task transformer network for joint MRI reconstruction and super-resolution,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24_. Springer, 2021, pp. 307–317.
  * [145] R. Souza, M. Bento, N. Nogovitsyn, K. J. Chung, W. Loos, R. M. Lebel, and R. Frayne, “Dual-domain cascade of U-nets for multi-channel magnetic resonance image reconstruction,” _Magnetic resonance imaging_ , vol. 71, pp. 140–153, 2020.
  * [146] S. Wang, Z. Ke, H. Cheng, S. Jia, L. Ying, H. Zheng, and D. Liang, “DIMENSION: dynamic MR imaging with both k-space and spatial prior knowledge obtained via multi-supervised network training,” _NMR in Biomedicine_ , vol. 35, no. 4, p. e4131, 2022.
  * [147] M. Ran, W. Xia, Y. Huang, Z. Lu, P. Bao, Y. Liu, H. Sun, J. Zhou, and Y. Zhang, “Md-recon-net: A parallel dual-domain convolutional neural network for compressed sensing mri,” _IEEE Transactions on Radiation and Plasma Medical Sciences_ , vol. 5, no. 1, pp. 120–135, 2020.
  * [148] N. M. Singh, J. E. Iglesias, E. Adalsteinsson, A. V. Dalca, and P. Golland, “Joint frequency and image space learning for MRI reconstruction and analysis,” _The journal of machine learning for biomedical imaging_ , vol. 2022, 2022.
  * [149] X. Zhao, T. Yang, B. Li, and X. Zhang, “SwinGAN: A dual-domain Swin Transformer-based generative adversarial network for MRI reconstruction,” _Computers in Biology and Medicine_ , vol. 153, p. 106513, 2023.
  * [150] Y. Gao and S. J. Reeves, “Optimal k-space sampling in MRSI for images with a limited region of support,” vol. 19, no. 12, pp. 1168–1178, 2000.
  * [151] D. Xu, M. Jacob, and Z. Liang, “Optimal sampling of k-space with Cartesian grids for parallel MR imaging,” in _Proc. Int. Soc. Magn. Reson. Med._ , vol. 13, 2005, p. 2450.
  * [152] J. P. Haldar and D. Kim, “OEDIPUS: An experiment design framework for sparsity-constrained MRI,” 2019.
  * [153] E. Levine and B. Hargreaves, “On-the-fly adaptive k-space sampling for linear MRI reconstruction using moment-based spectral analysis,” vol. 37, no. 2, pp. 557–567, 2017.
  * [154] L. K. Senel, T. Kilic, A. Gungor, E. Kopanoglu, H. E. Guven, E. U. Saritas, A. Koc, and T. Çukur, “Statistically Segregated k-Space Sampling for Accelerating Multiple-Acquisition MRI,” vol. 38, no. 7, pp. 1701–1714, 2019.
  * [155] F. Sherry, M. Benning, J. C. D. l. Reyes, M. J. Graves, G. Maierhofer, G. Williams, C.-B. Schönlieb, and M. J. Ehrhardt, “Learning the sampling pattern for MRI,” _arXiv preprint arXiv:1906.08754_ , 2019.
  * [156] B. Gözcü, R. K. Mahabadi, Y.-H. Li, E. Ilıcak, T. Çukur, J. Scarlett, and V. Cevher, “Learning-based compressive MRI,” vol. 37, no. 6, pp. 1394–1406, 2018.
  * [157] C. Lazarus, P. Weiss, N. Chauffert, F. Mauconduit, L. El Gueddari, C. Destrieux, I. Zemmoura, A. Vignaud, and P. Ciuciu, “SPARKLING: Variable-density k-space filling curves for accelerated T2*-weighted MRI,” _Magn. Reson. Med._ , vol. 81, no. 6, pp. 3643–3661, 2019.
  * [158] S. Ravula, B. Levac, A. Jalal, J. I. Tamir, and A. G. Dimakis, “Optimizing Sampling Patterns for Compressed Sensing MRI with Diffusion Generative Models,” 6 2023.
  * [159] T. Weiss, O. Senouf, S. Vedula, O. Michailovich, M. Zibulevsky, and A. Bronstein, “PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI,” _Machine Learning for Biomedical Imaging_ , vol. 1, pp. 1–23, 9 2019.
  * [160] C. D. Bahadir, A. Q. Wang, A. V. Dalca, and M. R. Sabuncu, “Deep-learning-based optimization of the under-sampling pattern in MRI,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 1139–1152, 2020.
  * [161] G. R. Chaithya, Z. Ramzi, and P. Ciuciu, “Learning the sampling density in 2D SPARKLING MRI acquisition for optimized image reconstruction,” _European Signal Processing Conference_ , vol. 2021-August, pp. 960–964, 2021.
  * [162] M. V. W. Zibetti, F. Knoll, and R. R. Regatte, “Alternating learning approach for variational networks and undersampling pattern in parallel MRI applications,” _IEEE Transactions on Computational Imaging_ , vol. 8, pp. 449–461, 2022.
  * [163] H. K. Aggarwal and M. Jacob, “J-modl: Joint model-based deep learning for optimized sampling and reconstruction,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 6, pp. 1151–1162, 2020.
  * [164] G. Wang, T. Luo, J. F. Nielsen, D. C. Noll, and J. A. Fessler, “B-Spline Parameterized Joint Optimization of Reconstruction and K-Space Trajectories (BJORK) for Accelerated 2D MRI,” _IEEE Transactions on Medical Imaging_ , vol. 41, pp. 2318–2330, 9 2022.
  * [165] G. Wang, J.-F. Nielsen, J. A. Fessler, and D. C. Noll, “Stochastic optimization of 3D non-cartesian sampling trajectory (SNOPY),” _arXiv preprint arXiv:2209.11030_ , 2022.
  * [166] C. G. Radhakrishna and P. Ciuciu, “Jointly Learning Non-Cartesian k-Space Trajectories and Reconstruction Networks for 2D and 3D MR Imaging through Projection,” _Bioengineering 2023, Vol. 10, Page 158_ , vol. 10, p. 158, 1 2023.
  * [167] C. Alkan, M. Mardani, S. S. Vasanawala, and J. M. Pauly, “Learning to Sample MRI via Variational Information Maximization,” 10 2020.
  * [168] J. Xie, J. Zhang, Y. Zhang, and X. Ji, “PUERT: Probabilistic under-sampling and explicable reconstruction network for CS-MRI,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 4, pp. 737–749, 2022.
  * [169] J. Zou and Y. Cao, “Joint Optimization of kt Sampling Pattern and Reconstruction of DCE MRI for Pharmacokinetic Parameter Estimation,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 11, pp. 3320–3331, 2022.
  * [170] B. Zhu, J. Liu, N. Koonjoo, B. R. Rosen, and M. S. Rosen, “AUTOmated pulse SEQuence generation (AUTOSEQ) using Bayesian reinforcement learning in an MRI physics simulation environment,” in _Proceedings of Joint Annual Meeting ISMRM-ESMRMB_ , 2018.
  * [171] B. Zhu, J. Liu, N. Koonjoo, B. Rosen, and M. S. Rosen, “AUTOmated pulse SEQuence generation (AUTOSEQ) and neural network decoding for fast quantitative MR parameter measurement using continuous and simultaneous RF transmit and receive,” in _ISMRM Annual Meeting & Exhibition_, vol. 1090, 2019.
  * [172] P. K. Lee, L. E. Watkins, T. I. Anderson, G. Buonincontri, and B. A. Hargreaves, “Flexible and efficient optimization of quantitative sequences using automatic differentiation of Bloch simulations,” _Magnetic resonance in medicine_ , vol. 82, no. 4, pp. 1438–1451, 2019.
  * [173] A. Loktyushin, K. Herz, N. Dang, F. Glang, A. Deshmane, S. Weinmüller, A. Doerfler, B. Schölkopf, K. Scheffler, and M. Zaiss, “MRzero-Automated discovery of MRI sequences using supervised learning,” _Magnetic Resonance in Medicine_ , vol. 86, no. 2, pp. 709–724, 2021.
  * [174] H. N. Dang, J. Endres, S. Weinmüller, F. Glang, A. Loktyushin, K. Scheffler, A. Doerfler, M. Schmidt, A. Maier, and M. Zaiss, “MR-zero meets RARE MRI: Joint optimization of refocusing flip angles and neural networks to minimize T2-induced blurring in spin echo sequences,” _Magnetic Resonance in Medicine_ , vol. 90, no. 4, pp. 1345–1362, 2023.
  * [175] O. Perlman, B. Zhu, M. Zaiss, M. S. Rosen, and C. T. Farrar, “An end-to-end AI-based framework for automated discovery of rapid CEST/MT MRI acquisition protocols and molecular parameter quantification (AutoCEST),” _Magnetic Resonance in Medicine_ , vol. 87, no. 6, pp. 2792–2810, 2022.
  * [176] B. Kang, M. Singh, H. Park, and H.-Y. Heo, “Only-train-once MR fingerprinting for B0 and B1 inhomogeneity correction in quantitative magnetization-transfer contrast,” _Magnetic Resonance in Medicine_ , vol. 90, no. 1, pp. 90–102, 2023.
  * [177] I. Beracha, A. Seginer, and A. Tal, “Adaptive model-based Magnetic Resonance,” _Magnetic Resonance in Medicine_ , 2023.
  * [178] B. Zhao, K. Setsompop, H. Ye, S. F. Cauley, and L. L. Wald, “Maximum likelihood reconstruction for magnetic resonance fingerprinting,” _IEEE transactions on medical imaging_ , vol. 35, no. 8, pp. 1812–1823, 2016.
  * [179] N. Seiberlich, V. Gulani, A. Campbell-Washburn, S. Sourbron, M. I. Doneva, F. Calamante, and H. H. Hu, “Quantitative magnetic resonance imaging,” 2020.
  * [180] N. Vladimirov and O. Perlman, “Molecular MRI-Based Monitoring of Cancer Immunotherapy Treatment Response,” _International Journal of Molecular Sciences_ , vol. 24, no. 4, p. 3151, 2023.
  * [181] L. Feng, D. Ma, and F. Liu, “Rapid MR relaxometry using deep learning: An overview of current techniques and emerging trends,” _NMR in Biomedicine_ , vol. 35, no. 4, p. e4416, 2022.
  * [182] F. Liu, L. Feng, and R. Kijowski, “MANTIS: model-augmented neural network with incoherent k-space sampling for efficient MR parameter mapping,” _Magnetic resonance in medicine_ , vol. 82, no. 1, pp. 174–188, 2019.
  * [183] F. Liu, R. Kijowski, G. El Fakhri, and L. Feng, “Magnetic resonance parameter mapping using model-guided self-supervised deep learning,” _Magnetic resonance in medicine_ , vol. 85, no. 6, pp. 3211–3226, 2021.
  * [184] H. Li, M. Yang, J. H. Kim, C. Zhang, R. Liu, P. Huang, D. Liang, X. Zhang, X. Li, and L. Ying, “SuperMAP: Deep ultrafast MR relaxometry with joint spatiotemporal undersampling,” _Magnetic Resonance in Medicine_ , vol. 89, no. 1, pp. 64–76, 2023.
  * [185] D. Ma, V. Gulani, N. Seiberlich, K. Liu, J. L. Sunshine, J. L. Duerk, and M. A. Griswold, “Magnetic resonance fingerprinting,” _Nature_ , vol. 495, no. 7440, pp. 187–192, 2013.
  * [186] J. Weigand-Whittier, M. Sedykh, K. Herz, J. Coll-Font, A. N. Foster, E. R. Gerstner, C. Nguyen, M. Zaiss, C. T. Farrar, and O. Perlman, “Accelerated and quantitative three-dimensional molecular MRI using a generative adversarial network,” _Magnetic Resonance in Medicine_ , vol. 89, no. 5, pp. 1901–1914, 2023.
  * [187] A. Panda, B. B. Mehta, S. Coppo, Y. Jiang, D. Ma, N. Seiberlich, M. A. Griswold, and V. Gulani, “Magnetic resonance fingerprinting–an overview,” _Current opinion in biomedical engineering_ , vol. 3, pp. 56–66, 2017.
  * [188] O. Cohen, B. Zhu, and M. S. Rosen, “MR fingerprinting deep reconstruction network (DRONE),” _Magnetic resonance in medicine_ , vol. 80, no. 3, pp. 885–894, 2018.
  * [189] F. Balsiger, A. Shridhar Konar, S. Chikop, V. Chandran, O. Scheidegger, S. Geethanath, and M. Reyes, “Magnetic resonance fingerprinting reconstruction via spatiotemporal convolutional neural networks,” in _Machine Learning for Medical Image Reconstruction: First International Workshop, MLMIR 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 1_. Springer, 2018, pp. 39–46.
  * [190] P. A. Gómez, M. Molina-Romero, C. Ulas, G. Bounincontri, J. I. Sperl, D. K. Jones, M. I. Menzel, and B. H. Menze, “Simultaneous parameter mapping, modality synthesis, and anatomical labeling of the brain with MR fingerprinting,” in _Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part III 19_. Springer, 2016, pp. 579–586.
  * [191] P. A. Gómez, M. Cencini, M. Golbabaee, R. F. Schulte, C. Pirkl, I. Horvath, G. Fallo, L. Peretti, M. Tosetti, B. H. Menze _et al._ , “Rapid three-dimensional multiparametric MRI with quantitative transient-state imaging,” _Scientific reports_ , vol. 10, no. 1, p. 13769, 2020.
  * [192] O. Cohen, V. Y. Yu, K. R. Tringale, R. J. Young, O. Perlman, C. T. Farrar, and R. Otazo, “CEST MR fingerprinting (CEST-MRF) for brain tumor quantification using EPI readout and deep learning reconstruction,” _Magnetic Resonance in Medicine_ , vol. 89, no. 1, pp. 233–249, 2023.
  * [193] O. Perlman, H. Ito, K. Herz, N. Shono, H. Nakashima, M. Zaiss, E. A. Chiocca, O. Cohen, M. S. Rosen, and C. T. Farrar, “Quantitative imaging of apoptosis following oncolytic virotherapy by magnetic resonance fingerprinting aided by deep learning,” _Nature biomedical engineering_ , vol. 6, no. 5, pp. 648–657, 2022.
  * [194] O. Perlman, K. Herz, M. Zaiss, O. Cohen, M. S. Rosen, and C. T. Farrar, “CEST MR-fingerprinting: practical considerations and insights for acquisition schedule design and improved reconstruction,” _Magnetic resonance in medicine_ , vol. 83, no. 2, pp. 462–478, 2020.
  * [195] D. Nagar, N. Vladimirov, C. T. Farrar, and O. Perlman, “Dynamic and rapid deep synthesis of chemical exchange saturation transfer and semisolid magnetization transfer MRI signals,” _Scientific Reports_ , vol. 13, no. 1, p. 18291, 2023.
  * [196] M. Singh, S. Jiang, Y. Li, P. van Zijl, J. Zhou, and H.-Y. Heo, “Bloch simulator–driven deep recurrent neural network for magnetization transfer contrast MR fingerprinting and CEST imaging,” _Magnetic Resonance in Medicine_ , 2023.
  * [197] I. Blystad, J. B. M. Warntjes, O. Smedby, A.-M. Landtblom, P. Lundberg, and E.-M. Larsson, “Synthetic MRI of the brain in a clinical setting,” _Acta radiologica_ , vol. 53, no. 10, pp. 1158–1163, 2012.
  * [198] O. Perlman, C. T. Farrar, and H.-Y. Heo, “MR fingerprinting for semisolid magnetization transfer and chemical exchange saturation transfer quantification,” _NMR in Biomedicine_ , vol. 36, no. 6, p. e4710, 2023.
  * [199] K. Wang, M. Doneva, J. Meineke, T. Amthor, E. Karasan, F. Tan, J. I. Tamir, S. X. Yu, and M. Lustig, “High-fidelity direct contrast synthesis from magnetic resonance fingerprinting,” _Magnetic Resonance in Medicine_ , 2023.
  * [200] “CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions,” _Scientific Reports 2020 10:1_ , vol. 10, pp. 1–13, 8 2020.
  * [201] C. Qin, J. Schlemper, J. Caballero, A. N. Price, J. V. Hajnal, and D. Rueckert, “Convolutional recurrent neural networks for dynamic MR image reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 38, pp. 280–290, 1 2019.
  * [202] P. M. Johnson and M. Drangova, “Conditional generative adversarial network for 3D rigid-body motion correction in MRI,” _Magnetic resonance in medicine_ , vol. 82, no. 3, pp. 901–910, 2019.
  * [203] Y. Arefeen, J. Xu, M. Zhang, Z. Dong, F. Wang, J. White, B. Bilgic, and E. Adalsteinsson, “Latent signal models: Learning compact representations of signal evolution for improved time-resolved, multi-contrast MRI,” _Magnetic Resonance in Medicine_ , 2023.
  * [204] S. Biswas, H. K. Aggarwal, and M. Jacob, “Dynamic MRI using model-based deep learning and SToRM priors: MoDL-SToRM,” _Magnetic Resonance in Medicine_ , vol. 82, pp. 485–494, 7 2019.
  * [205] J. Yoo, K. H. Jin, H. Gupta, J. Yerly, M. Stuber, and M. Unser, “Time-dependent deep image prior for dynamic MRI,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 12, pp. 3337–3348, 2021.
  * [206] C. M. Sandino, F. Ong, S. S. Iyer, A. Bush, and S. Vasanawala, “Deep subspace learning for efficient reconstruction of spatiotemporal imaging data,” in _NeurIPS 2021 Workshop on Deep Learning and Inverse Problems_ , 2021.
  * [207] C. Qin, W. Bai, J. Schlemper, S. E. Petersen, S. K. Piechnik, S. Neubauer, and D. Rueckert, “Joint learning of motion estimation and segmentation for cardiac MR image sequences,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11_. Springer, 2018, pp. 472–480.
  * [208] S. Ghadimi, D. A. Auger, X. Feng, C. Sun, C. H. Meyer, K. C. Bilchick, J. J. Cao, A. D. Scott, J. N. Oshinski, D. B. Ennis _et al._ , “Fully-automated global and segmental strain analysis of DENSE cardiovascular magnetic resonance using deep learning for segmentation and phase unwrapping,” _Journal of Cardiovascular Magnetic Resonance_ , vol. 23, pp. 1–13, 2021.
  * [209] C. M. Scannell, M. Veta, A. D. Villa, E. C. Sammut, J. Lee, M. Breeuwer, and A. Chiribiri, “Deep-learning-based preprocessing for quantitative myocardial perfusion MRI,” _Journal of Magnetic Resonance Imaging_ , vol. 51, no. 6, pp. 1689–1696, 2020.
  * [210] E. J. Zucker, C. M. Sandino, A. Kino, P. Lai, and S. S. Vasanawala, “Free-breathing accelerated cardiac MRI using deep learning: Validation in children and young adults,” _Radiology_ , vol. 300, no. 3, pp. 539–548, 2021.
  * [211] Q. Zou, A. H. Ahmed, P. Nagal, S. Kruger, and M. Jacob, “Alignment & joint recovery of multi-slice dynamic MRI using deep generative manifold model,” _arxiv.org/abs/2101.08196_ , 2021.
  * [212] Q. Zou, A. H. Ahmed, P. Nagpal, S. Priya, R. F. Schulte, and M. Jacob, “Variational manifold learning from incomplete data: application to multislice dynamic MRI,” _IEEE transactions on medical imaging_ , vol. 41, no. 12, pp. 3552–3561, 2022.
  * [213] M. Sermesant, H. Delingette, H. Cochet, P. Jais, and N. Ayache, “Applications of artificial intelligence in cardiovascular imaging,” _Nature Reviews Cardiology_ , vol. 18, no. 8, pp. 600–609, 2021.
  * [214] Y. Chang, Z. Li, G. Saju, H. Mao, and T. Liu, “Deep learning-based rigid motion correction for magnetic resonance imaging: A survey,” _Meta-Radiology_ , p. 100001, 2023.
  * [215] X. Zhao and X.-M. Zhao, “Deep learning of brain magnetic resonance images: A brief review,” _Methods_ , vol. 192, pp. 131–140, 2021.
  * [216] M. W. Haskell, S. F. Cauley, B. Bilgic, J. Hossbach, D. N. Splitthoff, J. Pfeuffer, K. Setsompop, and L. L. Wald, “Network Accelerated Motion Estimation and Reduction (NAMER): Convolutional neural network guided retrospective motion correction using a separable motion model,” _Magnetic Resonance in Medicine_ , vol. 82, pp. 1452–1461, 10 2019.
  * [217] N. M. Singh, N. Dey, M. Hoffmann, B. Fischl, E. Adalsteinsson, R. Frost, A. V. Dalca, and P. Golland, “Data Consistent Deep Rigid MRI Motion Correction,” _Proceedings of Machine Learning Research_ , vol. 177, pp. 1–14, 1 2023.
  * [218] H. Eichhorn, K. Hammernik, V. Spieker, S. M. Epp, D. Rueckert, C. Preibisch, and J. A. Schnabel, “Physics-Aware Motion Simulation for T2*-Weighted Brain MRI,” pp. 42–52, 3 2023.
  * [219] B. Levac, A. Jalal, and J. I. Tamir, “Accelerated motion correction for MRI using score-based generative models,” in _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2023, pp. 1–5.
  * [220] T. Küstner, J. Pan, C. Gilliam, H. Qi, G. Cruz, K. Hammernik, B. Yang, T. Blu, D. Rueckert, R. Botnar _et al._ , “Deep-learning based motion-corrected image reconstruction in 4D magnetic resonance imaging of the body trunk,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2020, pp. 976–985.
  * [221] X. Wang, M. Uecker, and L. Feng, “Fast real-time cardiac MRI: a review of current techniques and future directions,” _Investigative Magnetic Resonance Imaging_ , vol. 25, no. 4, pp. 252–265, 2021.
  * [222] A. Singh, S. S. M. Salehi, and A. Gholipour, “Deep predictive motion tracking in magnetic resonance imaging: application to fetal imaging,” _IEEE transactions on medical imaging_ , vol. 39, no. 11, pp. 3523–3534, 2020.
  * [223] E. Shimron, A. De Goyeneche, K. Wang, A. Halgren, A. B. Syed, S. Vasanawala, and M. Lustig, “BladeNet: Rapid PROPELLER Acquisition and Reconstruction for High Spatio-Temporal Resolution Abdominal MRI,” _Proceedings of the 31st Annual International Society for Magnetic Resonance in Medicine, London, UK_ , pp. 7–12, 2022.
  * [224] V. Murray, S. Siddiq, C. Crane, M. E. Homsi, T. H. Kim, C. Wu, and R. Otazo, “Movienet: Deep space–time-coil reconstruction network without k-space data consistency for fast motion-resolved 4D MRI,” _Magnetic Resonance in Medicine_ , vol. 91, pp. 600–614, 2 2024.
  * [225] M. O. Malavé, C. A. Baron, S. P. Koundinyan, C. M. Sandino, F. Ong, J. Y. Cheng, and D. G. Nishimura, “Reconstruction of undersampled 3D non-Cartesian image-based navigators for coronary MRA using an unrolled deep learning model,” _Magnetic resonance in medicine_ , vol. 84, no. 2, pp. 800–812, 2020.
  * [226] Q. Zou, L. A. Torres, S. B. Fain, N. S. Higano, A. J. Bates, and M. Jacob, “Dynamic imaging using motion-compensated smoothness regularization on manifolds (MoCo-SToRM),” _Physics in medicine & biology_, vol. 67, no. 14, p. 144001, 2022.
  * [227] J. N. Freedman, O. J. Gurney-Champion, S. Nill, A.-M. Shiarli, H. E. Bainbridge, H. C. Mandeville, D.-M. Koh, F. McDonald, M. Kachelrieß, U. Oelfke _et al._ , “Rapid 4D-MRI reconstruction using a deep radial convolutional neural network: Dracula,” _Radiotherapy and Oncology_ , vol. 159, pp. 209–217, 2021.
  * [228] M. L. Terpstra, M. Maspero, F. d’Agata, B. Stemkens, M. P. Intven, J. J. Lagendijk, C. A. Van den Berg, and R. H. Tijssen, “Deep learning-based image reconstruction and motion estimation from undersampled radial k-space for real-time MRI-guided radiotherapy,” _Physics in Medicine & Biology_, vol. 65, no. 15, p. 155015, 2020.
  * [229] D. E. Waddington, N. Hindley, N. Koonjoo, C. Chiu, T. Reynolds, P. Z. Liu, B. Zhu, D. Bhutto, C. Paganelli, P. J. Keall _et al._ , “Real-time radial reconstruction with domain transform manifold learning for MRI-guided radiotherapy,” _Medical Physics_ , vol. 50, no. 4, pp. 1962–1974, 2023.
  * [230] C. M. Sandino, E. K. Cole, C. Alkan, A. S. Chaudhari, A. M. Loening, D. Hyun, J. Dahl, A.-A.-Z. Imran, A. S. Wang, and S. S. Vasanawala, “Upstream Machine Learning in Radiology,” _Radiologic Clinics of North America_ , vol. 59, no. 6, p. 967–985, Nov. 2021.
  * [231] A. S. Chaudhari, C. M. Sandino, E. K. Cole, D. B. Larson, G. E. Gold, S. S. Vasanawala, M. P. Lungren, B. A. Hargreaves, and C. P. Langlotz, “Prospective Deployment of Deep Learning in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices,” _Journal of Magnetic Resonance Imaging_ , vol. 54, no. 2, p. 357–371, Aug. 2020.
  * [232] R. Zhao, B. Yaman, Y. Zhang, R. Stewart, A. Dixon, F. Knoll, Z. Huang, Y. W. Lui, M. S. Hansen, and M. P. Lungren, “fastMRI+, Clinical pathology annotations for knee and brain fully sampled magnetic resonance imaging data,” _Scientific Data_ , vol. 9, no. 1, Apr. 2022.
  * [233] R. Zhao, Y. Zhang, B. Yaman, M. P. Lungren, and M. S. Hansen, “End-to-End AI-based MRI Reconstruction and Lesion Detection Pipeline for Evaluation of Deep Learning Image Reconstruction,” 2021.
  * [234] T. Weber, M. Ingrisch, B. Bischl, and D. Rügamer, “Constrained Probabilistic Mask Learning for Task-Specific Undersampled MRI Reconstruction,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , January 2024, pp. 7665–7674.
  * [235] A. D. Desai, F. Caliva, C. Iriondo, A. Mortazi, S. Jambawalikar, U. Bagci, M. Perslev, C. Igel, E. B. Dam, S. Gaj, M. Yang, X. Li, C. M. Deniz, V. Juras, R. Regatte, G. E. Gold, B. A. Hargreaves, V. Pedoia, A. S. Chaudhari, N. Khosravan, D. Torigian, J. Ellermann, M. Akcakaya, R. Tibrewala, I. Flament, M. O’Brien, S. Majumdar, K. Nakamura, and A. Pai, “The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset,” _Radiology: Artificial Intelligence_ , vol. 3, no. 3, p. e200078, May 2021.
  * [236] A. M. Schmidt, A. D. Desai, L. E. Watkins, H. A. Crowder, M. S. Black, V. Mazzoli, E. B. Rubin, Q. Lu, J. W. MacKay, R. D. Boutin, F. Kogan, G. E. Gold, B. A. Hargreaves, and A. S. Chaudhari, “Generalizability of Deep Learning Segmentation Algorithms for Automated Assessment of Cartilage Morphology and ¡scp¿MRI¡/scp¿ Relaxometry,” _Journal of Magnetic Resonance Imaging_ , vol. 57, no. 4, p. 1029–1039, Jul. 2022.
  * [237] Z. Wu, T. Yin, Y. Sun, R. Frost, A. van der Kouwe, A. V. Dalca, and K. L. Bouman, “Learning Task-Specific Strategies for Accelerated MRI,” 2023.
  * [238] M. Uecker _et al._ , “ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA,” _Magnetic Resonance in Medicine_ , vol. 71, no. 3, pp. 990–1001, 2014.
  * [239] M. Uecker, T. Hohage, K. T. Block, and J. Frahm, “Image reconstruction by regularized nonlinear inversion—joint estimation of coil sensitivities and image content,” _Magnetic Resonance in Medicine_ , vol. 60, no. 3, pp. 674–682, 2008.
  * [240] L. Ying and J. Sheng, “Joint image reconstruction and sensitivity estimation in SENSE (JSENSE),” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 57, no. 6, pp. 1196–1202, 2007.
  * [241] A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson, “End-to-end variational networks for accelerated MRI reconstruction,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23_. Springer, 2020, pp. 64–73.
  * [242] Y. Jun, H. Shin, T. Eo, and D. Hwang, “Joint deep model-based MR image and coil sensitivity reconstruction network (joint-ICNet) for fast MRI,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5270–5279.
  * [243] G. Luo, X. Wang, V. Roeloffs, Z. Tan, and M. Uecker, “Joint estimation of coil sensitivities and image content using a deep image prior,” in _Proceedings of the 29th Annual Meeting of ISMRM_ , 2021, p. 280.
  * [244] M. Zhang, J. Xu, Y. Arefeen, and E. Adalsteinsson, “Zero-Shot Self-Supervised Joint Temporal Image and Sensitivity Map Reconstruction via Linear Latent Space,” _arXiv preprint arXiv:2303.02254_ , 2023.
  * [245] J. Liu, S. Pasumarthi, B. Duffy, E. Gong, K. Datta, and G. Zaharchuk, “One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation,” _IEEE Transactions on Medical Imaging_ , 2023.
  * [246] B. Sveinsson, A. S. Chaudhari, B. Zhu, N. Koonjoo, M. Torriani, G. E. Gold, and M. S. Rosen, “Synthesizing quantitative T2 maps in right lateral knee femoral condyles from multicontrast anatomic data with a conditional generative adversarial network,” _Radiology: Artificial Intelligence_ , vol. 3, no. 5, p. e200122, 2021.
  * [247] C. Zhao, M. Shao, A. Carass, H. Li, B. E. Dewey, L. M. Ellingsen, J. Woo, M. A. Guttman, A. M. Blitz, M. Stone _et al._ , “Applications of a deep learning method for anti-aliasing and super-resolution in MRI,” _Magnetic resonance imaging_ , vol. 64, pp. 132–141, 2019.
  * [248] Y. Li, B. Sixou, and F. Peyrin, “A review of the deep learning methods for medical images super resolution problems,” _Irbm_ , vol. 42, no. 2, pp. 120–133, 2021.
  * [249] M. W. Haskell, J.-F. Nielsen, and D. C. Noll, “Off-resonance artifact correction for mri: A review,” _NMR in Biomedicine_ , vol. 36, no. 5, p. e4867, 2023.
  * [250] R. Ayde, T. Senft, N. Salameh, and M. Sarracanie, “Deep learning for fast low-field MRI acquisitions,” _Scientific reports_ , vol. 12, no. 1, p. 11394, 2022.
  * [251] N. Koonjoo, B. Zhu, G. C. Bagnall, D. Bhutto, and M. Rosen, “Boosting the signal-to-noise of low-field MRI with deep learning image reconstruction,” _Scientific reports_ , vol. 11, no. 1, pp. 1–16, 2021.
  * [252] K. Lei, A. B. Syed, X. Zhu, J. M. Pauly, and S. V. Vasanawala, “Automated MRI field of view prescription from region of interest prediction by intra-stack attention neural network,” _Bioengineering_ , vol. 10, no. 1, p. 92, 2023.
  * [253] L. C. Bell, E. Shimron, and D. W. S. S. Francisco, “Sharing Data Is Essential for the Future of AI in Medical Imaging,” _Radiology: Artificial Intelligence_ , 11 2023.
  * [254] Z. Ramzi, P. Ciuciu, and J.-L. Starck, “Benchmarking Deep Nets MRI Reconstruction Models on the FastMRI Publicly Available Dataset,” in _ISBI 2020 - International Symposium on Biomedical Imaging_ , 2020.
  * [255] Y. Lim, A. Toutios, Y. Bliesener, Y. Tian, S. G. Lingala, C. Vaz, T. Sorensen, M. Oh, S. Harper, W. Chen _et al._ , “A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images,” _Scientific data_ , vol. 8, no. 1, pp. 1–14, 2021.
  * [256] C. C. et al., “OCMR (v1.0)–Open-Access Multi-Coil k-Space Dataset for Cardiovascular Magnetic Resonance Imaging,” _arXiv: 2008.03410v2_ , 2020.
  * [257] M. Lyu, L. Mei, S. Huang, S. Liu, Y. Li, K. Yang, Y. Liu, Y. Dong, L. Dong, and E. X. Wu, “M4Raw: A multi-contrast, multi-repetition, multi-channel MRI k-space dataset for low-field MRI research,” _Scientific Data_ , vol. 10, no. 1, p. 264, 2023.
  * [258] “The Human Connectome Project, University of Souterhn California,,” http://www.humanconnectomeproject.org/, 2011, accessed: 2021-03-22. [Online]. Available: http://www.humanconnectomeproject.org/
  * [259] “IXI dataset,” http://brain-development.org/ixi-dataset/, 2010, accessed: 2021-03-22. [Online]. Available: http://brain-development.org/ixi-dataset/
  * [260] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest _et al._ , “The multimodal brain tumor image segmentation benchmark (BRATS),” _IEEE transactions on medical imaging_ , vol. 34, no. 10, pp. 1993–2024, 2014.
  * [261] M. Ghaffari, A. Sowmya, and R. Oliver, “Automated brain tumor segmentation using multimodal brain scans: a survey based on models submitted to the BraTS 2012–2018 challenges,” _IEEE reviews in biomedical engineering_ , vol. 13, pp. 156–168, 2019.
  * [262] C. R. Jack Jr, M. A. Bernstein, N. C. Fox, P. Thompson, G. Alexander, D. Harvey, B. Borowski, P. J. Britson, J. L. Whitwell, C. Ward _et al._ , “The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods,” _Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 27, no. 4, pp. 685–691, 2008.
  * [263] W. Ollier, T. Sprosen, and T. Peakman, “UK Biobank: from concept to reality,” 2005.
  * [264] B. Kurdi, S. Lozano, and M. R. Banaji, “Introducing the open affective standardized image set (OASIS),” _Behavior research methods_ , vol. 49, pp. 457–470, 2017.
  * [265] A. Paszke _et al._ , “PyTorch: An imperative style, high-performance deep learning library,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [266] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [267] M. Uecker _et al._ , “Berkeley advanced reconstruction toolbox,” in _Proc. Intl. Soc. Mag. Reson. Med_ , vol. 23, no. 2486, 2015.
  * [268] M. Blumenthal, G. Luo, M. Schilling, H. C. M. Holme, and M. Uecker, “Deep, deep learning with BART,” _Magnetic Resonance in Medicine_ , vol. 89, no. 2, pp. 678–693, 2023.
  * [269] M. S. Hansen and T. S. Sørensen, “Gadgetron: an open source framework for medical image reconstruction,” _Magnetic resonance in medicine_ , vol. 69, no. 6, pp. 1768–1776, 2013.
  * [270] F. Ong and M. Lustig, “SigPy: a python package for high performance iterative reconstruction,” in _Proc. Intl. Soc. Mag. Reson. Med_ , vol. 4819, 2019.
  * [271] K. J. Layton, S. Kroboth, F. Jia, S. Littin, H. Yu, J. Leupold, J.-F. Nielsen, T. Stöcker, and M. Zaitsev, “Pulseq: a rapid and hardware-independent pulse sequence prototyping framework,” _Magnetic resonance in medicine_ , vol. 77, no. 4, pp. 1544–1552, 2017.
  * [272] K. Herz, S. Mueller, O. Perlman, M. Zaitsev, L. Knutsson, P. Z. Sun, J. Zhou, P. van Zijl, K. Heinecke, P. Schuenke _et al._ , “Pulseq-CEST: Towards multi-site multi-vendor compatibility and reproducibility of CEST experiments using an open-source sequence standard,” _Magnetic resonance in medicine_ , vol. 86, no. 4, pp. 1845–1858, 2021.
  * [273] L. Ning, E. Bonet-Carne, F. Grussu, F. Sepehrband, E. Kaden, J. Veraart, S. B. Blumberg, C. S. Khoo, M. Palombo, I. Kokkinos _et al._ , “Cross-scanner and cross-protocol multi-shell diffusion MRI data harmonization: Algorithms and results,” _Neuroimage_ , vol. 221, p. 117128, 2020.
  * [274] A. Karakuzu, M. Boudreau, T. Duval, T. Boshkovski, I. Leppert, J.-F. Cabana, I. Gagnon, P. Beliveau, G. Pike, J. Cohen-Adad _et al._ , “qMRLab: Quantitative MRI analysis, under one umbrella,” _Journal of Open Source Software_ , vol. 5, no. 53, 2020.
  * [275] S. J. Inati, J. D. Naegele, N. R. Zwart, V. Roopchansingh, M. J. Lizak, D. C. Hansen, C.-Y. Liu, D. Atkinson, P. Kellman, S. Kozerke _et al._ , “ISMRM Raw data format: A proposed standard for MRI raw datasets,” _Magnetic resonance in medicine_ , vol. 77, no. 1, pp. 411–421, 2017.
  * [276] G. Elmas, S. U. Dar, Y. Korkmaz, E. Ceyani, B. Susam, M. Ozbey, S. Avestimehr, and T. Çukur, “Federated learning of generative image priors for MRI reconstruction,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [277] B. R. Levac, M. Arvinte, and J. I. Tamir, “Federated End-to-End Unrolled Models for Magnetic Resonance Image Reconstruction,” _Bioengineering_ , vol. 10, no. 3, p. 364, 2023.
  * [278] P. M. Johnson, G. Jeong, K. Hammernik, J. Schlemper, C. Qin, J. Duan, D. Rueckert, J. Lee, N. Pezzotti, E. De Weerdt _et al._ , “Evaluation of the robustness of learned MR image reconstruction to systematic deviations between training and test data for the models from the fastMRI challenge,” in _Machine Learning for Medical Image Reconstruction: 4th International Workshop, MLMIR 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings 4_. Springer, 2021, pp. 25–34.
  * [279] M. Z. Darestani, A. Chaudhari, and R. Heckel, “Measuring Robustness in Deep Learning Based Compressive Sensing,” arXiv preprint arXiv:2102.06103, 2021.
  * [280] N. Avidan and M. Freiman, “Physically-primed deep-neural-networks for generalized undersampled MRI reconstruction,” _arXiv preprint arXiv:2209.00462_ , 2022.
  * [281] F. Knoll, K. Hammernik, E. Kobler, T. Pock, M. P. Recht, and D. K. Sodickson, “Assessment of the generalization of learned image reconstruction and the potential for transfer learning,” _Magnetic resonance in medicine_ , vol. 81, no. 1, pp. 116–128, 2019.
  * [282] J. Huang, S. Wang, G. Zhou, W. Hu, and G. Yu, “Evaluation on the generalization of a learned convolutional neural network for MRI reconstruction,” _Magnetic Resonance Imaging_ , vol. 87, pp. 38–46, 2022.
  * [283] S. U. H. Dar, M. Özbey, A. B. Çatlı, and T. Çukur, “A transfer-learning approach for accelerated MRI using deep neural networks,” _Magnetic resonance in medicine_ , vol. 84, no. 2, pp. 663–685, 2020.
  * [284] K. Lin and R. Heckel, “Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Data,” _arXiv preprint arXiv:2312.10271_ , 2023.
  * [285] G. Varoquaux and V. Cheplygina, “Machine learning for medical imaging: methodological failures and recommendations for the future,” _NPJ digital medicine_ , vol. 5, no. 1, p. 48, 2022.
  * [286] L. Seyyed-Kalantari, H. Zhang, M. B. McDermott, I. Y. Chen, and M. Ghassemi, “Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations,” _Nature medicine_ , vol. 27, no. 12, pp. 2176–2182, 2021.
  * [287] R. J. Chen, J. J. Wang, D. F. Williamson, T. Y. Chen, J. Lipkova, M. Y. Lu, S. Sahai, and F. Mahmood, “Algorithmic fairness in artificial intelligence for medicine and healthcare,” _Nature Biomedical Engineering_ , vol. 7, no. 6, pp. 719–742, 2023.
  * [288] E. Shimron, J. I. Tamir, K. Wang, and M. Lustig, “Implicit data crimes: Machine learning bias arising from misuse of public data,” _Proceedings of the National Academy of Sciences_ , vol. 119, no. 13, p. e2117203119, 2022.
  * [289] N. Deveshwar, A. Rajagopal, S. Sahin, E. Shimron, and P. E. Larson, “Synthesizing Complex-Valued Multicoil MRI Data from Magnitude-Only Images,” _Bioengineering_ , vol. 10, no. 3, p. 358, 2023.
  * [290] Q. Yang, Y. Lin, J. Wang, J. Bao, X. Wang, L. Ma, Z. Zhou, Q. Yang, S. Cai, H. He _et al._ , “Model-based synthetic data-driven learning (MOST-DL): Application in single-shot T 2 mapping with severe head motion using overlapping-echo acquisition,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 11, pp. 3167–3181, 2022.
  * [291] G. Luo, X. Wang, M. Blumenthal, M. Schilling, E. H. U. Rauf, R. Kotikalapudi, N. Focke, and M. Uecker, “Generative Image Priors for MRI Reconstruction Trained from Magnitude-Only Images,” _arXiv preprint arXiv:2308.02340_ , 2023.
  * [292] C. Zhang, Q. Yang, L. Fan, S. Yu, L. Sun, C. Cai, and X. Ding, “Towards better generalization using synthetic data: A domain adaptation framework for t 2 mapping via multiple overlapping-echo acquisition,” _IEEE Transactions on Medical Imaging_ , 2023.
  * [293] S. Bhadra, V. A. Kelkar, F. J. Brooks, and M. A. Anastasio, “On hallucinations in tomographic image reconstruction,” _IEEE transactions on medical imaging_ , vol. 40, no. 11, pp. 3249–3260, 2021.
  * [294] J. P. Cohen, M. Luck, and S. Honari, “Distribution matching losses can hallucinate features in medical image translation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I_. Springer, 2018, pp. 529–536.
  * [295] N. M. Gottschling, V. Antun, B. Adcock, and A. C. Hansen, “The troublesome kernel: why deep learning for inverse problems is typically unstable,” _arXiv preprint arXiv:2001.01258_ , 2020.
  * [296] Y. Huang, T. Würfl, K. Breininger, L. Liu, G. Lauritsch, and A. Maier, “Some investigations on robustness of deep learning in limited angle tomography,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 145–153.
  * [297] V. Antun, F. Renna, C. Poon, B. Adcock, and A. C. Hansen, “On instabilities of deep learning in image reconstruction and the potential costs of ai,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 48, pp. 30 088–30 095, 2020.
  * [298] A. Krainovic, M. Soltanolkotabi, and R. Heckel, “Learning Provably Robust Estimators for Inverse Problems via Jittering,” 2023.
  * [299] J. N. Morshuis, S. Gatidis, M. Hein, and C. F. Baumgartner, “Adversarial robustness of MR image reconstruction under realistic perturbations,” in _International Workshop on Machine Learning for Medical Image Reconstruction_. Springer, 2022, pp. 24–33.
  * [300] A. Goujon, S. Neumayer, P. Bohra, S. Ducotterd, and M. Unser, “A neural-network-based convex regularizer for inverse problems,” _IEEE Transactions on Computational Imaging_ , 2023.
  * [301] M. John, J. R. Chand, and M. Jacob, “Local monotone operator learning using non-monotone operators: MnM-MOL,” _arXiv preprint arXiv:2312.00386_ , 2023.
  * [302] F. Calivá, K. Cheng, R. Shah, and V. Pedoia, “Adversarial robust training of deep learning MRI reconstruction models,” _arXiv preprint arXiv:2011.00070_ , 2020.
  * [303] K. Cheng, F. Calivá, R. Shah, M. Han, S. Majumdar, and V. Pedoia, “Addressing the False Negative Problem of Deep Learning MRI Reconstruction Models by Adversarial Attacks and Robust Training,” 2020, pp. 121–135.
  * [304] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [305] A. Mason, J. Rioux, S. E. Clarke, A. Costa, M. Schmidt, V. Keough, T. Huynh, and S. Beyea, “Comparison of objective image quality metrics to expert radiologists’ scoring of diagnostic quality of MR images,” _IEEE transactions on medical imaging_ , vol. 39, no. 4, pp. 1064–1072, 2019.
  * [306] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_. Springer, 2016, pp. 694–711.
  * [307] P. M. Adamson, A. D. Desai, J. Dominic, C. Bluethgen, J. P. Wood, A. B. Syed, R. D. Boutin, K. J. Stevens, S. Vasanawala, J. M. Pauly _et al._ , “Using Deep Feature Distances for Evaluating MR Image Reconstruction Quality,” in _NeurIPS 2023 Workshop on Deep Learning and Inverse Problems_ , 2023.
  * [308] K. Wang, J. I. Tamir, A. D. Goyeneche, U. Wollner, R. Brada, S. X. Yu, and M. Lustig, “High fidelity deep learning-based MRI reconstruction with instance-wise discriminative feature matching loss,” _Magnetic Resonance in Medicine_ , vol. 88, pp. 476–491, 7 2022.
  * [309] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” 2018.
  * [310] P. M. Adamson, B. Gunel, J. Dominic, A. D. Desai, D. Spielman, S. Vasanawala, J. M. Pauly, and A. Chaudhari, “SSFD: Self-supervised feature distance as an MR image reconstruction quality metric,” in _NeurIPS 2021 Workshop on Deep Learning and Inverse Problems_ , 2021.
  * [311] A. S. Chaudhari, F. Kogan, V. Pedoia, S. Majumdar, G. E. Gold, and B. A. Hargreaves, “Rapid Knee MRI Acquisition and Analysis Techniques for Imaging Osteoarthritis,” _Journal of Magnetic Resonance Imaging_ , vol. 52, no. 5, p. 1321–1339, Nov. 2019.
  * [312] A. S. Chaudhari, M. J. Grissom, Z. Fang, B. Sveinsson, J. H. Lee, G. E. Gold, B. A. Hargreaves, and K. J. Stevens, “Diagnostic accuracy of quantitative multicontrast 5-minute knee MRI using prospective artificial intelligence image quality enhancement,” _AJR. American journal of roentgenology_ , vol. 216, no. 6, p. 1614, 2021.
  * [313] K. C. Tezcan, N. Karani, C. F. Baumgartner, and E. Konukoglu, “Sampling possible reconstructions of undersampled acquisitions in MR imaging with a deep learned prior,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 7, pp. 1885–1896, 2022.
  * [314] V. Edupuganti, M. Mardani, S. Vasanawala, and J. Pauly, “Uncertainty quantification in deep MRI reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 1, pp. 239–250, 2020.
  * [315] D. Narnhofer, A. Effland, E. Kobler, K. Hammernik, F. Knoll, and T. Pock, “Bayesian uncertainty estimation of learned variational MRI reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 2, pp. 279–291, 2021.
  * [316] T. Küstner, K. Hammernik, D. Rueckert, T. Hepp, and S. Gatidis, “Predictive uncertainty in deep learning-based MR image reconstruction using deep ensembles: Evaluation on the fastMRI data set.” _Magnetic Resonance in Medicine_ , 2024.
  * [317] K. Wang, A. Angelopoulos, A. De Goyeneche, A. Kohli, E. Shimron, S. Yu, J. Malik, and M. Lustig, “Rigorous Uncertainty Estimation for MRI Reconstruction.”
  * [318] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [319] Z. Zhang, A. Romero, M. J. Muckley, P. Vincent, L. Yang, and M. Drozdzal, “Reducing uncertainty in undersampled MRI reconstruction with active acquisition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2049–2058.
  * [320] P. Fischer, K. Thomas, and C. F. Baumgartner, “Uncertainty Estimation and Propagation in Accelerated MRI Reconstruction,” in _International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging_. Springer, 2023, pp. 84–94.
  * [321] T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” _arXiv preprint arXiv:1604.06174_ , 2016.
  * [322] M. Kellman, K. Zhang, E. Markley, J. Tamir, E. Bostan, M. Lustig, and L. Waller, “Memory-efficient learning for large-scale computational imaging,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 1403–1414, 2020.
  * [323] S. Bai, J. Z. Kolter, and V. Koltun, “Deep equilibrium models,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.