# Multi-Contrast MRI Segmentation Trained on Synthetic Images

Ismail Irmakci1, Zeki Emre Unel2, Nazli Ikizler-Cinbis2 and Ulas Bagci1
1Machine and Hybrid Intelligence Lab, Northwestern University Feinberg School
of Medicine, Chicago, IL 60611, USA2Department of Computer Engineering,
Hacettepe University, Ankara, Turkey ∗Correspondence to
<EMAIL_ADDRESS>
###### Abstract

In our comprehensive experiments and evaluations, we show that it is possible
to generate multiple contrast (even all synthetically) and use synthetically
generated images to train an image segmentation engine. We showed promising
segmentation results tested on real multi-contrast MRI scans when delineating
muscle, fat, bone and bone marrow, all trained on synthetic images. Based on
synthetic image training, our segmentation results were as high as 93.91%,
94.11%, 91.63%, 95.33%, for muscle, fat, bone, and bone marrow delineation,
respectively. Results were not significantly different from the ones obtained
when real images were used for segmentation training: 94.68%, 94.67%, 95.91%,
and 96.82%, respectively.

Clinical relevance— Synthetically generated images could potentially be used
in large-scale training of deep networks for segmentation purpose. Small data
set problem of many clinical imaging problems can potentially be addressed
with the proposed algorithm.

##  I INTRODUCTION

Segmentation of medical images is a long-standing problem with excessive
number of deep learning based methods already available nowadays. Although
there are recent paradigm-shifting works at segmentation literature such as
capsule based segmentation [1] and transformer based segmentation [2], most of
the medical image segmentation literature in deep learning field are based on
standard U-Net or its derivation based methodologies. In this study, we
approach the segmentation from a slightly different angle where our unique
clinical imaging conditions infer some constraints on the problem formulation.
In many clinical scenarios, for instance, multi-modality images are necessary
for a more appropriate evaluation of the clinical condition through better
tissue characterization (anatomically and/or functionally). Multi-modal brain
imaging, PET/CT, PET/MRI, and multi-contrast MRIs are some of the mostly used
examples in this context.

Figure 1: MRI contrasts (first row): fat-suppressed, water-fat, water-
suppressed. Segmented tissues-muscle, fat, bone and bone marrow-(second row).

Despite the strengths of combining multiple modality images to characterize a
clinical condition better, or quantify, there are further challenges to be
addressed. First, handling more than one modality for image segmentation is
already more challenging than single modality images. Second, multi-object
segmentation is another hurdle compared to single object segmentation, which
is often the case in multi-modality image analysis. Third, clinical workflow
has deficiencies and not always all modalities are available for further
analysis. Missing slices or missing scans are not rare especially in multi-
contrast evaluation of MRI scans. In this study, our goal is to develop a
successful segmentation strategy, based on deep networks, that accepts multi-
contrast MRI scans and perform a multi-tissue segmentation even there are
missing scans.

To achieve our overall goal by addressing the challenges defined above, we
focus on musculoskeletal (MSK) radiology examples: delineation of thigh
tissues from multi-contrast MRI scans. Figure 1 demonstrates such a multi-
contrast MRI scan’s slices from the same patient, from left to right in top
row: Fat-suppressed: MRI1, water-fat: MRI2, and water-suppressed: MRI3. Our
clinical motivation comes from the fact that MSK radiology applications are
critical for several diseases spanning from obesity, metabolic syndromes to
cartilage quantification. For instance, according to American Cancer Society
studies in 2021 [3], some of the most effective measures for decreasing cancer
risk are having a healthy body weight, healthy diet and being physically
active. Excess body weight (obesity), alcohol consumption, physical
inactivity, and a poor diet are thought to be responsible for 18% of cancer
cases and 16% of cancer deaths. Of all cancer risk factors, excess body weight
is believed to be responsible for 5% of cancers in males and 11% of cancers in
women. In this respect, sarcopenia is related with general loss of body mass
and excess body weight , and have strong relation with cancer risk factors
[4].

In this work, we propose a systematic approach to (1) synthesize MRI
contrasts, (2) train synthesized images on a deep learning based segmentation
engine, and (3) evaluate the efficacy of the segmentation model on true multi-
contrast MRI images. We target segmenting thigh tissues (muscle, fat, bone and
bone marrow). We also conduct an ablation study where training model includes
true, synthesized, and mixed (true and synthesized) images towards a
segmentation model. Comprehensive quantitative and qualitative segmentation
results showed that proposed approach can be used effectively for multi-modal
image analysis. This is essentially useful when there is not enough medical
imaging data, a typical constraint in medical imaging problems. Our major
contributions are as follows:

  * •

Application-wise our study is the first one handling missing contrast issue
while retaining a high accuracy in segmenting multiple tissues from thigh MRI.

  * •

Our method is generic, any deep segmentation or GAN based methods can be
replaced within our framework.

  * •

We present a comprehensive evaluation, carefully analyzing three contrasts of
MRI, their relations and effect on the final segmentation results.

  * •

We examine whether it is robust and feasible enough to run a segmentation
training on completely synthesized and mixed data, opening new discussions
about the use of completely synthesized data to obtain clinically accepted
segmentation results on real MRI data.

##  II Related Work

There is a relatively small body of literature that is concerned with muscle,
fat, bone and bone marrow segmentation in MSK radiology despite its clinical
importance [5]. Available deep learning based studies focus on U-Net based
standard segmentation methods on single or multi-tissues but mostly in single
modality MRI scans. When there are missing scans, there is no particular
method presented for MSK applications.

GAN (generative adversarial networks) based methods are being increasingly
used for several applications spanning from brain imaging to functional
imaging. One interesting work utilizes a multi-modal generative adversarial
network (MM-GAN) [6], a variant of pix2pix [7] network. Authors integrate
multi-modal data from existing Brain image sequences in a single forward pass
training to synthesize missing sequences. In another major work [8], authors
used popular CycleGAN network on thigh MRI for increasing the data size for
segmentation purposes. Our work has some similarities with this work, but
unlike focusing on data augmentation aspect of particular tissue, we generate
the whole sequence(s), using them to train segmentation models, and explore
the relationship of MRI contrast in an ablation study, leading us to train the
complete segmentation process on synthetic MRI scans.

In pre-deep learning era, there are some segmentation studies available too.
It might be worth to mention that [9] proposed an architecture based novel
affinity propagation within the fuzzy connectivity for segmenting multi
contrast thigh MRI. The most recent work in this domain is handling the lack
of labeling problem from a semi-supervised deep learning aspect [10] utilizing
Tiramisu network. However, the synthesis of one or more MRI contrasts remains
a major challenge, and not considered in those works. Herein we propose a
comprehensive evaluation and generic approach for handling missing MRI
contrast and its effect on multi-tissue segmentation problems.

Figure 2: Fat-suppressed: MRI1, water-fat: MRI2, water-suppressed: MRI3. Left.
Generation procedure for all Synthesized MRI contrasts. (A) Synthesized
generations from only R MRI1 B) Synthesized generations from only R MRI2 C)
Synthesized generations from only R MRI3. Right. Different combinations of MRI
synthesis procedure are shown where ($\leftarrow$ or $\rightarrow$) indicates
synthesis. For example, R MRI1$\rightarrow$F MRI2 or F MRI1$\leftarrow$R MRI2
both indicates synthesize operation from Real contrasts. TABLE I: Segmentation
performance of Single Input Multi Output MR contrasts (5-fold cross
validation) (Avg.=Average and Std.=Standard deviation)

SINGLE INPUT | MUSCLE | FAT | BONE | BONE MARROW
---|---|---|---|---
|  | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC.
R MRI1 | Avg. | 0,9264 | 0,9831 | 0,9521 | 0,9868 | 0,8826 | 0,9793 | 0,8985 | 0,9897 | 0,8245 | 0,9985 | 0,8383 | 0,9992 | 0,8397 | 0,9994 | 0,8482 | 0,9997
| Std | 0,0340 | 0,0110 | 0,0337 | 0,0097 | 0,0956 | 0,0206 | 0,0622 | 0,0126 | 0,0943 | 0,0008 | 0,0969 | 0,0005 | 0,0943 | 0,0004 | 0,1134 | 0,0002
R MRI2 | Avg. | 0,9312 | 0,9846 | 0,9541 | 0,9883 | 0,9100 | 0,9870 | 0,9246 | 0,9917 | 0,9591 | 0,9997 | 0,9612 | 0,9999 | 0,9682 | 0,9999 | 0,9693 | 1,0000
| Std | 0,0384 | 0,0100 | 0,0448 | 0,0078 | 0,0811 | 0,0122 | 0,0619 | 0,0087 | 0,0321 | 0,0002 | 0,0422 | 0,0001 | 0,0254 | 0,0001 | 0,0420 | 0,0001
R MRI3 | Avg. | 0,9468 | 0,9884 | 0,9587 | 0,9919 | 0,9467 | 0,9925 | 0,9608 | 0,9945 | 0,8296 | 0,9985 | 0,8324 | 0,9992 | 0,8897 | 0,9996 | 0,8848 | 0,9998
| Std | 0,0211 | 0,0057 | 0,0276 | 0,0061 | 0,0411 | 0,0057 | 0,0205 | 0,0058 | 0,0919 | 0,0009 | 0,1045 | 0,0007 | 0,0846 | 0,0003 | 0,1023 | 0,0002
F MRI1($\leftarrow$ R MRI2) TEST ON R MRI1 | Avg. | 0,9063 | 0,9774 | 0,9805 | 0,9770 | 0,8815 | 0,9817 | 0,9112 | 0,9882 | 0,8445 | 0,9986 | 0,8665 | 0,9992 | 0,8514 | 0,9994 | 0,8527 | 0,9997
| Std. | 0,0387 | 0,0130 | 0,0248 | 0,0127 | 0,0813 | 0,0157 | 0,0526 | 0,0115 | 0,0999 | 0,0008 | 0,0999 | 0,0006 | 0,0970 | 0,0004 | 0,1192 | 0,0002
F MRI1($\leftarrow$ R MRI3) TEST ON R MRI1 | Avg. | 0,9239 | 0,9824 | 0,9550 | 0,9856 | 0,8920 | 0,9828 | 0,9301 | 0,9875 | 0,8206 | 0,9985 | 0,8263 | 0,9992 | 0,8362 | 0,9994 | 0,8372 | 0,9997
| Std. | 0,0326 | 0,0108 | 0,0334 | 0,0100 | 0,0951 | 0,0172 | 0,0537 | 0,0139 | 0,1017 | 0,0008 | 0,1021 | 0,0005 | 0,0951 | 0,0004 | 0,1168 | 0,0002
F MRI2($\leftarrow$ R MRI1) TEST ON R MRI2 | Avg. | 0,9189 | 0,9813 | 0,9661 | 0,9831 | 0,9049 | 0,9856 | 0,9468 | 0,9889 | 0,9163 | 0,9993 | 0,9387 | 0,9995 | 0,9443 | 0,9998 | 0,9550 | 0,9999
| Std. | 0,0364 | 0,0106 | 0,0309 | 0,0095 | 0,0815 | 0,0136 | 0,0539 | 0,0104 | 0,0434 | 0,0003 | 0,0403 | 0,0003 | 0,0315 | 0,0001 | 0,0521 | 0,0001
F MRI2($\leftarrow$ R MRI3) TEST ON R MRI2 | Avg. | 0,9211 | 0,9824 | 0,9422 | 0,9873 | 0,9094 | 0,9863 | 0,9312 | 0,9907 | 0,9044 | 0,9992 | 0,9061 | 0,9996 | 0,9533 | 0,9998 | 0,9547 | 0,9999
| Std. | 0,0400 | 0,0106 | 0,0491 | 0,0081 | 0,0812 | 0,0129 | 0,0567 | 0,0098 | 0,0414 | 0,0003 | 0,0552 | 0,0003 | 0,0277 | 0,0001 | 0,0532 | 0,0000
F MRI3($\leftarrow$ R MRI1) TEST ON R MRI3 | Avg. | 0,9386 | 0,9861 | 0,9622 | 0,9891 | 0,9411 | 0,9921 | 0,9754 | 0,9933 | 0,8089 | 0,9983 | 0,8496 | 0,9989 | 0,8796 | 0,9995 | 0,8785 | 0,9998
| Std. | 0,0249 | 0,0077 | 0,0323 | 0,0075 | 0,2356 | 0,2036 | 0,2067 | 0,2037 | 0,0998 | 0,0008 | 0,1052 | 0,0006 | 0,0954 | 0,0003 | 0,1075 | 0,0002
F MRI3($\leftarrow$ R MRI2) TEST ON R MRI3 | Avg. | 0,9391 | 0,9863 | 0,9677 | 0,9885 | 0,9382 | 0,9916 | 0,9709 | 0,9929 | 0,8358 | 0,9987 | 0,8370 | 0,9993 | 0,8952 | 0,9996 | 0,8891 | 0,9998
| Std. | 0,0269 | 0,0075 | 0,0239 | 0,0078 | 0,0506 | 0,0064 | 0,0165 | 0,0068 | 0,0900 | 0,0006 | 0,0937 | 0,0005 | 0,0861 | 0,0003 | 0,0983 | 0,0002

TABLE II: Segmentation performance of Multi Input Multi Output MR contrasts
(5-fold cross validation) (Avg.=Average and Std.=Standard deviation)

MULTI INPUT | MUSCLE | FAT | BONE | BONE MARROW
---|---|---|---|---
|  | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC. | DSC. | ACC. | SENS. | SPEC.
R MRI1 R MRI2 R MRI3 | Avg. | 0,9541 | 0,9898 | 0,9655 | 0,9927 | 0,9461 | 0,9923 | 0,9595 | 0,9944 | 0,9522 | 0,9996 | 0,9502 | 0,9998 | 0,9438 | 0,9998 | 0,9400 | 0,9999
| Std | 0,0200 | 0,0056 | 0,0290 | 0,0061 | 0,0419 | 0,0062 | 0,0234 | 0,0059 | 0,0410 | 0,0003 | 0,0555 | 0,0001 | 0,0617 | 0,0002 | 0,0833 | 0,0001
F MRI1($\leftarrow$ R MRI2) F MRI2 ($\leftarrow$ R MRI1) F MRI3 ($\leftarrow$ R MRI1) | Avg. | 0,9365 | 0,9854 | 0,9660 | 0,9878 | 0,9357 | 0,9909 | 0,9557 | 0,9933 | 0,8795 | 0,9990 | 0,9079 | 0,9993 | 0,8928 | 0,9996 | 0,8941 | 0,9998
| Std. | 0,0280 | 0,0087 | 0,0338 | 0,0083 | 0,0520 | 0,0078 | 0,0288 | 0,0072 | 0,0862 | 0,0007 | 0,0784 | 0,0005 | 0,0868 | 0,0003 | 0,0971 | 0,0001
F MRI1($\leftarrow$ R MRI2) F MRI2 ($\leftarrow$ R MRI1) F MRI3 ($\leftarrow$ R MRI2) | Avg. | 0,9370 | 0,9860 | 0,9419 | 0,9916 | 0,9341 | 0,9904 | 0,9656 | 0,9917 | 0,8913 | 0,9991 | 0,9146 | 0,9994 | 0,9145 | 0,9997 | 0,9152 | 0,9998
| Std. | 0,0327 | 0,0090 | 0,0533 | 0,0067 | 0,0557 | 0,0087 | 0,0252 | 0,0092 | 0,0841 | 0,0007 | 0,0774 | 0,0005 | 0,0794 | 0,0002 | 0,0863 | 0,0001
F MRI1($\leftarrow$ R MRI2) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R MRI1) | Avg. | 0,9281 | 0,9835 | 0,9496 | 0,9878 | 0,9249 | 0,9889 | 0,9544 | 0,9915 | 0,8960 | 0,9992 | 0,9056 | 0,9996 | 0,9276 | 0,9997 | 0,9337 | 0,9999
| Std. | 0,0306 | 0,0095 | 0,0424 | 0,0075 | 0,0618 | 0,0105 | 0,0363 | 0,0089 | 0,0803 | 0,0006 | 0,0788 | 0,0004 | 0,0700 | 0,0002 | 0,0850 | 0,0001
F MRI1($\leftarrow$ R MRI2) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R MRI2) | Avg. | 0,9408 | 0,9868 | 0,9549 | 0,9908 | 0,9314 | 0,9900 | 0,9558 | 0,9924 | 0,9011 | 0,9992 | 0,9108 | 0,9996 | 0,9240 | 0,9997 | 0,9303 | 0,9999
| Std. | 0,0292 | 0,0081 | 0,0430 | 0,0066 | 0,0582 | 0,0093 | 0,0374 | 0,0078 | 0,0765 | 0,0006 | 0,0727 | 0,0004 | 0,0773 | 0,0002 | 0,0858 | 0,0001
F MRI1($\leftarrow$ R MRI3) F MRI2 ($\leftarrow$ R MRI1) F MRI3 ($\leftarrow$ R MRI1) | Avg. | 0,9375 | 0,9860 | 0,9526 | 0,9902 | 0,9346 | 0,9909 | 0,9552 | 0,9930 | 0,8879 | 0,9991 | 0,8919 | 0,9995 | 0,8934 | 0,9997 | 0,8876 | 0,9999
| Std. | 0,0263 | 0,0076 | 0,0395 | 0,0069 | 0,0515 | 0,0072 | 0,0295 | 0,0073 | 0,0788 | 0,0006 | 0,0772 | 0,0004 | 0,0887 | 0,0003 | 0,1073 | 0,0001
F MRI1($\leftarrow$ R MRI3) F MRI2 ($\leftarrow$ R MRI1) F MRI3 ($\leftarrow$ R MRI2) | Avg. | 0,9406 | 0,9868 | 0,9518 | 0,9911 | 0,9349 | 0,9908 | 0,9680 | 0,9919 | 0,8925 | 0,9991 | 0,8990 | 0,9995 | 0,9127 | 0,9997 | 0,9069 | 0,9999
| Std. | 0,0249 | 0,0071 | 0,0412 | 0,0062 | 0,0515 | 0,0075 | 0,0257 | 0,0078 | 0,0693 | 0,0005 | 0,0734 | 0,0003 | 0,0757 | 0,0003 | 0,0857 | 0,0001
F MRI1($\leftarrow$ R MRI3) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R MRI1) | Avg. | 0,9288 | 0,9843 | 0,9344 | 0,9906 | 0,9254 | 0,9888 | 0,9489 | 0,9913 | 0,9061 | 0,9992 | 0,9203 | 0,9996 | 0,9375 | 0,9998 | 0,9295 | 0,9999
| Std. | 0,0336 | 0,0094 | 0,0519 | 0,0064 | 0,0624 | 0,0108 | 0,0421 | 0,0096 | 0,0570 | 0,0004 | 0,0586 | 0,0003 | 0,0526 | 0,0002 | 0,0737 | 0,0001
F MRI1($\leftarrow$ R MRI3) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R MRI2) | Avg. | 0,9399 | 0,9865 | 0,9519 | 0,9909 | 0,9335 | 0,9906 | 0,9611 | 0,9924 | 0,8840 | 0,9991 | 0,8808 | 0,9996 | 0,9180 | 0,9997 | 0,9193 | 0,9999
| Std. | 0,0256 | 0,0074 | 0,0384 | 0,0063 | 0,0537 | 0,0077 | 0,0275 | 0,0075 | 0,0794 | 0,0005 | 0,0892 | 0,0003 | 0,0756 | 0,0002 | 0,0882 | 0,0001

TABLE III: Quality of Single Modality Thigh MRIs Synthesis with 5-fold cross
validation. (Avg. = Average (Mean) and Std.=Standard deviation)

|  | F MRI1($\leftarrow$ R MRI2) | F MRI1($\leftarrow$ R MRI3) | F MRI2($\leftarrow$ R MRI1) | F MRI2($\leftarrow$ R MRI3) | F MRI3($\leftarrow$ R MRI1) | F MRI3($\leftarrow$ R MRI2)
---|---|---|---|---|---|---|---
PSNR | Avg. | 28,3153 | 27,6520 | 27,2156 | 27,9233 | 28,5335 | 28,0810
| Std. | 3,3855 | 2,9039 | 3,2992 | 3,4147 | 3,2828 | 3,7948
SSIM | Avg. | 0,8786 | 0,8848 | 0,8728 | 0,8827 | 0,8968 | 0,8890
| Std. | 0,0496 | 0,0510 | 0,0520 | 0,0608 | 0,0601 | 0,0616
FID | Avg. | 42,5333 | 41,6491 | 57,5200 | 68,3417 | 39,8365 | 54,2351
| Std. | 4,4439 | 4,6984 | 10,9396 | 20,8823 | 4,4900 | 13,9198

##  III METHOD

The proposed segmentation strategy includes both synthesis of missing
contrasts with a generator and a segmentor (Figure 2). In the generation
stage, we adapted popular pix2pix [7] conditional GAN method for synthesising
all synthesized contrasts from real (true) contrasts. Any other GAN method can
be used too. Briefly, pix2pix uses a conditional generative adversarial
network to learn mapping from a source domain $x$ and random noise $z$ to an
target domain $y$. The network is made up of two blocks, the generator ${G}$,
and the discriminator ${D}$. The generator transforms the source domain ($x$)
with random noise ($z$) to get the target domain ($y$) while discriminator
learns how much target domain is similar to source domain. As shown in Figure
2, then we use all real MR contrasts (R MRI) for synthesizing (generating)
other contrasts (F MRI) using Equation 1 and Equation 2 where $x$ is source
contrast, $y$ is target contrast, $z$ is random noise, $\lambda$ is
hyperparameter for adjusting blurriness, and $\mathcal{L}_{L1}$ mean square
error:

$\displaystyle\mathcal{L}_{cGAN}(G,D)=$ $\displaystyle\mathbb{E}_{x,y}[\log
D(x,y)]+$ (1) $\displaystyle\mathbb{E}_{x,z}[\log(1-D(x,G(x,z))].$

$G^{*}=\arg\min_{G}\max_{D}\mathcal{L}_{cGAN}(G,D)+\lambda\mathcal{L}_{L1}(G).$
(2)

First, we condition on source contrast, Real MRI1 for generating Synthesized
MRI2 (R MRI1 $\rightarrow$ F MRI2) or Synthesized MRI3 (R MRI1 $\rightarrow$ F
MRI3) separately, then we condition on Real MRI2 for generating Synthesized
MRI1 (R MRI2 $\rightarrow$ F MRI1) or Synthesized MRI3 (R MRI2 $\rightarrow$ F
MRI3), separately. Finally, we condition on Real MRI3 for generating
Synthesized MRI1 ((R MRI3 $\rightarrow$ F MRI1) and Synthesized MRI2 (R MRI3
$\rightarrow$ F MRI2) for getting all six different combinations.

For delineation of multiple tissues, we devise commonly used standard U-Net
segmentor [11]. We speculate that if synthesized images are in good quality
(Table III), overall segmentation of tissues should be accurate. In addition
to fully synthesized, mixed, and real images based training of segmentation,
we also apply multi-contrast and single-contrast segmentation settings to
validate the necessity of additional contrasts and their complementary
strengths to each other.

##  IV EXPERIMENTS and RESULTS

Dataset and Preprocessing: We have used multi-contrast MRI data from Baltimore
Longitudinal Study of Aging (BLSA) [12]. Experiments were performed on three
different T1-weighted MR contrasts: fat-suppressed (MRI1), water and fat
(MRI2),and water-suppressed (MRI3). These images are abbreviated as ”real” in
our experiments to separate them from synthesized ones. Original data set
contains 150 volumetric MRI scans from 50 subjects acquired using a 3T Philips
Achieva MRI scanner (Philips Healthcare, Best, The Netherlands). A voxel size
of $1\times 1$ mm2 in-plane, and slice thickness varies from 1 mm to 3 mm in
different scans. Details of contrasts and other imaging parameters can be
found in [12]. Prior to our experiments, we used non-uniform non-parametric
intensity normalization technique (N4ITK) [13] to remove bias afterwards edge-
preserving diffusive filter for removing noise without any distortion on
tissue structures in MR images. Finally, we used whitening transformation then
scaled voxel values between $0$ and $1$.

Network settings and Training Procedure: pix2pix was trained for 250 epoch
based on 2D slices with learning rate of 0.0001. Generator and discriminators
consist of U-Net and PatchGAN. Best models were selected on validation portion
of the whole data set. In segmentation stage, we optimized network with cross
entropy loss with ADAM optimizer and a learning rate of 0.0001. Early stopping
criteria was used. We did not use any data augmentation techniques as we did
not see any overfitting problem in training. We have performed 90 (18x5 Fold)
experiments in segmentation stage and 30 (6x5 Fold) experiments for generation
stage with 5 fold cross validation ($70\%$ training, $10\%$ validation, and
$20\%$ test). All experiments were performed on Nvidia Titan-XP GPUs with 12GB
memory. Proposed approach was implemented in PyTorch framework.

Quantitative evaluations: We report our GAN results in three different
metrics: the Frechet Inception Distance (FID, lower is better), Peak Signal to
Noise Ratio (PSNR, higher is better) and Structural Similarity Index Measure
(SSIM, higher is better) (Table III). We observed that synthesized MRI3 from
real MRI1 gives the best PSNR, SSIM and FID outperforming other synthesized
MRI contrasts. However, results are not hugely different from each other, this
is likely due to the one-to-one mapping nature between MRI contrasts and the
learned maps between contrasts are highly informative about each other. This
is a strength that can be attributed to the power of GAN.

We also summarized segmentation results with evaluation metrics of DICE,
accuracy, sensitivity, and specificity (Table III and Table III). We analyzed
the muscle, fat, bone and bone marrow segmentation results on single
synthesized and real MRI contrasts (Table III). Muscle and fat tissue shows
higher DICE scores when real MRI3 (water-suppressed) is used, and bone and
bone marrow show higher DICE scores when R MRI2 (water-fat) is used.
Surprisingly, synthesized MRI3 ($\leftarrow$ R MRI2), synthesized MRI3
($\leftarrow$ R MRI1), synthesized MRI2 ($\leftarrow$ R MRI1), synthesized
MRI2 ($\leftarrow$ R MRI3) show similar results for muscle, fat, bone and bone
marrow tissues, respectively (Table III), thanks to strongly learned mapping
between each contrast. For multi-contrast input (Table III), although true
MRIs trained segmentation results show higher DICE scores than other
strategies, synthesized images from real MRI2 and MRI3 show very close DICE
scores to that of the best results. Synthesized images based segmentation
results showed similar trends to true images based segmentation results even
when the tissue of interest is small such as bone marrow, indicating that
high-quality synthesis was achieved.

Qualitative evaluations: Qualitative results are shown in Figure 3 for muscle,
fat, bone and bone marrow tissues. We compare some best synthesized MRI image
results (Table III and Table III) to original MRIs. We observed that
synthesizing multi-contrasts MRI were high quality such that even small
details on soft tissues were preserved, justified in segmentation results too.
This observation is promising as lack of data, missing contrast and other data
problems can be addressed with synthetic images as an intermediate step while
diagnostic path is still using true/real images, thus avoiding the concern of
using synthetic images as diagnostic purpose.

Figure 3: Comparison of fat, muscle, bone and bone marrow tissue segmentation
results for a given MRI slice. (a) Original MRI, b) F MRI1($\leftarrow$ R
MRI2) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R MRI2) c) F
MRI1($\leftarrow$ R MRI3) F MRI2 ($\leftarrow$ R MRI3) F MRI3 ($\leftarrow$ R
MRI2) d) R MRI1 R MRI2 R MRI3 e) F MRI2 ($\leftarrow$ R MRI1) f) F MRI2
($\leftarrow$ R MRI3).

##  V CONCLUSIONS

In this work, we conducted extensive experiments to explore the use of
synthetic MRI scans for training segmentation engine for multi-tissue
analysis. We showed that an accurate segmentation model can be built on solely
based on synthetic scans or mixed (real + synthetic) images with a precision
level close to a segmentation model completely trained with true images. In
addition, we have demonstrated that multi-modality combination of scans
provide better segmentation results even when some of the modalities are
synthetically generated.

Acknowledgments Our study is exempt from human subject section as the data is
publicly available and fully anonymized. This study is approved under the
existing IRB at Baltimore Longitudinal Study of Aging (BLSA) [12]. This study
is partially supported by the NIH grant R01-CA246704-01 and R01-CA240639-01.
We thank Ege University for letting us to use their servers for running our
experiments.

## References

  * [1] Rodney LaLonde, Ziyue Xu, Ismail Irmakci, Sanjay Jain, and Ulas Bagci,  “Capsules for biomedical image segmentation,”  Medical image analysis, vol. 68, pp. 101889, 2021.
  * [2] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou,  “Transunet: Transformers make strong encoders for medical image segmentation,”  arXiv preprint arXiv:2102.04306, 2021.
  * [3] American Cancer Society,  “Cancer facts & figures,”  American Cancer Society, 2021.
  * [4] Jennifer A Ligibel, Kathryn H Schmitz, and Nathan A Berger,  “Sarcopenia in aging, obesity, and cancer,”  Translational cancer research, vol. 9, no. 9, pp. 5760, 2020.
  * [5] YiRang Shin, Jaemoon Yang, and Young Han Lee,  “Deep generative adversarial networks: Applications in musculoskeletal imaging,”  Radiology: Artificial Intelligence, vol. 3, no. 3, pp. e200157, 2021\.
  * [6] Anmol Sharma and Ghassan Hamarneh,  “Missing mri pulse sequence synthesis using multi-modal generative adversarial network,”  IEEE transactions on medical imaging, vol. 39, no. 4, pp. 1170–1183, 2019.
  * [7] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros,  “Image-to-image translation with conditional adversarial networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [8] Michael Gadermayr, Kexin Li, Madlaine Müller, Daniel Truhn, Nils Krämer, Dorit Merhof, and Burkhard Gess,  “Domain-specific data augmentation for segmenting mr images of fatty infiltrated human thighs with neural networks,”  Journal of Magnetic Resonance Imaging, vol. 49, no. 6, pp. 1676–1683, 2019.
  * [9] Ismail Irmakci, Sarfaraz Hussein, Aydogan Savran, Rita R Kalyani, David Reiter, Chee W Chia, Kenneth W Fishbein, Richard G Spencer, Luigi Ferrucci, and Ulas Bagci,  “A novel extension to fuzzy connectivity for body composition analysis: Applications in thigh, brain, and whole body tissue segmentation,”  IEEE Transactions on Biomedical Engineering, vol. 66, no. 4, pp. 1069–1081, 2018.
  * [10] Syed Muhammad Anwar, Ismail Irmakci, Drew A Torigian, Sachin Jambawalikar, Georgios Z Papadakis, Can Akgun, Jutta Ellermann, Mehmet Akcakaya, and Ulas Bagci,  “Semi-supervised deep learning for multi-tissue segmentation from multi-contrast mri,”  Journal of Signal Processing Systems, pp. 1–14, 2020.
  * [11] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [12] Luigi Ferrucci,  “The baltimore longitudinal study of aging (blsa): a 50-year-long journey and plans for the future,” 2008.
  * [13] Nicholas J Tustison, Brian B Avants, Philip A Cook, Yuanjie Zheng, Alexander Egan, Paul A Yushkevich, and James C Gee,  “N4itk: improved n3 bias correction,”  IEEE transactions on medical imaging, vol. 29, no. 6, pp. 1310–1320, 2010.


#  Interference of Holon Strings in 2D Hubbard Model

Chang-Yan Wang  Department of Physics, The Ohio State University, Columbus, OH
43210  Tin-Lun Ho  Department of Physics, The Ohio State University, Columbus,
OH 43210

###### Abstract

The 2D Hubbard model with large repulsion is a central and yet unsolved
problem in condensed matter physics for decades. The challenge appears below
half filling, where the system is a doped antiferromagnet. In this regime, the
fermion excitations are nothing like those in a Fermi liquid, which carry both
spin and charge. Rather, they split up into holons and spinons, carrying
charge and spin separately. Moreover, the motion of a holon is believed to
stir up the underlying antiferromagnetic order, leaving behind it a string of
“wrong” spins. While direct observation of the holon string is difficult in
electron systems, it has become possible in cold atom experiments due to
recent experimental advances. Here, we point out the key feature of the holon
strings, i.e. its Marshall phase, can be observed through measurements of spin
correlations. Moreover, the interference of these strings leads to an
anisotropic holon propagation clearly distinguishable than those of spinless
fermions, as well as a large suppression of the magnetic order in the region
swept through by the strings, as if the system is driven towards a spin
liquid. We further illustrate the effect of the Marshall phase by showing the
motion of a holon in the so-called $\sigma tJ$-model where the Marshall phase
is removed.

##  I Introduction

The Fermi Hubbard model is one of the most important model in condensed matter
physics. It has been intensely studied since its appearance[1]. The model has
a very simple form, consisting a term describing electron hopping on a lattice
and a term describing short range interaction between electrons. There is
considerable evidence showing that the model can exhibit a wide range of
phenomena in various parameter regime, such as metal-insulator transition,
antiferromagnetism, superconductivity, etc [2]. The 2D Hubbard model with
strong repulsion is of particular interest, as it is believed to capture the
key physics of high-$T_{c}$ superconductivity [3]. Yet despite decades of
studies, the model remains unsolved.

For simplicity, we shall consider the 2D Hubbard model on a square lattice.
For strong repulsion, the system at half filling has one electron per site,
with an antiferromagnetic (AF) ground state. The central question has been how
this ground state is changed as the system is doped below half filling. The
problem is challenging because the fermion excitations of a doped AF are
believed to split up into ”holons” and ”spinons”, which carry charge and spin
separately. They are very different from the excitations of a Fermi liquid,
which carry both spin and charge. The unusual properties of the spinons and
holons are believed to be the cause of many unusual properties of high $T_{c}$
superconductors including the “strange metal” behavior[3]. Yet despite their
central role in theoretical studies, there are no known ways to observe them
directly in current solid state experiments. However, an exciting possibility
has emerged in the last few years. With the rapid advances in cold atom
experiments, one can now simulate with great precision the Fermi Hubbard model
using ultra-cold fermions in optical lattices, and observe the AF order at
half filling[4, 5, 6, 7, 8]. Moreover, with the development of atom microscope
that can image atoms with single site resolution, one can now image the many-
body wavefunction of a quantum state with unprecedented detail.

In the case of a holon, it is supposed to leave behind it a string of “wrong”
spins as it moves through the AF background. Recently, Markus Greiner’s group
tried to identify the holon strings by comparing the images of a doped AF with
a classical AF background, and had concluded their presence by comparing the
observed images with a theoretical model [9]. Propagation of holon has also
been studied in [10, 11, 12]. More recently, one of us (TLH) has introduced a
method to identify the holon string directly from the spin density of the
system without making reference to the results of specific theories[13]. The
method relies only on an exact property of the Heisenberg AF – that its ground
state obeys the Marshall sign rule. In ref.[13], it is showed that if the
holon moves along the $x$ or $y$ axis of a square lattice, then its string has
a very clear signature – that the spin correlation of neighboring sites is
ferromagnetic in the immediate vicinity of the string, while being AF
everywhere else. This behavior follows from a key property of the holon string
– that it carries a phase reflecting the Marshall sign of the fluctuating AF
background. We shall refer to this phase as the “Marshall” phase[14, 13].

In this paper, we focus on the interference effect of holon strings and how
they are reflected in the spin correlation of nearest neighbors. The
interference effects occur when a holon travels away from the $x$ or $y$ axis.
In this case, there are many strings of the same length connecting the initial
and the final position of the holon. As we shall see, the different Marshall
phases of different strings will lead to a strong suppression of nearest
neighbor spin correlations, as if driving the system towards a spin-liquid. We
should mention that the ground state wavefunction of a holon has also been
studied for finite systems with periodic boundary condition [15] and for
ladders systems [16]. Here, rather than focusing on the ground state, we study
the propagation of holons in physical environments created in current
experiments. After this study on the interference of strings, we further
demonstrate the effect of the Marshall phase by comparing our results with
those of the so-called $\sigma tJ$-model [17], which is the Hubbard in strong
repulsion limit but with the Marshall phase stripped off. In this case, a
single holon in an AF background behaves more like a fermion in an empty
lattice, as if recovering the Fermi liquid behavior. In other words, the
Marhshall phase is responsible for the non-Fermi behavior of the holon.

##  II  $tJ$ model, Marshall sign, and amplitudes for holon propagation

We start with the Hubbard model,

$\displaystyle
H=-t\sum_{<i,j>,\sigma}c_{\sigma}^{\dagger}(i)c_{\sigma}(j)+U\sum_{i}n_{\uparrow}(i)n_{\downarrow}(i)$
(1)

The first term describes the hopping of a fermion $c_{\sigma}(j)$ with spin
$\sigma$ hopping from site $i$ to a neighboring sites $j$, $t>0$, $U$ is the
on-site interaction between opposite spins, and
$n_{\sigma}(i)=c_{\sigma}^{\dagger}(i)c_{\sigma}(i)$. For strong repulsion
($U>>t$), each site can at most be occupied by one fermion. The Hubbard model
then reduces to the $tJ$-model $H_{tJ}=\mathcal{T}+H_{J}$:

$\displaystyle
H_{J}=J\sum_{<i,j>}\mathbf{S}_{i}\cdot\mathbf{S}_{j},\,\,\,\,J=t^{2}/U>0,\hskip
86.72377pt$ (2)
$\displaystyle\mathcal{T}=-t\sum_{<i,j>,\sigma}\overline{c}_{\sigma}^{\dagger}(i)\overline{c}_{\sigma}(j),\
\overline{c}_{\sigma}(i)=c_{\sigma}(i)(1-n_{-\sigma}(i)),\hskip 21.68121pt$
(3)

where $\mathbf{S}_{i}=c_{\mu}^{\dagger}(i)\bm{\sigma}_{\mu\nu}c_{\nu}(i)/2$.
$H_{J}$ is the nearest neighbor AF Heisenberg Hamiltonian with spin
interaction $J$, and $\mathcal{T}$ is the hopping of fermions subject to the
constraint of no double occupancy. This constraint is the origin of the
intricate transport of the system. At half filling, $\mathcal{T}$ vanishes.
The $tJ$-model reduces to the AF Heisenberg model $H_{J}$. Experimentally, an
immobile hole at a selected site can be created by piercing through it with a
focused blued detuned laser. Note that $\mathcal{T}$ remains zero as long as
the hole is immobile.

We shall denote the spin states of the Heisenberg system with an immobile hole
at ${\bf R}$ as $|\bm{\nu};{\bf R}\rangle$, where
$\bm{\nu}\equiv(1\nu_{1},2\nu_{2},..)$ represents the spin configuration where
the fermion at site $i$ has spin $\nu_{i}$, $\nu_{i}=\uparrow,\downarrow$. Let
$|G;{\bf 0}\rangle$ be the ground state of this Heisenberg AF with an immobile
hole at the origin. It has the expansion

$|G,{\bf 0}\rangle=\sum_{\bm{\nu}}G(\bm{\nu},{\bf 0})|\bm{\nu},{\bf
0}\rangle.$ (4)

It is shown in ref.[13] that the ground state wavefunction $G(\bm{\nu},{\bf
0})$ obeys the Marshall sign rule as in the hole free case, i.e. it changes
sign when two opposite spins at nearest neighbor sites $\langle i,j\rangle$
are interchanged,

$\displaystyle G(i\uparrow,j\downarrow,...,{\bf 0})=-G_{{\bf
0}}(i\downarrow,j\uparrow,...,{\bf 0}),$ (5)

where $(\cdots)$ means the configuration of other spins. Because of this
property, it is convenient to use the new spin basis $\overline{|\bm{\nu},{\bf
0}}\rangle=(-1)^{N_{a}^{\downarrow}}|\bm{\nu},{\bf 0}\rangle$, where
$N_{a}^{\downarrow}(\bm{\nu})$ is the number of down-spins in configuration
$\bm{\nu}$ in one of the two sub-lattices of the square lattice (denoted as
$a$). (See Figure 1). We can expand any state $|\Phi,{\bf 0}\rangle$
(including the ground state $|G,{\bf 0}\rangle$) as $|\Phi,{\bf
0}\rangle=\sum_{\bm{\nu}}\overline{\Phi}(\bm{\nu},{\bf
0})\overline{|\bm{\nu},{\bf 0}\rangle}$, where $\overline{\Phi}(\bm{\nu},{\bf
0})=(-1)^{N_{a}^{\downarrow}}\Phi(\bm{\nu},{\bf 0})$. If $|\Phi,{\bf
0}\rangle$ satisfies the Marshall sign rule, Eq.(5), then we have
$\overline{\Phi}(\bm{\nu})\geq 0$.

To allow the hole to move, we remove the focused laser at time $\tau=0$. The
quantum state at later time is

$\displaystyle|\Psi(\tau)\rangle=e^{-i\tau H_{tJ}}|G;{\bf 0}\rangle=\sum_{\bf
R}\sum_{\bm{\mu}}\overline{\Psi}(\bm{\mu},{\bf
R};\tau)\overline{|\bm{\mu},{\bf R}\rangle},\hskip 21.68121pt$ (6)
$\displaystyle\overline{\Psi}(\bm{\mu},{\bf
R};\tau)=\sum_{\bm{\nu}}\overline{\langle\bm{\mu};{\bf
R}|}e^{-iH_{tJ}\tau}\overline{|\bm{\nu};{\bf
0}\rangle}\,\,\overline{G}(\bm{\nu},{\bf 0}).\hskip 28.90755pt$ (7)

Although $\overline{G}(\bm{\nu},{\bf 0})$ is positive,
$\overline{\Psi}(\bm{\mu},{\bf R};\tau)$ needs not to be because of the holon
propagator.

As discussed in ref.[13], the Marshall sign of a state $\Phi$ shows up in the
”exchange overlap” of opposite spins at neighboring sites $\langle
i,j\rangle$, $\rho_{ij}=\langle
c_{\downarrow}^{\dagger}(i)c_{\uparrow}^{\dagger}(j)c_{\downarrow}(j)c_{\uparrow}(i)\rangle_{\Phi}$,

$\rho_{ij}=\sum_{(...)}\Phi(i\downarrow;j\uparrow;...)^{\ast}\Phi(i\uparrow;j\downarrow;...)=\langle
S_{i}^{+}S_{j}^{-}\rangle_{\Phi}.$ (8)

The satisfaction of Marshall sign implies $\rho_{ij}<0$. For systems with
SU(2) symmetry (such as the $tJ$-model), Eq.(8) can be simplified as
$\rho_{ij}=2\langle S_{i}^{z}S_{j}^{z}\rangle_{\Phi}$. For systems without
$SU(2)$ symmetry, such as the $\sigma tJ$-model we discuss later, the exchange
correlation can be obtained using the interference method discussed in
ref.[13]. To measure the spin-spin correlation of the state when the hole has
moved to ${\bf R}$ after time $\tau$, one can first takes many images of the
spin density after the hole is released for a time $\tau$, and then post
select from these images the subset where the hole has arrived at ${\bf R}$.

Figure 1: (1a) shows one of the spin configurations $\bm{\nu}$ in the ground
state of an antiferromagnet with a hole fixed at site-0. When the hole is
released, it will travel to sites through various paths. Here, we show a
particular path for the holon to travel from site-0 to 9 through the sequence
of sites $0,1,2,..9$, which corresponds to ${\bf R}_{0}...{\bf R}_{9}$ in
Section (III.1), with ${\bf R}_{0}\equiv{\bf 0}$, ${\bf R}_{9}\equiv{\bf R}$.
The spin configuration of the initial state and final state are denoted as
$\bm{\nu}$ and $\bm{\mu}$. (1b) shows the spin configuration when the holon
arrives at ${\bf R}$ through the hopping ${\cal T}$ along the path indicated.
This path is one of the terms in Eq.(11),
$\prod_{i=1}^{9}\overline{\langle\bm{\mu}_{j},j|}(-{\cal
T}/t)\overline{|\bm{\mu}_{j-1},j-1\rangle}$. Here, we have
$\bm{\mu}_{0}=\bm{\nu}$, $\bm{\mu}_{9}=\bm{\mu}$. From this product, we sees
that $\mu_{\ell}=\nu_{\ell}$ for all sites $\ell$ not on the string. For the
sites $i$ on the string, we have $\mu_{i}=\nu_{i+1}$. The value of the product
is $(-1)^{N^{\downarrow}}$ where $N^{\downarrow}$ is the number of down spins
on the string [13, 14]. (1c) shows different strings connecting the initial
position ${\bf 0}$ and the final position ${\bf R}$ of the holon. Among the
three strings (green, black, and red) shown, the latter two have minimum
length. The rectangle $\Lambda$ (orange color) is the area swept through by
all the strings of minimum length. Each string has a Marshall phase
$(-1)^{N^{\downarrow}}$ defined in Eq.(10). The nearest neighbors (A) and (B)
have both sites outside and inside $\Lambda$ respectively. The The nearest
neighbor (C) has one site in $\Lambda$ and one site outside. (1d) shows that
the spin correlation $\langle S^{z}_{i}S^{z}_{j}\rangle$ of the nearest
neighbor $\langle i,j\rangle$ after the holon has travelled along a straight
line in x direction. All the correlations are antiferromagnetic (blue) except
those in the immediate vicinity of the string, which are ferromagnetic (red).

##  III Holon strings in the AF background

(III.1) Marshall phase effects on holon propagation:  Exact calculation of the
propagation amplitude $\Psi(\bm{\mu},{\bf R};\tau)$ is formidable. However,
the situation is simplified considerably when $J/t\ll 1$, where spins flips
are much slower than the motion of holes. In this limit, which is satisfied in
current experiments[6], one can expand Eq.(7) in powers of $J/t$. To the
lowest order, one can replace the $H_{tJ}$ by ${\cal T}$, and Eq.(7) becomes

$\overline{\Psi}(\bm{\mu},{\bf R};\tau)=\sum_{\bm{\nu}}\Gamma(\bm{\mu},{\bf
R};\bm{\nu},{\bf 0};\tau)\overline{G_{{\bf 0}}}(\bm{\nu})$ (9)
$\Gamma(\bm{\mu},{\bf R};\bm{\nu},{\bf
0};\tau)=\sum_{n=0,1,2..}\frac{(it\tau)^{n}}{n!}\Gamma^{(n)}(\bm{\mu},{\bf
R};\bm{\nu},{\bf 0};\tau)$ (10)

where $\Gamma^{(n)}$ is the transition amplitude of the hole reaching ${\bf
R}$ from ${\bf 0}$ through $n$ nearest neighbor hops,

$\Gamma^{(n)}(\bm{\mu},{\bf R};\bm{\nu},{\bf
0};\tau)=\overline{\langle\bm{\mu},{\bf R}|}\left(\frac{-{\cal
T}}{t}\right)^{n}\overline{|\bm{\nu},{\bf 0}\rangle}.$ (11)

The string connecting ${\bf 0}$ to ${\bf R}$ through $n$-hops will be referred
to as an $n$-string, and is denoted by the sequence of $n+1$ sites $({\bf
R}_{0},{\bf R}_{1},..,{\bf R}_{n})$, where ${\bf R}_{0}={\bf 0}$, ${\bf
R}_{n}={\bf R}$. See Figure 1a and 1b. Successive ${\bf R}$’s are nearest
neighbors. It is clear from Eq.(11) that $\Gamma^{(n)}$ is non-vanishing only
when the final and initial spin configurations $\bm{\mu}$ and $\bm{\nu}$ are
identical ($\bm{\mu}_{i}=\bm{\nu}_{i}$) on all the sites $i$ not on the
string. For the spins on the string, the final spin configuration is given by
the initial one sliding along the string by one lattice site, i.e.
$\mu_{{}_{{\bf R}_{i}}}=\nu_{{}_{{\bf R}_{i+1}}}$ for $i=0,1,2,..,n-1$. With
this relation between the initial and the final spin configurations, the value
of $\Gamma^{(n)}$ is [13, 14],

$\Gamma^{(n)}(\bm{\mu},{\bf R};\bm{\nu},{\bf 0};\tau)=\sum_{\rm
n-string}(-1)^{N^{\downarrow}_{\rm n-string}},$ (12)

where $N^{\downarrow}_{\rm n-string}$ is the number of down spins on the
$n$-string, and the factor $(-1)^{N^{\downarrow}_{\rm n-string}}$ (which is
$\pm 1$) is the Marshall phase of the string.

From Eq.(11), it is clear that $\Gamma^{(n)}$ is non-zero only when the length
of the string $n$ exceeds a minimum value $n^{\ast}\geq R_{x}+R_{y}$, which is
the minimum number of hops from ${\bf 0}$ to ${\bf R}$ through ${\cal T}$. See
Figure 1(c) and 1(d) The number of such strings is $L_{\bf
R}=(R_{x}+R_{y})!/(R_{x}!R_{y}!)$. The area swept through by these strings is
a rectangle ($\Lambda$) of size $R_{x}\times R_{y}$. If the amplitude $\Gamma$
is approximated by the leading term $\Gamma^{(n^{\ast})}$, we then have

$\overline{\Psi(\bm{\mu},{\bf
R};\tau)}\propto\sum_{\bm{\nu}}\Gamma^{(n^{\ast})}(\bm{\mu},{\bf
R};\bm{\nu},{\bf 0};\tau)\,\overline{G}(\bm{\nu},{\bf 0})$ (13)

From the constraints on spin configurations imposed by $\Gamma^{(n)}$ we have
just discussed, it is clear from Eq.(13) that the final state
$\overline{\Psi}$ and the initial state $\overline{G}$ have identical spin
configurations for all sites $i$ outside $\Lambda$, i.e. $\mu_{i}=\nu_{i}$.
Consequently, the exchange overlap $\rho_{ij}$ (or the spin correlation
$\langle S^{z}_{i}S^{z}_{j}\rangle$) of both states are the same for all
neighboring pairs $<i,j>$ outside $\Lambda$. (See Figure 1(c)). On the other
hand, if $i$ is outside $\Lambda$ and $j$ is inside, then the spin at site $j$
of the two states $\overline{\Psi}$ and $\overline{G}$ need not be the same.
This is because some strings in $\Lambda$ will pass through site $j$, each of
which will change the spin $\nu_{j}$ of the original AF configuration
($\bm{\nu}$) by the sliding motion along its respective path as discussed
before, while carrying its Marshall phase. For strings with opposite Marshall
phases, they interfere destructively and hence weaken the original AF order.

Similar weakening occurs with both $i$ and $j$ are inside $\Lambda$. However,
the case when ${\bf R}$ is on the $x$ or $y$ axis is an exception. In this
case, there is only one string of minimum length, which is the straight line
connecting ${\bf 0}$ and ${\bf R}$. If both neighboring sites $\langle
i,i+1\rangle$ are on this line, then the final spin configuration is simply
the spin configuration sliding backward along $x$ by one lattice site., i.e.
$(\mu_{i},\mu_{i+1})=(\nu_{i+1},\nu_{i+1})$. The AF correlation of neighboring
sites are therefore unchanged[13]. (See Figure 1(d)).

If longer strings are included in $\Gamma^{(n)}$, (i.e. with $n>n^{\ast}$),
the region $\Lambda$ swept through by the strings will expand beyond the
rectangle $R_{x}\times R_{y}$. Still, the region is of finite extent. The
behaviors of spin correlations of neighboring sites with different overlap
with $\Lambda$ remain unchanged. Moreover, since the number of strings grows
rapidly as $n$ increases, and their contributions to $\Gamma^{(n)}$ tend to
cancel each other. As a result, for time interval $\tau<1/t$, the magnitudes
of $\Gamma^{(n)}$ decreases rapidly with $n$. As we shall see, the leading
term $\Gamma^{(n^{\ast})}$ provides a good approximation to the full amplitude
$\Gamma$.

In the next section, we shall present numerical results to demonstrate all the
effects mentioned above. We shall show that when the holon propagates away
from the $x$-axis, the interference of holon strings strongly reduces the
magnitude of spin-spin correlation, as if driving the system towards a spin
liquid.

(III.2) Switching off the Marshall phase, the $\sigma tJ$-model: Another way
to demonstrate the Marshall phase is to study the motion of the holon with the
Marshall phase is “switched off”. This can be achieved by changing the kinetic
energy in the $tJ$-model to $\widetilde{\cal
T}=-t\sum_{<i,j>,\sigma}\sigma\overline{c}^{\dagger}_{\sigma}(i)\overline{c}_{\sigma}(j)$,
resulting in the so-called $\sigma tJ$-model [17] Although we do not yet have
a simple way to generate this hamiltonian in cold atom experiments, the model
is worth studying because it strips off the Marshall phases of the original
model. At half filling, $H_{\sigma tJ}$ again reduces to the Heisenberg AF
since $\widetilde{\cal T}$ also vanishes. The amplitude for the propagation of
a hole is still given by $\Gamma$ and $\Gamma^{(n)}$ in Eq.(10) and (11) with
${\cal T}$ replaced by $\widetilde{\cal T}$. It is easy to see that when a
hole travels from ${\bf 0}$ to ${\bf R}$ through a particular path (or
$n$-string), the hopping $\widetilde{\cal T}$ accumulates a phase
$(-1)^{N^{\downarrow}_{\rm n-string}}$ cancelling exactly the same factor in
Eq.(10) arising from AF background. Consequently, all strings amplitudes add
coherently, and the amplitude $\langle{\bf R}|(-T/t)^{n}|{\bf 0}\rangle$ is
simply the total number of strings connecting ${\bf 0}$ and ${\bf R}$. This
feature is identical to that of a spinless fermion in an empty square lattice,
although in that case there is no spin sums. Due to the difference in the
phase coherence of the strings, the behavior of holons in the $\sigma tJ$
model is very different from that in the $tJ$-model, which we shall show in
the next section.

Figure 2: The density distribution of the holon after released from the center
(site-12) for time $\tau=0.8/t$ on a $5\times 5$ lattice with open boundary
condition: Lattice sites are labelled as 1,2,3, etc.. They are represented by
a square. The number on site ${\bf R}$ is the holon density $n({\bf R},\tau)$.
(a) and (b) are the results for the $tJ$ and $\sigma tJ$ model respectively.
(c) is the result for the a spinless fermion on empty lattice. The quantum
state is calculated by keeping up to eighth order in $\tau t$.

##  IV Numerical results

(IV.1) Holon density and spin-spin correlation :  We have performed numerical
calculations for the density distribution and spin correlations of the holon
after it has traveled over a time interval $\tau$. We consider a $5\times 5$
square lattice. The lattice sites are labelled as 0 to 24 as shown in Fig.2.
The center is at site-12. Initially, the hole is held fix at the center. All
other sites are occupied by a fermion. We have computed the AF ground state
$G(\bm{\mu},{\bf 0})$ using exact diagonalization [18, 19]. This result allows
us to calculate the wavefunction $\Psi(\bm{\mu},{\bf R};\tau)$ as an expansion
of $\tau t$ as shown in Eq.(9) and (10); and hence the holon distribution at
time $\tau$, $n({\bf R},\tau)={\cal N}^{-1}\sum_{\bm{\mu}}|\Psi(\bm{\mu},{\bf
R};\tau)|^{2}$, where ${\cal N}$ is the normalization constant.

In Figure 2(a) and 2(b), we show the holon distribution $n({\bf R},\tau)$ for
the $tJ$-model and the $\sigma tJ$-model respectively, with the wavefunction
$\Psi(\bm{\mu},{\bf R};\tau)$ calculated upto 8-th order of $\tau t$. Figure
2(a) shows that the holon propagation in the $tJ$-model is very anisotropic,
being strongest in the $x$ or $y$ direction. This is due to the destructive
interference of the strings when the holon travel at an angle less then 90
degrees from the x (or y) axis as discussed in the previous section. In
contrast, Figure 2(b) shows that the holon propagation in the $\sigma
tJ$-model is much less anisotropic. In fact, it resembles more the density
distribution of a spinless fermion on an empty lattice, shown in Figure 2(c).
This is because neither $\sigma tJ$-model nor the spinless fermion models has
a Marshall phase in holon propagation, in contrast to the $tJ$-model.

Figure 3(a) to 3(c) show the density $n({\bf R},\tau)$ at different time $\tau
t$ when the holon is found is at the center ${\bf 0}$, at the nearest neighbor
and at the next nearest neighbor from the center. When $\tau t$ reaches 0.8,
the difference between the $tJ$\- and the $\sigma tJ$\- model is very apparent
when the final position ${\bf R}$ is the nearest neighbor and the next nearest
neighbor. See Figure 3(b) and 3(c). These differences should be experimentally
measurable. Note also that the holon density of the $\sigma tJ$-model is
closer to the spinless fermion than to the $tJ$-model. The return
probabilities to the origin of all three cases are quite similar, as shown in
Figure 3(a).

Figure 3: The densities $n({\bf R},\tau)$ in Figure 2 are plotted as a
function of $\tau$ for different position ${\bf R}$. (a) The holon returns to
the origin (site-12). (b) The holon arrives at the nearest neighbor (site-13).
(c) The holon arrives at the next nearest neighbor (site-18). The results for
holons in the $tJ$-model and the $\sigma tJ$-model are shown in green and red.
The result for a spinless fermion in an empty lattice is shown in blue. At
time $\tau=0.8/t$, the distinction between $tJ$-model and the $\sigma
tJ$-model is apparent. For the $tJ$-model, the holon propagates more
efficiently along $x$ (case (b)) than along the diagonal (case (c)), with
probability ratio $\sim 0.14/0.05$. In contrast, the corresponding ratio for
the $\sigma tJ$-model is $\sim 0.08/0.08$. As explained in the text, the
strong anisotropy of holon propagation in the $tJ$-model is due to the
destructive interference of the holon strings. It is also pointed out in the
text that the propagation of a holon in the $\sigma tJ$-model (red curve) is
similar to that of a single fermion in an empty lattice (blue curve) because
the propagators in both cases have similar phase coherence. In (a), the
$t\tau\in[0.4,0.8]$ part of the main plot is plotted in the inset using log
scale.

Figure 4: Distribution of the exchange overlap $\rho_{ij}$ (or spin
correlation) when the holon is at different locations ${\bf R}$ at time
$\tau=0.8/t$: In this $5\times 4$ lattice, the small squares (labelled 0,1,2,
etc) represent the lattice sites. The rectangle (or “bond”) between two
neighboring squares ($i,j$) and the number on it represent the exchange
overlap (or spin correlation ) $\rho_{ij}$. The quantum state after time
$\tau$ is calculated by expanding Eq.(9) up to seventh order in $\tau t$. (a)
$\rho_{ij}$ of the initial state with the holon at site-6 (${\bf R}={\bf 0}$):
Since the initial state is the AF ground state, it satisfies Marshall sign
rule. Hence all the bonds ($\rho_{ij}$) are negative. (b) $\rho_{ij}$ of the
final state with the holon at site-9: Our numerical results have confirmed the
picture discussed in Section I. All bonds not touching the string as well as
all bonds on the string are antiferromagnet (i.e. $\rho_{ij}<0$), while the
bonds at the immediate vicinity of the string is ferromagnetic (i.e.
$\rho_{ij}>0$). (c) Similar plot of (b) for the $\sigma tJ$-model: All the
bonds remain negative as all the strings of the same length are phase
coherent, as discussed in Section I.

Figure 5: (a) The distribution of spin correlation of the state at time
$\tau=0.8/t$ with the holon at site $n=13$, starting from the state at Fig.4.
The shortest string connecting the initial and final position (site-6 and
site-13) has length $n=3$. These $n=3$ strings sweep through a rectangle
$\Lambda$ covering sites 6, 7, 8, 11, 12, 13. For nearest neighbors $\langle
i,j\rangle$ that are inside $\Lambda$, such as (6,11), (7,12), as well as
those connected to it, such as (7,2), (12,17), their spin correlations are
strongly suppressed, due to the destructive interference of the holon strings.
(b) Comparison of spin correlations between the initial state at Fig.4, and
final state with final holon at site $n=13$ at time $t\tau=0.8$. The
horizontal axis labels of neighboring pairs $\langle i,j\rangle$ of interest.
The final state is calculated by expanding Eq.(9) up to 7th order in $\tau t$.
The result for the final state calculated with the only the shortest strings
($n=3$) in Eq.(9) is shown is green. Its closeness to the 7th order result
(red dots) shows the dominance of the $n=3$ terms over the $n=5$ and 7 terms.
To the right and the left of the vertical dashed line are the neighboring
pairs outside $\Lambda$ and those overlapping with it. For those to the right
(outside $\Lambda$) , the spin correlations of the final state is essentially
unchanged from the initial ones. For those to the left (overlapping with
$\Lambda$), the differences in spin correlation between the initial and final
state are significant. This is the result of destructive interference of the
strings that pass though one of the site (or both) in the neighboring pair
$\langle i,j\rangle$ as discussed in Section II. As discussed in Section III,
the spin correlation of the final state (the green dots) is made up of
“diagonal” and “off-diagonal” terms, $\langle S_{i}^{z}S_{j}^{z}\rangle={\cal
D}_{ij}+{\cal I}_{ij}$. Here, we have also plotted the direct term in purple.
One sees that ${\cal D}$ is close to the full value ${\cal D}+{\cal I}$ for
the nearest neighbors that overlap with $\Lambda$ (i.e. those to the right of
the vertical dashed line). It means the interference of holon string is close
to complete destruction, with ${\cal I}\sim 0$.

(IV.2) Marshall phase and spin correlations:  With the holon wavefunction
$\Psi(\bm{\mu},{\bf R};\tau)$ calculated in (IV.1), we can evaluate the spin
correlation $\langle S^{z}_{i}S^{z}_{j}\rangle$ for neighboring sites $\langle
i,j\rangle$. In Figure 4(a), we show this correlation of the initial AF ground
state $|G;{\bf 0}\rangle$ with the immobile hole at site-6 (which is the
origin ${\bf 0})$. The small squares represent the lattice sites. The number
on the rectangle linking two sites is the value spin correlation of these two
sites. Blue and red color correspond to negative and positive sign. For this
AF state, the correlations of all neighboring sites are negative, consistent
with the Marshall sign rule. The typical magnitude of the correlation is
around -0.12.

Figure 4(b) shows the spin correlation when the final position of the holon is
along $x$ (at site-9). In this case, the shortest path connecting the initial
and final position is a straight line. The numerical result confirms the
features discussed in the previous section, and in ref.[13], i.e. $\langle
S^{z}_{i}S^{z}_{j}\rangle$ is negative if $i$ and $j$ are both outside or on
the string; and is positive if on one of them ($i$ or $j$) is on the string.
The magnitude of the ferromagentic correlations (the red bonds) is weaker than
that of the background AF by a fact of 2. The exchange overlap of the $\sigma
tJ$-model (Figure 4(c)) is very different. It remains negative for all
neighboring sites. This is due to the phase coherence of all strings of the
same length as discussed before. The magnitude of the spin correlation in the
vicinity of the string, however, is weaker than that of the AF background by
roughly a factor of 4.

Figure 5 shows the spin correlation when the final position of the holon
(site-13) is not on the $x$-axis. In this case, the strings connecting site-6
to site-13 have lengths $n=3,5,7,..$. The pattern of spin correlation is very
different from that in Figure 4 when the final position is along x. We note
that the spin correlations of some neighboring sites (such as the “bonds”
between (17,12), (12,7), (7,2), (11, 6)) are reduced from the original AF
value by almost a factor of 20; as if the system is driven towards a spin
liquid state. This large reduction shows the strong destructive interference
effects of the holon strings.

In Figure 5, we show the different contributions that make up the spin-spin
correlation for the final state when the hole is at site-13 at time
$\tau=0.8/t$. The horizontal axis shows the nearest neighbor pair of interest.
The values of the spin correlations of the initial state and the final state
(when the hole is at site-6 and site-13) are given in blue and red color
respectively. These values are calculated from Eq.(10) by including up to
seventh order in $\tau t$. On the same figure, we also show (in green color)
the corresponding values for the final state including only the shortest
strings (i.e. $n=n^{\ast}=3$) contributions. The fact that the blue and green
dots almost overlap with each other shows that the $n=5$ and $n=7$-strings are
less important.

The $n=3$ strings connecting site-6 to site-13 sweep through a rectangle
$\Lambda$ of size $2\times 1$ (including sites - 6, 7, 8, 11, 12, 13). The
pairs of neighboring sites inside (and outside) of $\Lambda$ are collected on
the left (and the right) hand side of the vertical dashed line. For the pairs
outside $\Lambda$, there are little differences in the spin correlations
between the initial and the final states, (i.e. red and blue dots). In
contrast, the differences are significant for the neighboring sites inside
$\Lambda$, a feature discussed in the previous section.

To show quantitatively the different contributions to the string interference,
we write the quantum state in Eq.(9) as
$|\Psi^{(n)}\rangle=\sum_{\alpha}|\Psi^{(n)}_{\alpha}\rangle$, where
“$\alpha$” labels the strings. The spin correlation can be written as
$\langle\Psi^{(n)}|S_{i}^{z}S_{j}^{z}|\Psi^{(n)}\rangle=\cal{D}+\cal{I}$,
where ${\cal D}$ and ${\cal I}$ are the “diagonal” and the “off-diagonal”
terms, ${\cal
D}=\sum_{\alpha}\langle\Psi^{(n)}_{\alpha}|S^{z}_{i}S^{z}_{j}|\Psi^{(n)}_{\alpha}\rangle$,
and ${\cal
I}=\sum_{\alpha\neq\beta}\sum_{\alpha}\langle\Psi^{(n)}_{\alpha}|S^{z}_{i}S^{z}_{j}|\Psi^{(n)}_{\beta}\rangle$.
The off-diagonal terms represent the interference of the holon strings. In
Figure 5, the spin correlation ${\cal D}+{\cal I}$ and the diagonal terms are
plotted as green and purple dots respectively. We see that these values are
close to each other for the neighboring sites overlapping with $\Lambda$, i.e.
those pairs on left hand side of vertical dashed line. This means the
interference contribution ${\cal I}$ is small, precisely the destructive
interference physics we have discussed before when one or both of neighboring
sites ($i,j$) are inside $\Lambda$. On the other hand, we have shown
previously that the spin configurations of the final and the initial state are
identical outside $\Lambda$, independent of the strings inside it. This means
both diagonal and off-diagonal contributions are present, and their sum is
equal the original AF correlation, as seen on agreement between the green and
red data in Figure 5.

##  V Conclusion

We have studied the motion of the a single hole in the $tJ$-model in the limit
of slow spin motion, $t/J\ll 1$. In this limit, the propagation of a holon
generates strings equipped with a Marshall phase, which depends on the spin
configurations in the underlying antiferromagnetic state. The interference of
these strings leads to a holon propagation much more anisotropic than the that
of free fermions. It also reduces substantially the antiferromagnetic
correlation in the region swept through by the strings. We further demonstrate
the effect of the Marshall phase by considering the so-called $\sigma
tJ$-model. The spin dependent hopping of this model removes completely the
Marshall phase of the antiferromagnet, making the holon propagation similar to
that of a spinless fermion. Our method can be generalized to study multi-holon
transport, and include the effect of spinons, which we shall discuss
elsewhere.

Acknowledgement: We thank Professor Zheng-Yu Weng for helpful discussions, and
the kind support of the MacMaster Funds.

## References

  * Hubbard [1963] J. Hubbard, Electron correlations in narrow energy bands, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 276, 238 (1963).
  * Tasaki [1998] H. Tasaki, The Hubbard model - an introduction and selected rigorous results, J. Phys.: Condens. Matter 10, 4353 (1998).
  * Anderson [1997] P. W. Anderson,  _The Theory of Superconductivity in the High-Tc Cuprate Superconductors_ , first edition ed. (Princeton University Press, Princeton, N.J, 1997).
  * Mazurenko  _et al._ [2017] A. Mazurenko, C. S. Chiu, G. Ji, M. F. Parsons, M. Kanász-Nagy, R. Schmidt, F. Grusdt, E. Demler, D. Greif, and M. Greiner, A cold-atom Fermi–Hubbard antiferromagnet, Nature 545, 462 (2017).
  * Hilker  _et al._ [2017] T. A. Hilker, G. Salomon, F. Grusdt, A. Omran, M. Boll, E. Demler, I. Bloch, and C. Gross, Revealing hidden antiferromagnetic correlations in doped Hubbard chains via string correlators, Science 357, 484 (2017).
  * Greif  _et al._ [2013] D. Greif, T. Uehlinger, G. Jotzu, L. Tarruell, and T. Esslinger, Short-Range Quantum Magnetism of Ultracold Fermions in an Optical Lattice, Science 340, 1307 (2013).
  * Cheuk  _et al._ [2016] L. W. Cheuk, M. A. Nichols, K. R. Lawrence, M. Okan, H. Zhang, E. Khatami, N. Trivedi, T. Paiva, M. Rigol, and M. W. Zwierlein, Observation of spatial charge and spin correlations in the 2D Fermi-Hubbard model, Science 353, 1260 (2016).
  * Hart  _et al._ [2015] R. A. Hart, P. M. Duarte, T.-L. Yang, X. Liu, T. Paiva, E. Khatami, R. T. Scalettar, N. Trivedi, D. A. Huse, and R. G. Hulet, Observation of antiferromagnetic correlations in the Hubbard model with ultracold atoms, Nature 519, 211 (2015).
  * Chiu  _et al._ [2019] C. S. Chiu, G. Ji, A. Bohrdt, M. Xu, M. Knap, E. Demler, F. Grusdt, M. Greiner, and D. Greif, String patterns in the doped Hubbard model, Science 365, 251 (2019).
  * Bohrdt  _et al._ [2020] A. Bohrdt, F. Grusdt, and M. Knap, Dynamical formation of a magnetic polaron in a two-dimensional quantum antiferromagnet, New Journal of Physics 22, 123023 (2020).
  * Grusdt  _et al._ [2018] F. Grusdt, M. Kánasz-Nagy, A. Bohrdt, C. S. Chiu, G. Ji, M. Greiner, D. Greif, and E. Demler, Parton Theory of Magnetic Polarons: Mesonic Resonances and Signatures in Dynamics, Physical Review X 8, 011046 (2018).
  * Grusdt  _et al._ [2019] F. Grusdt, A. Bohrdt, and E. Demler, Microscopic spinon-chargon theory of magnetic polarons in the t-J model, Physical Review B 99, 224422 (2019).
  * Ho [2020] T.-L. Ho, Imaging the Holon string of the Hubbard model, PNAS 117, 26141 (2020).
  * Sheng  _et al._ [1996] D. N. Sheng, Y. C. Chen, and Z. Y. Weng, Phase String Effect in a Doped Antiferromagnet, Physical Review Letters 77, 5102 (1996).
  * Chen  _et al._ [2019] S. Chen, Q.-R. Wang, Y. Qi, D. N. Sheng, and Z.-Y. Weng, Single-hole wave function in two dimensions: A case study of the doped Mott insulator, Physical Review B 99, 205128 (2019).
  * Zhu  _et al._ [2016] Z. Zhu, Z.-Y. Weng, and T.-L. Ho, Spin and charge modulations in a single-hole-doped Hubbard ladder: Verification with optical lattice experiments, Physical Review A 93, 033614 (2016).
  * Zheng and Weng [2018] W. Zheng and Z.-Y. Weng, Charge-spin mutual entanglement: A case study by exact diagonalization of the one hole doped t-J loop, Scientific Reports 8, 3612 (2018).
  * Lin [1990] H. Q. Lin, Exact diagonalization of quantum-spin models, Phys. Rev. B 42, 6561 (1990).
  * Sandvik [2010] A. W. Sandvik, Computational Studies of Quantum Spin Systems, AIP Conference Proceedings 1297, 135 (2010).