# The pseudochiral Fermi surface of $\alpha$-RuI3

Alex Louat<EMAIL_ADDRESS>Diamond Light Source, Harwell Science and
Innovation Campus, Didcot OX11 0DE, UK  Matthew D. Watson  Diamond Light
Source, Harwell Science and Innovation Campus, Didcot OX11 0DE, UK  Timur K.
Kim  Diamond Light Source, Harwell Science and Innovation Campus, Didcot OX11
0DE, UK  Danrui Ni  Department of Chemistry, Princeton University
Princeton, NJ 08544, USA  Robert J. Cava  Department of Chemistry, Princeton
University
Princeton, NJ 08544, USA  Cephise Cacho<EMAIL_ADDRESS>Diamond
Light Source, Harwell Science and Innovation Campus, Didcot OX11 0DE, UK

###### Abstract

Abstract

In continuation of research into RuCl3 and RuBr3 as potential quantum spin
liquids, a phase with unique magnetic order characterised by long-range
quantum entanglement and fractionalised excitations, the compound RuI3 has
been recently synthesised. Here, we show RuI3 is a moderately correlated metal
with two bands crossing the Fermi level, implying the absence of any quantum
spin liquids phase. We find that the Fermi surface as measured or calculated
for a 2D ($k_{\text{x}},k_{\text{y}}$) slice at any $k_{\text{z}}$ lacks
mirror symmetry, i.e. is pseudochiral. We link this phenomenon to the ABC
stacking in the R$\bar{3}$ space group of $\alpha$-RuI3, which is achiral but
lacks any mirror or glide symmetries. We further provide a formal framework
for understanding when such a pseudochiral electronic structure may be
observed.

††preprint: APS/123-QED

Introduction

Quantum magnetism is an active research field for which materials with very
large degeneracy of the ground states are sought. Alongside some iridate
materials, $\alpha$-RuCl3 1, 2 and (more recently) RuBr3 3 have been
substantially investigated as a leading candidate for physical realisation of
the Kitaev model 4, 5, albeit that their zigzag magnetic order at low
temperatures 6 excludes an exact realisation of a quantum spin liquid ground
state. Very recently the “cousin” material $\alpha$-RuI3 has been synthesised
7, 8. These experimental reports immediately generated some controversy, even
over the basic classification of RuI3 being metallic or insulating, as the
interpretation of the resistivity measurements performed on compressed pellets
7, 8 is debated 9. It is important to understand the low-energy electronic
structure as it is directly related to the achievable magnetic states.

Figure 1: Image of the sample and associated ARPES spectrum at different
photon energy. a Scanning electron microscopy image of the $\alpha$-RuI3
single crystal after cleaving, showing laminations of the sample. The red
circle marks the area measured with the 4 $\mu$m beam spot. b Fermi surfaces
measured at selected photon energies corresponding to $k_{\text{z}}$ between 0
and 3$\pi$/c. The in-plane Brillouin zone boundaries at constant
$k_{\text{z}}$ are overlaid in red. c The 3D Brillouin Zone in red, and the
$\Gamma$MK$\Gamma$ and ALHA paths shown in green along which the experimental
dispersion has been extracted in Fig. 2a.

Motivated by the experimental ambiguity regarding its ground state, in this
paper we investigate the electronic structure of single crystals of honeycomb
RuI3 using micro-angle-resolved photoemission spectroscopy (micro-ARPES).
First and foremost, we find that RuI3 is, without question, a metal. However
we find some fingerprints of electronic correlations in the spectral function,
including a modest band renormalisation. Further, the low-symmetry R$\bar{3}$
space group has a profound impact on the measured electronic structure:
despite the crystal structure being formally centrosymmetric and thus achiral,
the theoretically calculated and experimentally measured bulk Fermi surfaces,
in the $k_{\text{x}}-k_{\text{y}}$ plane at any given $k_{\text{z}}$, appear
to be chiral. We describe this property as pseudochiral, in this case
associated with the lack of any in-plane mirrors, which in turn is a
consequence of the ABC stacking of the honeycomb lattice. Pseudochirality is a
concept more generally applicable to ARPES measurements, and we describe the
necessary conditions for observing a pseudochiral band structure in ARPES.

Results

3D electronic structure of $\alpha$-RuI3. The small size of the plate-like
single crystal samples, grown under pressure, necessitated a few variations to
the usual cleave-and-measure ARPES protocols. A fine tungsten wire was used
for cleaving the sample (see Methods). Capillary mirror micro-ARPES was used
to identify a uniform area with optimal spectral quality; later scanning
electron microscopy characterisation (Fig. 1a) confirmed this to be a flat and
homogenous region of the surface. The Fermi Surfaces were obtained by rotating
the electron analyser around the fixed sample position, maintaining a constant
beam spot on the sample.

In ARPES, the in-plane momentum ($k_{\text{x}}$ and $k_{\text{y}}$) and energy
of the initial state are conserved quantities and precisely measurable.
However, the out-of-plane momentum ($k_{\text{z}}$) is not conserved.
Nevertheless, within the free electron final state approximation, different
planes in $k_{\text{z}}$ can be probed by varying the energy of the incoming
photon 10, as shown in Fig. 1b. We find that 59 and 72 eV correspond to
$\Gamma$ and A planes respectively. At 60 eV, very close to a $\Gamma$ plane,
we observe a small central pocket connected to six spokes that disperse
towards K points, with approximately sixfold rotational symmetry. At 64 eV
only three spokes are well-defined, and the symmetry is only threefold. At 68
eV we observe the six-spoke structure in alternating second Brillouin zones,
best understood by considering the stacking of the bulk 3D Brillouin zones
(Fig. 1c). At 72 eV, we find the central pocket is the sharpest and brightest
feature. The observation of a Fermi surface in all cases is clear evidence
that RuI3 is metallic, while the strong variation with photon energy indicates
a complex multi-band 3D Fermi surface topology.

The origin of the three-dimensionality of the Fermi surface can be traced to
significant inter-layer hopping, despite the quasi-2D crystal structure. The
shortest interlayer bond distance $d_{I-I}=$3.9857 Å is larger than the
$d_{Cl-Cl}=$3.6649 Å in RuCl3, but on the other hand, the volume of the I
5p-orbitals are larger than the Cl 3p-orbitals. Thus, the ratio of the
interlayer bond distance to the Shannon ionic radius is smaller for RuI3
(1.81) than RuCl3 (2.02), implying significant overlap in RuI3. Therefore we
qualitatively expect a stronger interlayer hybridisation in $\alpha$-RuI3 \-
as is reflected by the $k_{\text{z}}$-dependence of the ARPES measurements.

Figure 2: Experimental band dispersion and comparison with DFT calculations. a
ARPES measurements along the $\Gamma$MK$\Gamma$ and ALHA paths presented on
Fig. 1c. b A zoomed band dispersion on the AL cut shows the presence of a
small electron pocket around A (red line) which is predicted by GGA-DFT
calculations (green lines). A more comprehensive comparison is available in
Supplementary Note A. c Comparison of the DFT-calculated Density of States and
the integrated ARPES spectrum (a summation of $\Gamma$M and $\Gamma$K
orientation at 72 and 83 eV with linear horizontal and vertical light
polarisation, 8 spectra in total).

Metallicity of $\alpha$-RuI3. Fig. 2a presents the experimental dispersion
along the paths $\Gamma$MK$\Gamma$ (h$\nu$=59 eV i.e. $k_{\text{z}}$=0) and
ALHA (h$\nu$=72 eV i.e. $k_{\text{z}}$=3$\pi$/c), as defined in Fig. 1c. There
are multiple Fermi crossings that derive from two bands as shown in Fig. 2b.
These two bands have a particularly narrow bandwidth, not dispersing below
-0.1 eV anywhere in the Brillouin zone. Experimentally, we find a dip in the
integrated spectral weight centered at -0.1 eV. A similar dip is found in the
Density of States as calculated by density-functional theory (DFT) at -0.17 eV
as shown in Fig. 2c, thus we ascribe the experimental dip as mainly a band
structure effect.

It is clearly apparent, however, that the fully occupied bands below the dip
are highly broadened compared with the Fermi-crossing bands. In fact, DFT
calculations predict numerous valence bands and continuous density of states
11 for several eV, but we find that below $\sim$-0.3 eV one could not resolve
any distinct dispersions within a very broad spectrum.

One contribution to the observed broadening of ARPES spectra is the disorder
induced by Ru on the vacancy site, and deficiency of Ru atoms on the metal
site, as reported in the sample characterisation 7. However the contribution
of disorder to spectral broadening is generally expected to be energy
independent, and while it may be limiting the sharpness of features at
$E_{\text{F}}$, it cannot explain the energy dependence of the broadening. The
$k_{\text{z}}$ broadening of the 3D-like band structure also plays a role, but
by itself cannot explain the absence of any resolvable features at all. The
energy scale of the observed broadening of ARPES spectra is also not
compatible with electron-phonon coupling.

Figure 3: Experimental Fermi surfaces compared with DFT and explanation of the
pseudochirality origin though the layer stacking and space group. a, c Fermi
surface measured at 59 and 72 eV, corresponding to $k_{\text{z}}$ = 0 and
3$\pi$/c, respectively. b, d Fermi surface calculated by DFT at
$\bar{k_{\text{z}}}$=0 and 3$\pi$/c, respectively. A Lorentzian energy
broadening (2$\gamma_{E}$=30 meV) and Lorentzian $k_{\text{z}}$ broadening
(2$\gamma_{k}$=1.2 nm-1) has been added to reproduce realistic experimental
resolution. Red and blue color indicate if the intensity comes mainly from a
band above or below $\bar{k_{\text{z}}}$. The dashed green line in a and b is
a guide to the eye, identical on both panels. e 3D representation of the Fermi
surface in the Brillouin Zone. In blue is the electron pocket and in grey are
the holes pockets. f Single layer of $\alpha$-RuI3. The pink and purple iodine
atoms represent the iodine above and below the Ru-plane, respectively. The
black line is the unit cell in the ab-plane. The nominally vacant sites have
2% Ru occupancy7. g Stacking along the c-axis of Ru hexagonal-lattice. The
almost white atoms represent the nominally vacant sites’ locations similar to
f. The planes along (1-10) direction are almost symmetry planes of each
individual layer, but due to the stacking, the crystal doesn’t have any global
mirror (or glide) symmetry. h Representation of the $\bar{3}$ Laue class’
equivalents points.

Instead, the energy dependency suggests that local electronic correlations
contribute significantly to the spectral broadening. The situation is
qualitatively similar to other correlated Ru-compound such as Sr2RuO4 12 or
Ca3Ru2O7 13; especially in the latter, one also observes a few sharp bands
close to $E_{\text{F}}$, giving way to a blur of continuous spectral weight
below $\sim$-0.1 eV. Thus the broadening gives the first fingerprint of local
electronic correlations.

Fig. 2b presents a zoom along the AL path, where the small electron pocket
centred around A is sharp enough to perform quantitative analysis. The
sharpness of this band, relative to the other branches near $E_{\text{F}}$,
comes from its minimal $k_{\text{z}}$ dispersion around
$k_{\text{z}}$=3$\pi$/c, as visible in Fig. 3e. The experimental Fermi
wavevector $k_{\text{F}}$ is 0.075 Å-1 while the one calculated by GGA-DFT is
0.120 Å-1, possibly due to a net hole doping from the Ru deficiency, or more
simply an inaccuracy of the DFT calculation. The measured Fermi velocity is
0.94 eV$\cdot$Å whereas the DFT velocity at the same k-vector is 1.42
eV$\cdot$Å which corresponds to a band renormalisation factor of $\eta$=1.52.
For comparison, Sr2RuO4 has a band renormalisation between 3 and 5 depending
on the band and position 12. Thus we conclude that RuI3 displays signatures
consistent with a moderate degree of electronic correlations.

The pseudochiral Fermi surface. The measured Fermi surface in the
$k_{\text{z}}$=0 plane shown in Fig. 3a is sixfold symmetric, but the spokes
emanating from the $\Gamma$ point towards K points are not precisely mirror
symmetric, and do not intersect the Brillouin zone boundary exactly at K. The
mirror symmetry breaking in the Fermi surface is accentuated by second
derivative and curvature analysis 14, available in Supplementary Note E. The
mirror symmetry-breaking cannot be attributed to the measurement geometry (see
methods), since the set-up preserves a mirror plane which maps $k_{\text{y}}$
to $-k_{\text{y}}$, which is nevertheless broken in the data.

To elucidate this observation, we simulate the ARPES spectrum using the DFT
calculation by integrating 2D ($k_{\text{x}},k_{\text{y}}$) slices over a
Lorentzian distribution in $k_{\text{z}}$ (FWHM=1.2 nm-1) and in energy
(FWHM=30 meV), accounting for the intrinsic $k_{\text{z}}$ broadening and the
experimental energy resolution respectively. The result of the convolved DFT
calculation is visible in Fig. 3b and d for $k_{\text{z}}$=0 and
$k_{\text{z}}$=3$\pi$/c, respectively. The color scale describes the low
(white) / high (black) intensity areas whereas the red/blue colors indicates
the $k_{\text{z}}$ origin of the contributions (red
$k_{\text{z}}>\bar{k_{\text{z}}}$, blue $k_{\text{z}}<\bar{k_{\text{z}}}$).
The main features of the measurements in both the $k_{\text{z}}$=0 and
3$\pi$/c planes are reproduced, including their symmetry as visible in Fig.
3(a-d). In the $\Gamma$ and A planes, the Fermi surface is sixfold symmetric,
but for a given $k_{\text{z}}$ plane only threefold rotation is expected, as
seen in the 3D Fermi surface in Fig. 3e. Importantly, both the simulations and
the experiment show rotational symmetry, but no mirror planes.

The absence of mirror planes can be understood from a real space perspective.
A monolayer of an idealised honeycomb lattice of RuI3 could host 3 mirror
planes, passing through the voids of the honeycombs (see Fig. 3f). However, if
we consider ABC stacking of the honeycomb lattice, where each layer is shifted
by ($\frac{-1}{3},\frac{1}{3},\frac{1}{3}$) with respect to the layer beneath,
the void positions do not align between layers, thus no global mirror planes
can exist, as illustrated in Fig. 3g. Therefore, the combination of the
honeycomb lattice (in its physical realisation with cations alternatively
above and below the plane) with ABC stacking excludes the existence of any
mirror planes. RuI3 thus forms in space group R$\bar{3}$ (No 148), where the
symmetry elements correspond only to the $\bar{3}$ rotoinversion operator (see
Fig. 3h). It is formally achiral as it is centrosymmetric, but has no mirror
or glide-mirror symmetries.

In 3D real space, a crystal is to be considered chiral if its space group
contains only proper operations15, i.e. does not contain inversion, mirror,
glide, or rotoinversion. The conditions to obtain true chirality in 3D k-space
are even more stringent since even if inversion symmetry is not present in the
crystal structure, it is restored in k-space by time-reversal symmetry. RuI3
has both inversion symmetry in real space and does not break time reversal.
Thus RuI3 is achiral in the crystallographic definition, and we also cannot
describe the electronic structure of RuI3 as chiral in a 3D sense.

Nevertheless, both the simulated and experimental Fermi surfaces projected in
the 2D $k_{\text{x}},k_{\text{y}}$ plane lack any mirror symmetry. This
motivates us to introduce the concept of pseudochirality: the case that a 2D
projection (relative to some given surface direction) of a bulk 3D band
structure does not contain any mirror symmetry.

More generally, we can ask the question: under what conditions can
pseudochirality be observed in a bulk-sensitive ARPES measurement? Formally,
the symmetry of electronic band dispersions in $k$-space is determined by the
Laue class of the material. The Laue classes are eleven geometric crystal
classes containing centrosymmetric crystallographic types of point groups and
their subgroups16, 17. Of the 11 Laue classes, $\bar{3}$ (the case for RuI3,
shown in Fig. 3h) and $\bar{1}$ have no mirrors at all, thus materials which
form in space groups with these Laue classes should generically show
pseudochiral electronic structure, independent of the cleavage plane.

For the cases $6/m$, $4/m$ and $2/m$ where the mirror plane is perpendicular
to the rotation axis, if we assume a cleave plane perpendicular to the
rotation axis, there would be no mirrors within a 2D projection in the
$k_{\text{x}}$ \- $k_{\text{y}}$ plane, and again the electronic structure
will be pseudochiral. In fact, pseudochirality can, theoretically, be observed
in any material; one simply has to choose a termination such that the surface
normal does not lie in any mirror planes of the Laue class. There is a close
relationship between the conditions for pseudochirality and surface
chirality18, 19, although they are not exactly equivalent, as discussed
further in Supplementary Note C.

Despite the many theoretical routes to pseudochirality as discussed further in
Supplementary Note D, RuI3 has a few characteristics which, practically
speaking, assist with the observation of pseudochiral electronic structure.
First, single crystals of good quality exist, can be cleaved, and are stable
in UHV. Second, it is metallic; having sharp quasiparticle bands at
$E_{\text{F}}$ will generally help with observing subtle deviations from the
symmetric case. Third, having large Fermi surfaces also helps, as the
deviations from mirror conditions become more apparent away from the $\Gamma$
point. In principle, the top/bottom faces of the same crystal of RuI3 would
present mirror-image spectra in 2D ARPES measurements (i.e. appearing
left/right handed), in otherwise equivalent conditions.

To our knowledge, there is only one previously reported system with
pseudochirality in measurements of bulk electronic structure, which is the
$\sqrt{13}\times{}\sqrt{13}R13.9^{\circ}$ phase of 1T-TaS2 and TaSe2 20, 21,
22. This is a special case where the mirror symmetry of the high-temperature
$P\bar{3}m1$ (No 164) phase is broken by the star-of-David superstructure; the
band structure is reconstructed and the ground state also has Laue class
$\bar{3}$, allowing pseudochiral electronic structure.

A conceptual difference between 1T-TaS2 and RuI3 is that in the former case,
there is a strong breaking of in-plane mirrors by the in-plane superstructure,
while in the case of RuI3, it is rather the out-of-plane stacking that breaks
the mirror symmetry. An alternative polytype of RuI3, grown under different
conditions, has been reported in space group P$\bar{3}$1c (No 163) 8,
differing principally in having an alternating two-layer stacking
configuration. Due to the additional $2/c$ glide-mirror symmetry element, the
Laue class is $\bar{3}$1m, which includes a mirror. Consistent with this, the
calculated Fermi surface for P$\bar{3}$1c phase is qualitatively similar to
the $R\bar{3}$ phase studied here, but not pseudochiral, having a mirror plane
parallel to $\Gamma-M$ (see Supplementary Note B). Thus in the context of
honeycomb lattice systems, it is the ABC stacking that specifically enables
pseudochiral electronic structure (for $\\{$001$\\}$ cleavage, which is the
only realistic possibility).

Discussion

The first outcome of this work is confirming the metallicity of $\alpha$-RuI3.
The multi-bands 3D Fermi surface revealed by ARPES in Fig. 1b clearly support
the metallic character of RuI3 reported by transport measurements 7, 8 on
compressed pellets or polycrystalline samples. Due to the 3D nature of the
Fermi surface, if single crystal resistivity measurements were performed, we
expect the resistivity anisotropy ratio between in-plane and out-of-plane to
be close to unity, unlike the three orders of magnitude reported in RuCl3 23.

While some theoretical work predicted $\alpha$-RuI3 to be possibly insulating,
other studies found that only an unrealistically large U repulsion could open
a Mott gap 11, instead expecting a metallic ground state. Our results support
the latter scenario, while also finding some spectral signatures of electronic
interactions through the energy-dependent broadening and band
renormalisations. Clearly, the on-site interactions, although suppressed
compared with the Mott insulating $\alpha$-RuCl3 due to the increased
hybridisation with the I orbitals, are not completely quenched in
$\alpha$-RuI3. It would be interesting to see if Dynamical mean-field theory
calculations can shed further light on the moderately correlated state we find
experimentally.

Although we have shown that there are many routes in theory, identifying
practical realisations of pseudochirality is challenging. We suggest four
avenues that could be fruitful: (1) other materials in space group 148,
$R\bar{3}$, which is fairly common in nature and where a layered structure
with good cleavage properties can be expected; (2) looking for materials with
Laue class $4/m$ and $6/m$ which might have (001) cleavage; (3) The
$\\{$111$\\}$ surfaces of cubic systems with Laue class $m\bar{3}$; (4)
Deliberate miscuts of elemental metals, such as (321) or (531), using soft
x-ray ARPES to access the bulk rather than the surface states.

The concept of pseudochirality is introduced here primarily for the
photoemission spectroscopy community, providing a framework for understanding
when bulk-sensitive ARPES measurements can appear chiral. However, other
physical responses and techniques are likely to be dependent on the
established concepts of real-space crystallographic chirality, or surface
chirality.

Conclusions

In conclusion, ARPES measurement on single crystal of $\alpha$-RuI3 reveals a
metallic 3D band structure and signatures of moderate electronic correlations.
The Fermi surface is close to the one calculated by DFT with a complex-shaped
electron and a hole-like pockets. As in the calculations, the experimental 2D
Fermi surface measured at any $k_{\text{z}}$ exhibits at least threefold
rotational symmetry, but never a mirror plane. We thus propose $\alpha$-RuI3
as an exemplar material for electronic pseudochirality.

Methods

High-quality $\alpha$-RuI3 crystals were synthesised by heating amorphous RuI3
powder at high pressure and subsequently purified by vapor transport
techniques in a quartz tube. The crystallographic phase was characterised by
single-crystal and powder X-ray diffraction at room temperature confirming the
three-layer centrosymmetric rhombohedral symmetry structure, with a honeycomb-
layer lattice. The structural refinement indicates a slightly Ru-deficient
composition Ru0.98I3 with 96% occupation of the Ru sites and 2% occupation of
the ideal vacant site7. To optimise the surface preparation procedure, a few
crystals were mounted on the same sample plate with a 50 $\mu$m W wire glued
on the top to cleave the surfaces in UHV. The typical size of the sample was
200 $\mu$m as visible on Fig. 1a. The ARPES measurements were performed at the
nano-ARPES branch of the Diamond I05 beamline 24 using a DA30 analyser with
vertical slit corresponding to $k_{\text{y}}$ in the experimental data. All
the constant energy cuts displayed have been measured with linear horizontal
light polarisation. A capillary mirror has been used to focus the photon beam
down to 4 $\mu$m. No sign of degradation was observed after 48 hours in 10-10
mbar pressure at 40 K temperature.

Density function theory (DFT) calculations have been performed using
generalized gradient approximations (GGA) functional with spin-orbit coupling
(SOC) but without Coulomb repulsion U. The calculations have been done with
WIEN2K 25 using the room temperature structure 7.

Data availability

The authors declare that the main data supporting the findings of this study
are available within the paper and its Supplementary Information. Extra data
are available from the corresponding authors upon request.

## References

  * Plumb  _et al._ 2014 K. Plumb, J. Clancy, L. Sandilands, V. V. Shankar, Y. Hu, K. Burch, H.-Y. Kee, and Y.-J. Kim, $\alpha$-RuCl3: A spin-orbit assisted Mott insulator on a honeycomb lattice, Physical Review B 90, 041112 (2014).
  * Koitzsch  _et al._ 2016 A. Koitzsch, C. Habenicht, E. Müller, M. Knupfer, B. Büchner, H. C. Kandpal, J. Van Den Brink, D. Nowak, A. Isaeva, and T. Doert, Jeff Description of the Honeycomb Mott Insulator $\alpha$-RuCl3, Physical Review Letters 117, 126403 (2016).
  * Imai  _et al._ 2022 Y. Imai, K. Nawa, Y. Shimizu, W. Yamada, H. Fujihara, T. Aoyama, R. Takahashi, D. Okuyama, T. Ohashi, M. Hagihala, _et al._ , Zigzag magnetic order in the Kitaev spin-liquid candidate material RuBr3 with a honeycomb lattice, Physical Review B 105, L041112 (2022).
  * Kitaev 2006 A. Kitaev, Anyons in an exactly solved model and beyond, Annals of Physics 321, 2 (2006).
  * Takagi  _et al._ 2019 H. Takagi, T. Takayama, G. Jackeli, G. Khaliullin, and S. E. Nagler, Concept and realization of Kitaev quantum spin liquids, Nature Reviews Physics 1, 264 (2019).
  * Johnson  _et al._ 2015 R. D. Johnson, S. C. Williams, A. A. Haghighirad, J. Singleton, V. Zapf, P. Manuel, I. I. Mazin, Y. Li, H. O. Jeschke, R. Valentí, and R. Coldea, Monoclinic crystal structure of $\alpha-{\mathrm{RuCl}}_{3}$ and the zigzag antiferromagnetic ground state, Phys. Rev. B 92, 235119 (2015).
  * Ni  _et al._ 2022 D. Ni, X. Gui, K. M. Powderly, and R. J. Cava, Honeycomb-Structure RuI3, A New Quantum Material Related to $\alpha$-RuCl3, Advanced Materials , 2106831 (2022).
  * Nawa  _et al._ 2021 K. Nawa, Y. Imai, Y. Yamaji, H. Fujihara, W. Yamada, R. Takahashi, T. Hiraoka, M. Hagihala, S. Torii, T. Aoyama, _et al._ , Strongly electron-correlated semimetal RuI3 with a layered honeycomb structure, Journal of the Physical Society of Japan 90, 123703 (2021).
  * Kaib  _et al._ 2022 D. A. Kaib, K. Riedl, A. Razpopov, Y. Li, S. Backes, I. I. Mazin, and R. Valentí, Electronic and magnetic properties of the RuX3 (X= Cl, Br, I) family: two siblings—and a cousin?, npj Quantum Materials 7, 75 (2022).
  * Damascelli 2004 A. Damascelli, Probing the electronic structure of complex systems by ARPES, Physica Scripta 2004, 61 (2004).
  * Zhang  _et al._ 2022 Y. Zhang, L.-F. Lin, A. Moreo, and E. Dagotto, Theoretical study of the crystal and electronic properties of $\alpha$-RuI3, Physical Review B 105, 085107 (2022).
  * Tamai  _et al._ 2019 A. Tamai, M. Zingl, E. Rozbicki, E. Cappelli, S. Riccò, A. De La Torre, S. M. Walker, F. Y. Bruno, P. D. King, W. Meevasana, _et al._ , High-resolution photoemission on Sr2RuO4 reveals correlation-enhanced effective spin-orbit coupling and dominantly local self-energies, Physical Review X 9, 021048 (2019).
  * Marković  _et al._ 2020 I. Marković, M. D. Watson, O. J. Clark, F. Mazzola, E. Abarca Morales, C. A. Hooley, H. Rosner, C. M. Polley, T. Balasubramanian, S. Mukherjee, _et al._ , Electronically driven spin-reorientation transition of the correlated polar metal Ca3Ru2O7, Proceedings of the National Academy of Sciences 117, 15524 (2020).
  * Zhang  _et al._ 2011 P. Zhang, P. Richard, T. Qian, Y.-M. Xu, X. Dai, and H. Ding, A precise method for visualizing dispersive features in image plots, Review of Scientific Instruments 82 (2011).
  * Fecher  _et al._ 2022 G. H. Fecher, J. Kübler, and C. Felser, Chirality in the Solid State: Chiral Crystal Structures in Chiral and Achiral Space Groups, Materials 15, 5812 (2022).
  * Shmueli  _et al._ 2016 U. Shmueli, H. Flack, and J. Spence, Methods of space-group determination, in  _International Tables for Crystallography Volume A: Space-group symmetry_, edited by M. Aroyo (pringer Dordrecht, 2016) pp. 107–131.
  * Vacher and Courtens 2003 R. Vacher and E. Courtens, Brillouin scattering, in  _International Tables for Crystallography Volume D: Physical properties of crystals_, edited by A. Authier (Springer Netherlands, 2003) pp. 329–335.
  * Pratt  _et al._ 2005 S. Pratt, S. Jenkins, and D. King, The symmetry and structure of crystalline surfaces, Surface science 585, L159 (2005).
  * Shukla and Gellman 2020 N. Shukla and A. J. Gellman, Chiral metal surfaces for enantioselective processes, Nature Materials 19, 939 (2020).
  * Yang  _et al._ 2022 H. Yang, K. He, J. Koo, S. Shen, S. Zhang, G. Liu, Y. Liu, C. Chen, A. Liang, K. Huang, _et al._ , Visualization of Chiral Electronic Structure and Anomalous Optical Response in a Material with Chiral Charge Density Waves, Physical Review Letters 129, 156401 (2022).
  * Bovet  _et al._ 2004 M. Bovet, D. Popović, F. Clerc, C. Koitzsch, U. Probst, E. Bucher, H. Berger, D. Naumović, and P. Aebi, Pseudogapped Fermi surfaces of 1T-TaS2 and 1T-TaSe2: A charge density wave effect, Physical Review B 69, 125117 (2004).
  * Ngankeu  _et al._ 2017 A. S. Ngankeu, S. K. Mahatha, K. Guilloy, M. Bianchi, C. E. Sanders, K. Hanff, K. Rossnagel, J. A. Miwa, C. B. Nielsen, M. Bremholm, _et al._ , Quasi-one-dimensional metallic band dispersion in the commensurate charge density wave of 1T-TaS2, Physical Review B 96, 195147 (2017).
  * Binotto  _et al._ 1971 L. Binotto, I. Pollini, and G. Spinolo, Optical and transport properties of the magnetic semiconductor $\alpha$-RuCl3, physica status solidi (b) 44, 245 (1971).
  * Hoesch  _et al._ 2017 M. Hoesch, T. Kim, P. Dudin, H. Wang, S. Scott, P. Harris, S. Patel, M. Matthews, D. Hawkins, S. Alcock, _et al._ , A facility for the analysis of the electronic structures of solids and their surfaces by synchrotron radiation photoelectron spectroscopy, Review of Scientific Instruments 88, 013106 (2017).
  * Blaha  _et al._ 2020 P. Blaha, K. Schwarz, F. Tran, R. Laskowski, G. K. Madsen, and L. D. Marks, WIEN2k: An APW+ lo program for calculating the properties of solids, The Journal of chemical physics 152, 074101 (2020).

###### Acknowledgements.

Acknowledgments Niels B. M. Schröter is warmly thanked for his constructive
comments as well as Anna Warren for the access to scanning electron microscopy
imaging. We acknowledge Diamond Light Source for time on Beamline i05-1 under
Proposal NT31246. We are grateful for the Computing resources provided by STFC
Scientific Computing Department’s SCARF cluster. The crystal synthesis was
founded by the Gordon and Betty Moore Foundation, EPiQS initiative, Grant No.
GBMF-9066.

Competing interests
The authors declare no competing interests.

Author Contributions
DN and RJC synthesised and characterised the samples. AL, MDW, TKK and CC
performed experimental measurements on the samples and contributed to the
analysis of the experimental data and manuscript writing.


# Integrated Digital Reconstruction of Welded Components: Supporting Improved
Fatigue Life Prediction
††thanks: The Innovation Fund Denmark supported this study, project: CeJacket
- Create Disruptive solutions for Design and Certification of Offshore Wind
Jacket foundations, project number 6154-00017B. Ørsted A/S supported the
Wenglor MLWL131 scanner used for this system.

1st Anders F. Mikkelstrup  Department of Materials and Production
Aalborg University
Aalborg, Denmark
Email<EMAIL_ADDRESS>2nd Morten Kristiansen  Department of Materials and
Production
Aalborg University
Aalborg, Denmark
Email<EMAIL_ADDRESS>
###### Abstract

In the design of offshore jacket foundations, fatigue life is crucial. Post-
weld treatment has been proposed to enhance the fatigue performance of welded
joints, where particularly high-frequency mechanical impact (HFMI) treatment
has been shown to improve fatigue performance significantly. Automated HFMI
treatment has improved quality assurance and can lead to cost-effective design
when combined with accurate fatigue life prediction. However, the finite
element method (FEM), commonly used for predicting fatigue life in complex or
multi-axial joints, relies on a basic CAD depiction of the weld, failing to
consider the actual weld geometry and defects. Including the actual weld
geometry in the FE model improves fatigue life prediction and possible crack
location prediction but requires a digital reconstruction of the weld. Current
digital reconstruction methods are time-consuming or require specialised
scanning equipment and potential component relocation. The proposed framework
instead uses an industrial manipulator combined with a line scanner to
integrate digital reconstruction as part of the automated HFMI treatment
setup. This approach applies standard image processing, simple filtering
techniques, and non-linear optimisation for aligning and merging overlapping
scans. A screened Poisson surface reconstruction finalises the 3D model to
create a meshed surface. The outcome is a generic, cost-effective, flexible,
and rapid method that enables generic digital reconstruction of welded parts,
aiding in component design, overall quality assurance, and documentation of
the HFMI treatment.

###### Index Terms:

3D scanning, point cloud registration, post-weld treatment, quality assurance,
FEM modelling

##  I Introduction

When designing jacket foundations for the offshore industry, fatigue life is
commonly a determining factor. Several methods for post-weld treatment have
been proposed, significantly improving the fatigue performance of welded
joints. One of which is high-frequency mechanical impact (HFMI) treatment,
which has been proven effective throughout the literature, outperforming
conventional methods such as burr-grinding [1]. HFMI treatment essentially
works by locally hammering the weld toe at a frequency exceeding 90 Hz. By
smoothing the transition between weld and base material, the stress
concentration is reduced, while the plastic deformation of the weld toe
introduces compressive residual stress; hence, counteracting the tensile
stresses that cause fatigue.

However, HFMI treatment must adhere to strict guidelines from the
International Institute of Welding (IIW) to yield the expected fatigue life
improvements. The IIW guidelines [2] include geometrical requirements for the
resulting treatment groove, such as width, depth and placement of the groove,
which can be difficult for a human operator to adhere to when operating a
heavy and vibrating tool for extended periods. Additionally, operator bias
cannot be omitted, which can further affect the resulting fatigue performance
of the treated weld [3].

As a result, certification agencies like DNV [4] do not recommend using HFMI
treatment at the design stage. To overcome this issue, [5] proposed a
methodology for performing automated HFMI treatment to improve quality
assurance. The proposed methodology used an industrial manipulator with input
from a 3D scanner to identify the areas requiring treatment. The study showed
a significant improvement in the treatment variance evaluated based on the
quantitative quality metrics proposed by IIW, i.e. groove depth, width and
placement. However, the proposed method for quality assurance only evaluates
the geometry and is thus unable to provide any fatigue life prediction. Hence,
accurate fatigue life prediction can support a more cost-effective design
through material reduction, as the expected fatigue improvement can be
documented.

In the case of complex or multi-axially loaded joints, the finite element
method (FEM) is commonly applied for fatigue life prediction. However,
typically FEM modelling is based on a simple and generic CAD representation of
the weld, which does not consider the actual weld geometry and weld defects,
which are well-known to influence the stress concentrations [6]. To improve
the FE model, the actual weld geometry can be included in the model, i.e. by
digitally reconstructing the welded geometry using 3D scanning. This method
has been shown to improve the fatigue life prediction and the prediction of
the possible crack location [6][7].

However, scanning the welded joints for digital reconstruction is often manual
and time-consuming or requires specialised scanning tools and possible
relocation of the part, making it unfit for use in a production setup [8].
Furthermore, the resulting point cloud should have the highest possible
resolution and accuracy, preferably below 0.1 $\mathrm{m}\mathrm{m}$, to
ensure that the areas of potential stress concentrations are captured in the
reconstructed geometry. Therefore, handheld techniques are unsuitable, as
these commonly offer a maximum achievable spatial resolution of up to 0.2
$\mathrm{m}\mathrm{m}$, even when using external targets [8].

Therefore, this work proposes a generic framework for cost-effective,
flexible, and rapid digital reconstruction of welded components for enhanced
FEM modelling. The overall aim is to support improved fatigue life prediction
for a more cost-effective design of welded components. The proposed framework
is developed to be applied as an integrated part of the setup in which
automated post-weld treatment is performed, such as the setup in [5]. The
suggested approach relies only on an industrial manipulator with a line
scanner to comprehensively scan the sample from multiple angles to produce a
complete surface representation. Prevalent image processing techniques and
simple filtering techniques combined with non-linear optimisation are applied
for aligning and merging the overlapping point clouds. This is followed by a
screened Poisson surface reconstruction to create the full watertight 3D
representation suitable for FEM modelling. Note that the proposed framework is
not limited to setups for post-weld treatment but could be used in a setup for
automated welding.

The paper is organised as follows: Section II presents the processing steps
and the main principles behind the framework. The experimental setup for
validating the framework is presented in Section III, while the results are
presented in Section IV. Lastly, the concluding remarks and suggestions for
future works are presented in Section V.

##  II Method

The proposed framework is based on small-scale S355 dog bone samples of welded
joints for the offshore industry, as shown in Fig. 1. However, the framework
can be applied to any component suitable and feasible to scan. A complete and
accurate representation of the entire sample is required to achieve the most
accurate fatigue life prediction, including any misalignment or distortion.
Although, a high resolution around the area of the weld is essential. Any
contours excessively smoothed in the reconstruction will significantly affect
the predicted fatigue life. Therefore, it is necessary to scan the sample from
a multitude of orientations and positions, resulting in multiple point clouds
that must be aligned, merged, and meshed to be suitable for use in FE
modelling.

Figure 1: The figure illustrates the transformation of a real-world welded dog
bone sample into an FE model, used to predict fatigue life and potential crack
location. The depicted targets aid in aligning the point clouds to form a
comprehensive surface representation of the sample. Note that the red areas of
the FEM model indicate the highest stresses and, consequently, the most
probable locations for cracks.

Estimating spatial transformations that align the point clouds are known as
point cloud registration. The point cloud alignment commonly consists of two
stages: a rough initial alignment and a secondary fine-tuning [9]. The rough
initial alignment is usually performed based on the scanner’s position during
acquisition, whereas the fine-tuning of the spatial transformation can roughly
be divided into targetless registration and target-based registration.

The most well-known algorithm for targetless point cloud registration is
perhaps the ICP (iterative closest point) proposed by [10]. However, as these
methods commonly rely on matching unique point pairs in the point clouds, they
are not well suited for texture-less and plain surfaces and suffer from a lack
of robustness.

Instead, the target-based approach uses artificial targets that have been well
distributed and placed in a unique pattern on the object, such that several
targets are visible from all scanning angles. Identifying and locating the
target pairs in the point clouds makes it possible to compute the spatial
transformation between them. In this work, the target-based approach has been
chosen, as it is beneficial when scanning texture-less objects that do not
have distinctive features to be used for aligning the point clouds, and
results in an accurate and robust registration [11][12]. Moreover, the targets
can be placed in areas that do not affect results from the FEM modelling.

Below, the framework for the digital target-based reconstruction of welded
components is presented in further detail.

###  II-A Selection of targets

The targets can be of various types, including spheres and high-contrast plane
paper targets. In this case, circular paper stickers of equal size are chosen
as target points. The placement of the targets is an important aspect, as
studies have shown that non-uniformly placed targets in a line at equal height
can result in inaccuracies in the computed spatial transformations [11]. The
chosen targets are, therefore, placed uniformly on the fixture in a semi-
random pattern with a density that ensures that a minimum of 3 targets are
present in two overlapping point clouds. The targets (circles) can be seen on
the sample in Fig. 1.

###  II-B Pre-processing of the point cloud

Initially, noisy data is removed from the scan to achieve a clean point cloud.
This is primarily done through a statistical approach, where the Euclidean
distance of each point is compared to a region of its neighbouring points. If
its distance exceeds the global mean and standard deviation, the point is
considered an outlier and removed [13].

###  II-C Alignment and merging

The initial course alignment is based on the defined robotic scanning
trajectories, while the subsequent fine adjustment is based on the approach
proposed by [14]. However, the proposed approach in this work makes use of
captured backscatter from the laser to create a 2D image grey-scale image of
the scanned surface, wherein the circular targets are identified and located
based on the Hough transform [15]. Note that a dictionary is created such that
a transformation back to 3D is possible. The principle is essentially to pair
matching targets between overlapping scans, i.e., by locating the same circles
in each overlapping scan, it is possible to compute the spatial transformation
between the scans.

The approach consists of two main steps:

  1. 1.

Filtering outlier circles: The first step is to filter out any target not
present in overlapping point clouds. The approach is to compute the Euclidean
distance between all targets in all possible combinations in each scan. Next,
the list of computed distances of each scan is compared to each other. If a
unique distance (within a specified tolerance) is found, and the associated
target does not have other matches, it is omitted from the list of potential
matches. The principle is illustrated in Fig. 2.

  2. 2.

Matching circles: The second step is to match the targets between the scans.
As the points clouds are rigid and an initial course alignment has been
performed, it is possible to assume a pure translation. Therefore, circles are
matched by ranking the Euclidean distances between the overlapping point
clouds. The distance with the most similarities is utilised as the reference
for selected matching target pairs. The principle is illustrated in Fig. 3. It
should be noted that the assumption of pure translation is only applied when
matching the targets and not when computing the transformation.

Figure 2: The principle behind filtering out targets. Point clouds 1 and point
cloud 2 depict overlapping areas. The blue markers indicate the centre
positions of the located targets. The green lines illustrate the matching
distances between the two overlapping point clouds, point cloud 1 and point
cloud 2. The red line and point illustrate a unique distance with an
associated point that only exists in point cloud 1; hence, the point is
omitted. Note that the example is based on a randomly selected reference
point, however, the operation is carried out for all points. Figure 3: The
principle behind matching the targets. Point clouds 1 and point cloud 2 depict
overlapping areas. Note that the green lines are parallel and with an equal
length for the matching points (1 and 1) and (2 and 2) in the two point
clouds. Due to the assumption of pure translation, this indicates that the
targets match. Instead, the crossing red lines indicate a mismatch, as such a
match would require a significant rotational transformation. The example
presents the use of points 1 and 2, although the operation is carried out for
all points that have been selected using the principle presented in Fig. 2.

With matched target pairs, it is possible to compute the spatial
transformation of the chain of overlapping point clouds with a point-to-point
approach based on [16]. As such, the transformation is determined through a
non-linear optimisation, which minimises the Euclidean distance between the
target pairs. After alignment, the point clouds must be merged to create a
homogeneous resolution. This is done by applying a box grid filter that merges
overlapping points. The entire process is performed automatically. The use of
prevalent and simple image processing operations removes the dependency on
expensive commercial software solutions.

###  II-D Removal of unwanted objects

A connectivity analysis segments the different objects in the point cloud to
remove unwanted objects. Since the targets have been placed on the sample, it
is possible to use this information to determine which segmented objects are
part of the sample. Any unwanted objects, such as parts of the fixture, can
then be removed from the scan. In other cases, the known size of the geometry
could also have been used as a feature to identify it.

###  II-E Surface reconstruction

The aligned and merged point cloud is automatically imported into Meshlab
using command-line integration. Meshlab is an open-source program for
processing and editing 3D meshes. In Meshlab, the normals of the point cloud
are computed using neighbouring points. This is followed by a screened Poisson
surface reconstruction, which meshes the final surface, creating a watertight
surface. Lastly, the reconstructed representation is exported as a .stl file,
which can be imported into the FEM software of choice, such as ANSYS.

TABLE I: Overview of the scanning parameters and the duration of scans. Note that the resolutions refer to the local coordinate system of the scanner, while the scan direction refers to the global coordinate system of the robot. The $x$-axis points along the length of the sample, while the $y$-axis points along the width. Scan direction | Res. $x$-axis | Res. $y$-axis | Res. Z | Velocity | Length of scan | Time per pass | Number of passes
---|---|---|---|---|---|---|---
Along length of sample ($x$-axis) |  $\sim$0.02 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ | 0.1 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ |  $\sim$0.004 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ | 15 $\tfrac{$\mathrm{m}\mathrm{m}$}{s}$ | 630 $\mathrm{m}\mathrm{m}$ | 42 $\mathrm{s}$ | 24
Along width of sample ($y$-axis) |  $\sim$0.02 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ | 0.1 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ |  $\sim$0.004 $\tfrac{$\mathrm{m}\mathrm{m}$}{px}$ | 15 $\tfrac{$\mathrm{m}\mathrm{m}$}{s}$ | 160 $\mathrm{m}\mathrm{m}$ | 11 $\mathrm{s}$ | 8
Total |  |  |  |  |  |  $\sim$18 min | 32

##  III Experimental setup

The experimental setup is similar to the setup presented in [5], designed to
perform automated post-weld treatment. The setup, illustrated in Fig. 4,
consists of two main components, a KUKA KR60-3 industrial manipulator and a
Wenglor MLWL131 line scanner. All computations are performed in MATLAB 2019b
and Meshlab v.2020.07 with an Intel Core i7 (i7- 9750H) CPU @ 2.6 GHz.

Figure 4: The experimental setup, consisting primarily of the industrial
manipulator and the line scanner.

As stated in Section II, the testing sample is a small-scale sample of an
offshore jacket structure which is welded using gas metal arc welding in a
double-sided joint configuration. The sample is made of S355 construction
steel and has been water jet cut into the shape of a dog bone with a size of
100 $\times$ 570 $\times$ 20 $$\mathrm{m}\mathrm{m}$^{3}$ for fatigue testing.

All robotic trajectories have been manually programmed and follow a linear
trajectory in which the velocity and orientation are kept constant throughout
the entire motion. Further details of scanning parameters can be found in
Table I.

Before scanning, the sample must be prepared. This includes removing any
surface contamination, such as paint and rust, around the weld area to avoid
any geometrical details being concealed. Additionally, the sample is sprayed
with scanning spray to reduce reflectivity (when scanning reflective samples),
and the reference targets (stickers) are manually and semi-randomly placed for
alignment. Upon scanning, the sample must be roughly positioned at a specific
position ($\pm$ 10 $\mathrm{m}\mathrm{m}$) to ensure it is correctly scanned
and merged.

##  IV Results and discussion

Due to the line scanner’s limited field of view (FoV), 32 scanning passes are
performed to ensure adequate overlay between individual scans, depicted in
Fig. 5. To cover the bottom half of the sample, it is manually rotated 180°
around a single axis. In total, the scanning duration of a sample is $\sim$30
min, where $\sim$18 min is pure scanning time. This is clear from Table I.
Note that with the applied scanner, the scanning speed and the resolution
between the scan lines ($y$-direction) are co-dependent. Hence, the scanning
speed can be increased at the expense of the resolution. Similarly, the FoV
can be expanded by moving further away from the sample, but it results in a
lower resolution. Lastly, the scanning time can be reduced by optimising the
overlay, which has not been done in this work.

Figure 5: Depiction of all 32 scans required to form a complete representation
of the scanned samples.

Having scanned the sample, the individual point clouds are then aligned,
merged and denoised based on the principles presented in Section II. This
resulted in a complete point cloud of $\sim$34,000,000 points, represented as
a $\sim$900 MB ASCII file with a resolution of 0.1
$\tfrac{$\mathrm{m}\mathrm{m}$}{px}$. The ASCII file is then imported into
Meshlab, where the normals are computed using 100 neighbouring points. This is
followed by the screened Poisson surface reconstruction, where a
reconstruction depth of 12 has been chosen to ensure a high level of detail.
The computational time is $\sim$15 min; however, the implementation is not
optimised. The result is a 504 MB .stl file consisting of $\sim$5,000,000
vertices and $\sim$10,000,000 facets. Reducing the construction depth could
reduce the file size, but also reduces the level of detail. The procedure is
illustrated in Fig. 6 along with the final reconstructed result.

Figure 6: The operations required to reconstruct the chosen sample. First, the
32 individual scans of the ends, top and bottom, are merged and aligned to
form the two halves, which are then merged, and the noise is removed. Lastly,
the sample is meshed, reconstructed, and saved as a .stl file for FEM
modelling. Note the detail included in the representation, such as the
reference targets and the course edges due to water jet cutting. Figure 7:
Observed errors in reconstructed samples. The illustration depicts a burr-
grinded sample, representing a case where the treated areas are highly
reflective.

When closely observing the reconstructed sample, slight misalignment of
overlapping scans can be noticed ($<$0.05 $\mathrm{m}\mathrm{m}$). This is due
to several factors, such as the scanning speed or orientation not being
constant, but primarily due to incorrect detection of the target centres.
Other errors have also been observed, such as floating points due to spurious
reflection or false representation of the surface from noisy points. Even a
pen ink on the surface affects the reconstruction, underlining the sensitivity
of the process. The problems primarily occur when scanning highly reflective
surfaces. This is illustrated in Fig. 7, which depicts the errors observed
when reconstructing a treated sample. Since the sample is scanned from all
directions, the lack of visible misalignment indicates that the framework
provides an accurate representation. However, to determine the precise
accuracy of the approach, the proposed framework must be applied to evaluate
the dimensions of a reference part with known dimensions.

The results demonstrate that the framework effectively reconstructs the welded
component with sufficient accuracy for fatigue life prediction through FEM
modelling.

##  V Conclusion

The proposed framework allows for the digital reconstruction of welded samples
directly in the production setup using an industrial manipulator and a line
scanner. The framework relies on prevalent image processing techniques and
non-linear optimisation to align and merge several point clouds, representing
the entire geometry of the sample. This is followed by a screened Poisson
surface reconstruction to form the final 3D representation. Focus has been
placed on developing a generic approach that does not rely on specific
hardware or expensive commercial software. The result is a flexible,
convenient, and cost-effective approach that can be applied for a range of
uses, including predicting the fatigue life improvements of the treatment,
aiding at the design stage of the component based on knowledge of the possible
crack location, and improved quality assurance and documentation of automated
HFMI treatment.

The future work includes some aspects, including optimisation of the scanning
procedure, e.g., by using robotic simulation software to reduce the number of
scans and targets and, thereby, the processing time. Similarly, a non-uniform
point cloud resolution could be applied, such that the resolution is increased
near the weld where a high level of detail is required and decreased away from
the weld. This would result in a reduced file size and reduced computational
time. Lastly, rotating the sample could be performed with a rotational axis
controlled by the robot to avoid manual interference.

## Acknowledgements

The authors would like to gratefully acknowledge Joachim Emil Kokholm Hersbøll
for providing insight into FEM modelling of welded components.

## References

  * [1] H. C. Yildirim, G. B. Marquis, and Z. Barsoum, “Fatigue assessment of high frequency mechanical impact (hfmi)-improved fillet welds by local approaches,” _International Journal of Fatigue_ , vol. 52, pp. 57–67, 2013\.
  * [2] G. B. Marquis and Z. Barsoum, _IIW Recommendations for the HFMI Treatment: For Improving the Fatigue Strength of Welded Joints_ , ser. IIW Collection. Singapore: Springer Singapore Pte. Limited, 2016.
  * [3] R. Aldén, Z. Barsoum, T. Vouristo, and M. Al-Emrani, “Robustness of the HFMI techniques and the effect of weld quality on the fatigue life improvement of welded joints,” _Welding in the World_ , vol. 64, no. 11, pp. 1947–1956, 2020.
  * [4] DNV-RP-C203:2019, “Fatigue design of offshore steel structures,” DNV, Høvik, NO, Recommended Practice, Sep. 2019, accessed: 21/03/2023.
  * [5] A. F. Mikkelstrup, M. Kristiansen, and E. Kristiansen, “Development of an automated system for adaptive post-weld treatment and quality inspection of linear welds,” _International Journal of Advanced Manufacturing Technology_ , vol. 119, no. 5-6, pp. 3675–3693, 2022.
  * [6] M. L. Larsen, V. Arora, M. Lützen, R. R. Pedersen, and E. Putnam, “Fatigue life estimation of the weld joint in K-node of the offshore jacket structure using stochastic finite element analysis,” _Marine Structures_ , vol. 78, no. October 2020, 2021.
  * [7] R. Späth and M. Steinebrunner, “Using 3D-scanners and finite element method to assess the fatigue life of welded joints based on the actual geometry of seam welds: realistic modelling of notches at the weld toe,” _Materialwissenschaft und Werkstofftechnik_ , vol. 54, no. 4, pp. 459–465, 2023.
  * [8] P. Rodríguez-Gonzálvez, M. Rodríguez-Martín, L. F. Ramos, and D. González-Aguilera, “3D reconstruction methods and quality assessment for visual inspection of welds,” _Automation in Construction_ , vol. 79, pp. 49–58, 2017.
  * [9] P. Minetola, “The importance of a correct alignment in contactless inspection of additive manufactured parts,” _International Journal of Precision Engineering and Manufacturing_ , vol. 13, no. 2, pp. 211–218, 2012.
  * [10] P. Besl and N. D. Mckay, “A method for registration of 3-d shapes,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 14, no. 2, pp. 239–256, 1992.
  * [11] B. Becerik-Gerber, F. Jazizadeh, G. Kavulya, and G. Calis, “Assessment of target types and layouts in 3d laser scanning for registration accuracy,” _Automation in Construction_ , vol. 20, no. 5, pp. 649–658, 2011.
  * [12] M. Franaszek, G. S. Cheok, and C. Witzgall, “Fast automatic registration of range images from 3d imaging systems using sphere targets,” _Automation in Construction_ , vol. 18, no. 3, pp. 265–274, 2009.
  * [13] R. B. Rusu, Z. C. Marton, N. Blodow, M. Dolha, and M. Beetz, “Towards 3D Point cloud based object maps for household environments,” _Robotics and Autonomous Systems_ , vol. 56, no. 11, pp. 927–941, 2008. [Online]. Available: http://dx.doi.org/10.1016/j.robot.2008.08.005
  * [14] D. Akca, “Full automatic registration of laser scanner point clouds,” ETH, Swiss Federal Institute of Technology Zurich, Institute of Geodesy and Photogrammetry, Zürich, Report, 2003.
  * [15] H. Yuen, J. Princen, J. Illingworth, and J. Kittler, “Comparative study of hough transform methods for circle finding,” _Image and Vision Computing_ , vol. 8, no. 1, pp. 71–77, 1990.
  * [16] K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-d point sets,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. PAMI-9, pp. 698–700, 1987.


11institutetext:  Department of Electrical and Computer Engineering, Boston
University, Boston, MA, USA
11email<EMAIL_ADDRESS>
22institutetext: Boston University Chobanian & Avedisian School of Medicine,
Boston, MA, USA
33institutetext: Department of Biomedical Informatics, University of
Pittsburgh, Pittsburgh, PA, USA

#  Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency
and Robustness in Mammography

Shantanu Ghosh${}^{(\textrm{{\char 0\relax}})}$ 11 Clare B. Poynton 22 Shyam
Visweswaran 33 Kayhan Batmanghelich 11

###### Abstract

The lack of large and diverse training data on Computer-Aided Diagnosis (CAD)
in breast cancer detection has been one of the concerns that impedes the
adoption of the system. Recently, pre-training with large-scale image text
datasets via Vision-Language models (VLM) (e.g., CLIP) partially addresses the
issue of robustness and data efficiency in computer vision (CV). This paper
proposes Mammo-CLIP, the first VLM pre-trained on a substantial amount of
screening mammogram-report pairs, addressing the challenges of dataset
diversity and size. Our experiments on two public datasets demonstrate strong
performance in classifying and localizing various mammographic attributes
crucial for breast cancer detection, showcasing data efficiency and robustness
similar to CLIP in CV. We also propose Mammo-FActOR, a novel feature
attribution method, to provide spatial interpretation of representation with
sentence-level granularity within mammography reports. Code is available
publicly: https://github.com/batmanlab/Mammo-CLIP.

###### Keywords:

Mammograms Vision Language Models CLIP.

##  1 Introduction

Breast cancer remains a leading cause of death among women, with a global
death toll of 670K in 2022. Creating AI tools for cancer in mammograms has
been a central focus for CAD [1, 18, 19]. Yet, training robust CAD models
requires a large and diverse dataset, which is expensive due to the high cost
of collecting large-scale annotations. VLMs (e.g., CLIP [17]) pre-trained on
large image-text pairs partially improve robustness in CV. This paper
introduces Mammo-CLIP, the first VLM trained with mammogram-report pairs.
Also, we propose Mammo-FActOR: Mammograms for Feature Attribution ConnecTing
Observations and Reports, novel feature attribution method, enhancing
interpretability by finding textual attribute-aligned visual representations.

VLMs align image-text representations in joint embedding space, providing
significant benefits. Pretraining on large datasets enhances generalizability
[3, 11] and data-efficiency [17, 8] for downstream tasks. The language encoder
reduces reliance on costly attribute annotations for developing interpretable
models [24, 16]. In medical imaging, radiology reports are a source of weak
labels, aiding disease localization [26, 14]. Generalizability,
interpretability, data efficiency, and weak supervision are crucial to
improving diagnostic precision and insightful analysis in medical imaging.

Application of VLM in medical imaging has been mostly limited in chest X-ray
(CXR) domain [27, 4, 23, 22, 25] due to availability of image-report dataset
e.g., MIMIC-CXR [6]. VLMs in other domains are mostly created with images and
captions from PubMed or public forums [7, 10, 5]. Although fine-tuning the
CLIP model on mammograms remains an option, fine-tinning reduces the image
resolution to be consistent with the input dimension of CLIP. Reducing the
resolution results in the loss of critical semantic visual cues essential for
accurate interpretation and diagnosis. Also, identifying mammographic
attributes requires the interpretation of high-resolution images, which CLIP
does not offer.

Contribution. Our contributions are two fold: (1) We present Mammo-CLIP, the
first VLM in mammography; (2) We introduce Mammo-FActOR, a novel feature
attribution method, to align visual representations with attributes in
mammography reports. Mammo-CLIP leverages a dataset of screening mammogram-
report pairs. This dataset is collected by the University of Pittsburgh
Medical Center (UPMC). Mammo-CLIP employs two key approaches for enhancing its
learning capability: 1) a data augmentation strategy to expand the training
data by incorporating external datasets containing images and mammographic
attributes (e.g., mass, calcification etc.) but lacking reports. We synthesize
reports based on these attributes. 2) multi-view supervision (MVS) [8], to
learn rich representations from limited data. Our experiments on various
breast cancer detection and localization tasks, conducted across two publicly
available datasets, exhibit data efficiency and robustness similar to those
observed in the CV community. Although Mammo-CLIP is trained on the screening
dataset, where the cancer is not explicitly mentioned, fine-tuning its
representation can bootstrap data-scare classification and abnormality
localization experiments in completely out-of-training distributions. The
final contribution, Mammo-FActOR, successfully localizes the attributes
without relying on the ground truth bounding boxes.

##  2 Method

Figure 1: Schematic view of our method. (a) Image-text augmentation for MVS.
(b) Dataset augmentation by synthesizing reports using image-attribute
datasets. (c) Mammo-CLIP pretraining strategy. (d) Feature attribution using
Mammo-FACtoR.

The goal is to align image ($\bm{x}^{I}$) and text ($\bm{x}^{T}$)
representations via image ($f^{I}(\cdot)$) and text encoders ($f^{T}(\cdot)$).
Our method consists of 3 components: 1) To enhance generalizability from
limited data, Mammo-CLIP is based on Multi-View Supervision (MVS) [8]. MVS
leverages cross-modal self-supervision across original and augmented image-
text pairs. 2) We use both instance and dataset augmentation strategies to
bootstrap the generalizability further. 3) Finally, we propose Mammo-FActOR, a
novel interpretable module to map the visual representation to the textual
attributes. We discuss the architectures of each component in detail in the
experiment section. Fig 1 illustrates our approach.

Notation. Let $(\bm{x}_{i}^{I},\bm{x}_{i}^{T})$ and
$(\tilde{\bm{x}}_{i}^{I},\tilde{\bm{x}}_{i}^{T})$ be the original and
augmented image-text pair of patient $i$ respectively. Let
$\mathcal{Z}^{I}=\\{\bm{z^{I}}_{i}\\}_{i=1}^{B}$ and
$\mathcal{Z}^{T}=\\{\bm{z^{T}}_{i}\\}_{i=1}^{B}$ be the normalized
representation of the image and text respectively of a batch of size $B$.
Similarly, $\tilde{\mathcal{Z}}^{I}$ and $\tilde{\mathcal{Z}}^{T}$ denote the
representation of the augmented image-text pairs. We obtain the normalized
representations by projecting the outputs from Mammo-CLIP’s image and text
encoders, $f^{I}(\cdot)$ and $f^{T}(\cdot)$, to the same dimensions, followed
by $\ell_{2}$ normalization.

###  2.1 Mammo-CLIP

Given a batch of representation pairs $\mathcal{Z},\tilde{\mathcal{Z}}$, the
following contrastive loss aligns the two representations by pulling paired
representations closer while pushing unpaired ones,

$\mathcal{L}(\mathcal{Z},\tilde{\mathcal{Z}})=-\sum_{\bm{z}\in\mathcal{Z}}\log\frac{\text{exp}(\langle\bm{z},\tilde{\bm{z}}\rangle/\tau)}{\sum_{\tilde{\bm{z}}\in\tilde{\mathcal{Z}}}\text{exp}(\langle\bm{z},\tilde{\bm{z}}\rangle/\tau)},$
(1)

where $\langle\cdot,\cdot\rangle$ and $\tau$ are the similarity function and a
learnable temperature to scale the logits, respectively. Mammo-CLIP utilizes
self-supervision across every pair of representations using the following MVS
loss,

$\mathcal{L}=\sum_{\mathcal{Z},\tilde{\mathcal{Z}}\in\\{\mathcal{Z}^{I},\tilde{\mathcal{Z}}^{I},\mathcal{Z}^{T},\tilde{\mathcal{Z}}^{T}\\},\mathcal{Z}\neq\tilde{\mathcal{Z}}}\mathcal{L}(\mathcal{Z},\tilde{\mathcal{Z}}).$
(2)

In our experiments, we down-weight
$\mathcal{L}(\mathcal{Z}^{T},\tilde{\mathcal{Z}}^{T})$, the loss for the
original and augmented text representation pair by half.

###  2.2 Instance and Dataset Augmentation

Instance augmentation. For $i^{th}$ patient, $\bm{x}^{I}_{i}$ and
$\tilde{\bm{x}}^{I}_{i}$ denote the CC and MLO views respectively. We also use
affine and elastic transformations to extend the augmented sets. Likewise,
$\bm{x}^{T}_{i}$ and $\tilde{\bm{x}}^{T}_{i}$ denote the impression and
findings sections of the mammography reports. We followed the technique
proposed in [25] that uses translation between two languages to generate
augmented text.

Dataset augmentation. In addition to the dataset containing paired mammograms
and reports, we leverage a separate dataset containing mammograms and
attributes (e.g., mass, calcification) to mitigate the challenge of the
limited dataset further. To synthesize reports from the attribute, we ask a
board-certified radiologist to construct a set of prompts using the values of
the attributes (positive, negative etc.), subtypes of the attribute (e.g.,
suspicious, obscured etc.), laterality of the breast (i.e., right and left),
depth of the breast (i.e., anterior, mid, posterior) and position of the
breast (e.g., upper, lower etc.). We synthesize the report by substituting the
attribute values for each patient to a randomly selected prompt (see Fig. 1 in
the Appendix).

Table 1: Classification performance on VinDr dataset. We report Area Under the
Curve (AUC) AUC for binary classification tasks to classify calcification and
mass. We report accuracy for multi-class classification tasks to predict
density.

Zero-shot (ZS) Linear Probe (LP) Finetune (FT) Label Model Pre-training Data
100% 10% 50% 100% 100% CLIP w/ RN-50 UPMC $0.37$ $0.42$ $0.51$ $0.55$ $0.68$
CLIP w/ EN-B5 UPMC $0.53$ $0.53$ $0.60$ $0.65$ $0.90$ Mammo-CLIP w/ EN-B2 UPMC
$\bm{0.68}$ $0.90$ $0.92$ $0.92$ $\bm{0.98}$ Mammo-CLIP w/ EN-B5 UPMC $0.61$
$0.91$ $0.93$ $0.94$ $\bm{0.98}$ Calcification Mammo-CLIP w/ EN-B5 UPMC, VinDr
$0.62$ $\bm{0.92}$ $\bm{0.94}$ $\bm{0.96}$ $\bm{0.98}$ CLIP w/ RN-50 UPMC
$0.20$ $0.34$ $0.42$ $0.49$ $0.57$ CLIP w/ EN-B5 UPMC $0.32$ $0.52$ $0.57$
$0.59$ $0.78$ Mammo-CLIP w/ EN-B2 UPMC $0.58$ $0.69$ $0.72$ $0.73$ $0.85$
Mammo-CLIP w/ EN-B5 UPMC $0.48$ $0.73$ $0.78$ $0.79$ $0.85$ Mass Mammo-CLIP w/
EN-B5 UPMC, VinDr $\bm{0.76}$ $\bm{0.80}$ $\bm{0.84}$ $\bm{0.86}$ $\bm{0.88}$
CLIP w/ RN-50 UPMC $0.04$ $0.55$ $0.61$ $0.69$ $0.72$ CLIP w/ EN-B5 UPMC
$0.10$ $0.76$ $0.76$ $0.78$ $0.83$ Mammo-CLIP w/ EN-B2 UPMC $0.13$ $0.80$
$0.82$ $0.84$ $0.85$ Mammo-CLIP w/ EN-B5 UPMC $0.15$ $\bm{0.83}$ $0.84$ $0.85$
$\bm{0.88}$ Density Mammo-CLIP w/ EN-B5 UPMC, VinDr $0.15$ $\bm{0.83}$
$\bm{0.86}$ $\bm{0.86}$ $\bm{0.88}$

###  2.3 Mammo-FActOR

Attributes mentioned in the radiology report can be viewed as weak labels. The
Mammon-CLIP learns to align image and text representation at the global level.
To obtain a spatial interpretation of the learned representation at the
granularity of the attribute, we develop Mammo-FActOR. The Mammo-FActOR learns
to align sentences in the radiology reports containing an attribute to the
channels of the frozen image encoder, $f^{I}(\cdot)$. Let’s assume we have $K$
different attributes and $f^{I}(\mathbf{x}_{i}^{I})\in\mathbb{R}^{C\times
H\times W}$ is the output of the frozen image encoder and
$\mathbf{t}_{i}^{k}\in\mathbb{R}^{d}$ is the representation of the sentence
containing the $k$’th attribute in the report of the patient $i$. We use
$\mathcal{X}_{-}^{k}$ to denote all mammography images not containing the
$k$’s attribute. For attribute $k$, the Mammo-FActOR learns a mapping
$\pi^{k}(\mathbf{t}_{i}^{k},\mathbf{x}_{i}^{I})=\left(\text{MLP}(f^{I}(\mathbf{x}_{i}^{I});\theta_{k})\mathbf{t}_{i}^{k}\right)\in\mathbb{R}^{C}$
representing the similarity of the channels with $\mathbf{t}_{i}^{k}$. The
$\text{MLP}(\cdot;\theta_{k})$ is a Multi-layer Perception parametrized by
$\theta_{k}$. To learn the parameters of the MLP, we minimize the contrastive
loss between images with and without attributes:

$\mathcal{L}^{\text{fac}}=-\sum_{i,c,k}{\log\frac{\exp\left(\left[\pi^{k}(\mathbf{t}_{i}^{k},\mathbf{x}_{i}^{I})\right]_{c}/\tau\right)}{\exp\left(\left[\pi^{k}(\mathbf{t}_{i}^{k},\mathbf{x}_{i}^{I})\right]_{c}/\tau\right)+\sum_{x\in\mathcal{X}_{-}^{k}}{{\exp\left(\left[\pi^{k}(\mathbf{t}_{i}^{k},\mathbf{x})\right]_{c}/\tau\right)}}}},$
(3)

where $\left[\cdot\right]_{c}$ denotes $c$’th element of the input vector
argument and $\tau$ is the temperature. After training, we weigh the
representations as, $\pi^{k}(\cdot)f^{I}(\cdot)\in\mathbb{R}^{H\times W}$ to
construct the textual aligned heatmap for attribute $k$.

##  3 Experiments

We aim to answer the following research questions: RQ1. Does Mammo-CLIP
enhance zero-shot (ZS) capabilities and labeling efficiency in classification
tasks? RQ2. Does Mammo-CLIP learn robust representations? RQ3. Does Mammo-CLIP
boost the performance of label-efficient localization? RQ4. Does Mammo-FActOR
enhance interpretability through attribute-aligned representation?

Table 2: Classification performance on RSNA dataset to classify malignancy.

Zero-shot (ZS) Finetune (FT) Linear Probe (LP) Model Pre-training Data 100%
10% 50% 100% 100% CLIP w/ RN-50 UPMC $0.28$ $0.62$ $0.67$ $0.72$ $0.58$ CLIP
w/ EN-B5 UPMC $0.42$ $0.80$ $0.84$ $0.86$ $0.70$ Mammo-CLIP w/ EN-B2 UPMC
$\underline{0.60}$ $\underline{0.82}$ $0.86$ $0.89$ $0.72$ Mammo-CLIP w/ EN-B5
UPMC $\bm{0.62}$ $\bm{0.85}$ $\underline{0.89}$ $\underline{0.90}$
$\underline{0.75}$ Mammo-CLIP w/ EN-B5 UPMC, VinDr $0.60$ $\bm{0.85}$
$\bm{0.90}$ $\bm{0.91}$ $\bm{0.79}$

###  3.1 Datasets

UPMC. The UPMC dataset includes 13,829 patient-report pairs, resulting in
25,355 screening mammograms (BI-RADS 0-2) with patients having at least one CC
or MLO view. We use 0.8/0.2 for the train/val split. VinDr. The VinDr-Mammo
[15] is a publicly available dataset comprising 5,000 exams (20,000 images)
from Vietnam, each with four views, breast level BI-RADS assessment category
(1-5), breast density category (A-D), annotating and location of mammographic
attributes (i.e., mass, calcifications, etc.). We use the train-test split in
[15]. RSNA. The RSNA-Mammo111https://www.kaggle.com/competitions/rsna-breast-
cancer-detection is a publicly available dataset having 11913 patients with
486 cancer cases. We use train/val/test split as 0.7/0.2/0.1.

###  3.2 Experimental details

Pre-processing the images. We use a rule-based approach to find the breast’s
ROI for all the datasets. We set values less than 40 to 0 and eliminate the
consistently identical rows and columns, as they denote the background. It
results in images with an average aspect ratio of 1:1.6 to 2, finally resized
to $1520\times 912$.

Pre-training. We develop 2 variant of Mammo-CLIP models: one pre-trained with
the UPMC dataset only and the other using both the UPMC and VinDr datasets.
Mammo-CLIP utilizes BioClinicalBERT [2] as the text encoder and
EfficientNet(EN)-B2 and B5 [20] with ImageNet pre-trained weights as the image
encoder. Image augmentations consist of 1) affine transformations with
rotations up to $20^{\circ}$, a minimum translation of 0.1%, scaling factors
[0.8, 1.2], and shearing by $20^{\circ}$; and 2) elastic transformations with
$(\alpha=10$, $\sigma=5)$. Text augmentations include sentence swaps and back-
translation from Italian to English222https://huggingface.co/Helsinki-NLP. We
train Mammo-CLIP for 10 epochs in a mixed-precision manner and optimize by
AdamW [13] with an initial learning rate 5e-5 and a weight decay 1e-4. We use
a cosine-annealing learning-rate scheduler [12] with a warm-up for 1 epoch.
Baseline. Using UPMC dataset and CLIP objective [17], we construct two
baselines: 1) an image encoder w/ ResNet (RN)-50 initialized with CLIP weights
and fine-tuned with $224\times 224$ images, 2) EfficientNet (EN)-B5 fine-tuned
using the same pre-processed images as Mammo-CLIP. Both the baselines are pre-
trained using the UPMC dataset, as CLIP only uses an image-text dataset, not
an image-attribute dataset. Mammo-FActOR. Our Mammo-Factor is the same light-
weight neural network [21] with linear
layer$\rightarrow$ReLU$\rightarrow$linear layer. We train it with a learning
rate 0.0001 and temperature ($\tau$) of 0.007 for 20 epochs.

Table 3: Localization performance (mAP) on VinDr dataset.

Finetune Freeze Encoder Label Model Pre-training Data 10% 50% 100% 100% CLIP
w/ RN-50 UPMC $0.03$ $0.11$ $0.17$ $0.08$ CLIP w/ EN-B5 UPMC $0.09$ $0.23$
$0.29$ $0.10$ Mammo-CLIP w/ EN-B2 UPMC $0.08$ $0.20$ $0.32$ $\bm{0.17}$ Mammo-
CLIP w/ EN-B5 UPMC $\bm{0.12}$ $\bm{0.28}$ $0.35$ $\bm{0.17}$ Calcification
Mammo-CLIP w/ EN-B5 UPMC, VinDr $0.10$ $0.25$ $0.35$ $\bm{0.17}$ CLIP w/ RN-50
UPMC $0.23$ $0.29$ $0.36$ $0.15$ CLIP w/ EN-B5 UPMC $0.34$ $0.45$ $0.49$
$0.28$ Mammo-CLIP w/ EN-B2 UPMC $0.38$ $0.50$ $0.55$ $0.37$ Mammo-CLIP w/
EN-B5 UPMC $0.41$ $0.52$ $\bm{0.58}$ $0.35$ Mass Mammo-CLIP w/ EN-B5 UPMC,
VinDr $\bm{0.43}$ $\bm{0.54}$ $\bm{0.58}$ $\bm{0.39}$

Figure 2: Mammo-FACtoR localizes mass and calcification w/o the ground-truth
bboxes from the VinDr dataset. For each pair, the left image denotes the
original image with the ground-truth bbox, while the right one is the bbox
predicted by Mammo-Factor.

Downstream evaluation. Evaluation of the downstream tasks utilizes both the
Mammo-CLIP’s image representations to 1) classify mass, calcification, and
density in the VinDr dataset and cancer in the RSNA dataset; 2) localize mass
and calcification in the VinDr dataset on a held-out test set. We evaluate the
classification tasks in 3 settings: zero-shot (ZS), linear probe (LP), and
fine-tuning (FT). In ZS and LP, we freeze both the encoders of Mammo-CLIP. For
FT, we jointly fine-tune the image encoder of Mammo-CLIP and the linear
classifier. For zero-shot (ZS) evaluation, we use prompts as {No $\langle
E\rangle$, $\langle E\rangle$}, where $E$ denotes mass, calcification, and
cancer. Refer to Tab. 1 in the Appendix for prompts for density. LP on VinDr
utilizes varying amounts of the training data (10%, 50%, or 100%). For FT on
VinDr, we use 100% data. As the screening mammogram reports do not mention
malignancy explicitly, data-efficient cancer classification for the RSNA
dataset involves fine-tuning with varying proportions of the training data
(10%, 50%, or 100%). We also report LP for the RSNA dataset using the entire
training set. The localization task includes fine-tuning the image encoder of
Mammo-CLIP using RetinaNet [9] ($\alpha=0.6$, $\gamma=2.0$) model and 10%,
50%, and 100% of training data from VinDR. We also report the localization
performance of RetinaNet by freezing Mammo-CLIP’s image encoder with 100% of
training data. Both setups in localization use only VinDr’s suspicious
findings. Metric. We report AUC scores for binary classification tasks to
predict mass, calcification, and cancer and accuracy for multi-class
classification tasks to predict density. For localization, we report mAP with
IoU=0.5 and detector confidence thresholded to 0.05 [9] unless specified.

##  4 Results

Classification (RQ1 & RQ2). Tab. 1 and 2 illustrate the classification results
for VinDr and RSNA, respectively. The Mammo-CLIP w/ EN-B5, pre-trained on the
UPMC and VinDr datasets, outperforms all the other models in most of the
evaluation settings. For VinDr, Mammo-CLIP’s LP efficacy with 10% of data
surpasses the FT performance of the baselines with 100% data. Also, Mammo-CLIP
excels in cancer classification on the RSNA dataset, showing its robustness,
even when cancer is not explicitly mentioned in the reports.

Table 4: Weakly supervised localization on VinDr using Mammo-FActOR.

Model Mass Calcification<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Mammo-CLIP w/
EN-B2 $0.38$ $0.22$ $0.2$ $0.13$ Mammo-CLIP w/ EN-B5 $0.45$ $0.37$ $0.25$
$0.17$

CLIP w/ RN-50 backbone fails to capture the visual cues from low-resolution
images. CLIP w/ EN-B5 addresses this by employing high-resolution images but
struggles to learn rich representation primarily due to its optimization
strategy focusing solely on the contrastive loss of the original image-text
pairs. High-resolution images and MVS optimization strategy allow Mammo-CLIP
w/ EN backbones to extract richer representations from limited data than
baselines, improving the classification performance. Recall that the UPMC
dataset consists of mammograms from a different population than the VinDr
dataset. The image encoder from Mammo-CLIP pre-trained with the UPMC dataset
shows excellent performance on both LP and FT settings on the VinDR dataset,
showcasing the robustness of the learned representations. ZS results on some
tasks (especially density) exhibit the need for more data.

Localization (RQ2 & RQ3). Tab. 3 shows localization results in a supervised
setting, utilizing the bounding boxes of VinDr. Mammo-CLIP w/EN-B5 pre-trained
with the UPMC and VinDr dataset outperforms other models in most settings.
Refer to the Classification subsection (described above) for insights into
such improvement. Also, the utilization of VinDr during the pre-training
improves the performance of both downstream tasks.

Evaluating Mammo-FActOR (RQ4). We evaluate the Mammo-FActOR using a weakly
supervised localization task to localize the mass and calcification of the
VinDr dataset without using the ground truth bounding boxes (bboxes). We
construct heatmaps using the Mammo-FActOR for both attributes. Following [26],
we extract isolated regions from the heatmap where pixel values are greater
than the 95% quantile of the heatmap’s pixel value distribution and generate
bboxes. We estimate mAP by comparing the IoU between generated and ground
truth bounding boxes. A generated box is true positive when
$\text{IoU}>T(\text{IoU})$, where $T(\cdot)$ is a threshold. Tab. 4
demonstrates the localization results. Mammo-CLIP w/ EN-B5 for $T(IoU)=0.25$
outperforms the other variants for both the attributes. Fig. 2 shows the
qualitative results. Mammo-FActOR aligns image representations with
attributes-specific textual descriptions from reports. Thus, it successfully
localizes the attributes without relying on ground-truth bounding boxes during
finetuning. Also, Fig. 2 in the Appendix shows the top 3 feature maps by
Mammo-FActOR for detecting these attributes.

##  5 Conclusion

This paper proposes a Mammo-CLIP and Mammo-FActOR to learn a cross-modal
embedding, improving generalizability, data efficiency, and interpretability.
Future directions include further leveraging vision transformers and cross-
attention mechanisms to enhance Mammo-CLIP and Mammo-FActOR.

## References

  * [1] Alberdi, E., Povyakalo, A., Strigini, L., Ayton, P., Hartswood, M., Procter, R., Slack, R.: Use of computer-aided detection (cad) tools in screening mammography: a multidisciplinary investigation. The British journal of radiology 78(suppl_1), S31–S40 (2005)
  * [2] Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.H., Jin, D., Naumann, T., McDermott, M.: Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 (2019)
  * [3] Eslami, S., de Melo, G., Meinel, C.: Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906 (2021)
  * [4] Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3942–3951 (2021)
  * [5] Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine 29(9), 2307–2316 (2023)
  * [6] Johnson, A., Lungren, M., Peng, Y., Lu, Z., Mark, R., Berkowitz, S., Horng, S.: Mimic-cxr-jpg-chest radiographs with structured labels. PhysioNet (2019)
  * [7] Kim, C., Gadgil, S.U., DeGrave, A.J., Cai, Z.R., Daneshjou, R., Lee, S.I.: Fostering transparent medical image ai via an image-text foundation model grounded in medical literature. medRxiv (2023)
  * [8] Li, Y., Liang, F., Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F., Yan, J.: Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=zq1iJkNk3uN
  * [9] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [10] Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., Xie, W.: Pmc-clip: Contrastive language-image pre-training using biomedical documents. arXiv preprint arXiv:2303.07240 (2023)
  * [11] Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., A Landman, B., Yuan, Y., Yuille, A., Tang, Y., Zhou, Z.: Clip-driven universal model for organ segmentation and tumor detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21152–21164 (2023)
  * [12] Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
  * [13] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [14] Müller, P., Meissen, F., Brandt, J., Kaissis, G., Rueckert, D.: Anatomy-driven pathology detection on chest x-rays. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 57–66. Springer (2023)
  * [15] Nguyen, H.T., Nguyen, H.Q., Pham, H.H., Lam, K., Le, L.T., Dao, M., Vu, V.: Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data 10(1), 277 (2023)
  * [16] Oikarinen, T., Das, S., Nguyen, L.M., Weng, T.W.: Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129 (2023)
  * [17] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [18] Srivastava, S., Sharma, N., Singh, S.K., Srivastava, R.: Design, analysis and classifier evaluation for a cad tool for breast cancer detection from digital mammograms. International journal of Biomedical Engineering and technology 13(3), 270–300 (2013)
  * [19] Srivastava, S., Sharma, N., Singh, S., Srivastava, R.: Quantitative analysis of a general framework of a cad tool for breast cancer detection from mammograms. Journal of Medical Imaging and Health Informatics 4(5), 654–674 (2014)
  * [20] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [21] Varma, M., Delbrouck, J.B., Hooper, S., Chaudhari, A., Langlotz, C.: Villa: Fine-grained vision-language representation learning from real-world data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22225–22235 (2023)
  * [22] Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163 (2022)
  * [23] Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Medklip: Medical knowledge enhanced language-image pre-training. Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [24] Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., Yatskar, M.: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19187–19197 (2023)
  * [25] You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E.K., Baek, W., Roh, B.: Cxr-clip: Toward large scale chest x-ray language-image pre-training. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 101–111. Springer (2023)
  * [26] Yu, K., Ghosh, S., Liu, Z., Deible, C., Batmanghelich, K.: Anatomy-guided weakly-supervised abnormality localization in chest x-rays. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 658–668. Springer (2022)
  * [27] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference. pp. 2–25. PMLR (2022)

## Supplementary materials

###  5.1 Zero-shot prompts for density

Table 1: Zero-shot prompts for classifying density Prompts
---
1.the breasts being almost entirely fatty,
2\. scattered areas of fibroglandular density,
3\. the breast tissue is heterogeneously dense,
4\. the breasts are extremely dense

###  5.2 Prompts to synthesize report-like sentences from attributes in image-
mammographic attribute dataset

Figure 1: Example of report-like sentence generation for the attribute mass
labeled positively in the VinDr dataset using the subtypes of mass and
position, laterality, and depth of the breast. We include all such prompts in
our codebase in details.

###  5.3 Visualization of activation maps identified by Mammo-FActOR

Figure 2: Visualization of activation maps identified by Mammo-FActOR to
localize mass and calcification attributes via report. For each row, the left
image is the original image. The middle three denote top-3 (left to right)
activation maps with the respective unit number. The right one indicates the
summation of the top-3 feature maps. We show the ground-truth bounding boxes
with the red rectangle. Notably, the optimal feature map identified by Mammo-
FActOR (Feature map unit 1305 for mass and 1150 for calcification)
successfully localizes the attributes.