# Demonstration of a Squeezed Light Source on Thin-Film Lithium Niobate with
Modal Phase Matching

Tummas Napoleon Arge  Center for Macroscopic Quantum States bigQ, Department
of Physics, Technical University of Denmark, Fysikvej 307, DK-2800 Kgs.
Lyngby, Denmark  Equal contribution  Seongmin Jo  CeNTech-Center for
Nanotechnology, Heisenbergstraße 11, 48149, Münster, Germany  SoN-Center for
Soft Nanoscience, Busso-Peus-Straße 10, 48149, Münster, Germany  Kirchoff-
Institute of Physics, Heidelberg University, Heidelberg, Germany  Equal
contribution  Huy Quang Nguyen  Center for Macroscopic Quantum States bigQ,
Department of Physics, Technical University of Denmark, Fysikvej 307, DK-2800
Kgs. Lyngby, Denmark  Francesco Lenzini  CeNTech-Center for Nanotechnology,
Heisenbergstraße 11, 48149, Münster, Germany  SoN-Center for Soft Nanoscience,
Busso-Peus-Straße 10, 48149, Münster, Germany<EMAIL_ADDRESS>Emma
Lomonte  CeNTech-Center for Nanotechnology, Heisenbergstraße 11, 48149,
Münster, Germany  SoN-Center for Soft Nanoscience, Busso-Peus-Straße 10,
48149, Münster, Germany  Jens Arnbak Holbøll Nielsen  Center for Macroscopic
Quantum States bigQ, Department of Physics, Technical University of Denmark,
Fysikvej 307, DK-2800 Kgs. Lyngby, Denmark  Renato R. Domeneguetti  Center for
Macroscopic Quantum States bigQ, Department of Physics, Technical University
of Denmark, Fysikvej 307, DK-2800 Kgs. Lyngby, Denmark  Jonas Schou Neergaard-
Nielsen  Center for Macroscopic Quantum States bigQ, Department of Physics,
Technical University of Denmark, Fysikvej 307, DK-2800 Kgs. Lyngby, Denmark
Wolfram Pernice  CeNTech-Center for Nanotechnology, Heisenbergstraße 11,
48149, Münster, Germany  SoN-Center for Soft Nanoscience, Busso-Peus-Straße
10, 48149, Münster, Germany  Kirchoff-Institute of Physics, Heidelberg
University, Heidelberg, Germany  Tobias Gehring  Center for Macroscopic
Quantum States bigQ, Department of Physics, Technical University of Denmark,
Fysikvej 307, DK-2800 Kgs. Lyngby, Denmark<EMAIL_ADDRESS>Ulrik
Lund Andersen  Center for Macroscopic Quantum States bigQ, Department of
Physics, Technical University of Denmark, Fysikvej 307, DK-2800 Kgs. Lyngby,
Denmark<EMAIL_ADDRESS>
###### Abstract

Squeezed states are essential for continuous variable (CV) quantum information
processing, with wide-ranging applications in computing, sensing and
communications. Integrated photonic circuits provide a scalable, convenient
platform for building large CV circuits. Thin-film Lithium Niobate (TFLN) is
particularly promising due to its low propagation loss, efficient parametric
down conversion, and fast electro-optical modulation. In this work, we
demonstrate a squeezed light source on an integrated TFLN platform, achieving
a measured shot noise reduction of 0.46 dB using modal phase matching and
grating couplers with an efficiency of up to -2.2 dB. The achieved squeezing
is comparable to what has been observed using more complex circuitry based on
periodic poling. The simpler design allows for compact, efficient and
reproducible sources of squeezed light.

††journal: opticajournal

##  1 Introduction

Squeezed states of light are a versatile and indispensable resource for
continuous-variable (CV) quantum optical information processing with
applications in quantum computing [1], quantum communication [2] and quantum
sensing [3]. Practical implementations of these protocols require a reliable
and scalable source of squeezed light.

Quantum optical systems using free space optics on optical benches occupy
significant space and suffer from the inherent instability of individual
components which hinders the development of large and complex systems.
Integrated photonics on the other hand offers scalability in the number of
components and inherent phase stability through accurate control of the
optical path length [4]. In addition, the high modal confinement of photonic
waveguides enables large optical nonlinearities and a strong electro-optic
response [5, 6].

Squeezed light sources have been developed on several integrated photonic
platforms. Using periodically poled titanium-indiffused lithium niobate
waveguides up to -3.4 dB of squeezing has been measured [7, 8]. However, the
large bending radii of the waveguides [9] limit circuit depth, making this
architecture unsuitable for practical applications of CV protocols. Ridge PPLN
waveguides bonded on a LiTaO3 substrate fabricated with a mechanical
structuring method, have enabled the measurement of squeezing levels up to -8
dB [10, 11]. However, the relatively weak confinement of these waveguides, as
well as the employed fabrication method, prevent the realization of highly
dense integrated photonic circuits.

Silicon nitride is another promising platform due to its low propagation loss,
high index contrast allowing for complex circuitry and CMOS compatibility
[12]. Using microring resonators, up to -1.65 dB of squeezed light has been
measured [13]. However, Silicon nitride is limited by the weaker $\chi^{(3)}$
non-linearity, slow modulation speed and parasitic non-linear effects which
hinder the performance at high powers needed to generate highly squeezed
states [12].

Thin film lithium niobate is an appealing platform for integrated CV quantum
photonics due to its intrinsic low propagation loss, large electro-optic
coefficient, which enables fast electro-optical modulators [6], and the high
$\chi^{(2)}$ nonlinearity, which allows for efficient generation of squeezed
states. Using periodic poling, ultra efficient wavelength conversion has been
achieved [14]. Squeezed states have been generated using both single-pass
configuration [15, 16, 17] and a cavity [18]. A reduction of the shot noise of
up to -0.5 dB has been measured, without correcting for or accounting for any
loss induced by the measurement equipment [17]. However, periodic poling
requires an advanced fabrication process that is not yet mastered by many
fabs. It demands a large surface area on the chip and is sensitive to
fabrication imperfections, which reduces reproducibility.

As an alternative, the geometric dispersion of optical transverse modes can be
engineered to achieve phase matching between far detuned modes [19]. This
approach has been utilised to demonstrate second harmonic generation [20] and
sum frequency generation [21].

In this work we demonstrate a compact squeezed light source on thin film
lithium niobate without using periodic poling. Instead, we employ modal
phasematching to generate 1550 nm squeezed light in the TE0 fundamental mode
while pumping from the higher ordered TM2 775 nm mode. We measured -0.46 dB of
squeezing collected into a fiber, close to the highest reported number with
periodic poling. This result is obtained using a compact, all passive device,
that does not require any complex circuitry, utilizing grating couplers with
an efficiency of up to -2.2 dB. Being limited by optical fiber coupling loss
and the photorefractive effect, we expect that future improvements could lead
to the observation of up to -3.6 dB squeezing.

##  2 Squeezed Light Source

Figure 1: Modal power profile of the a) TE0 1550 nm mode and b) TM2 775 nm
mode. c) Phasematching is achieved at a top width of around 1 µm. d) The TM2
775 nm mode is excited from the TM0 775 nm mode in the 0.1 µm wide input
waveguide. All simulations are made using Lumerical.

To achieve phase matching between the 775 nm pump light and the 1550 nm signal
light, we leverage the birefringence and geometric dispersion to counteract
the intrinsic dispersion between the two wavelengths. By using a Z-cut LN thin
film, where the extraordinary refractive index lies in the vertical plane, we
achieve an isotropic refractive index in the device plane. To minimize
propagation loss, we design for phasematching between the fundamental quasi-
transverse-electric mode (TE0) at 1550 nm, Fig. 1 a, and the high-order quasi-
transverse-magnetic mode (TM2) at 775 nm, Fig. 1 b, allowing for a greater
waveguide width. This design employs the weaker $d_{31}$ nonlinear term. Phase
matching is achieved between these modes at around a 1 µm waveguide width, as
shown in Fig. 1 c, where we plot the effective refractive indices for the 1550
nm TE0 and the 775 nm TM2 mode versus the waveguide width. Phase-matching
occurs when both modes have the same effective refractive index.

To enhance the non-linear efficiency, we employ a ring resonator cavity that
is resonant at both wavelengths. The radius of the cavity is set to 70 µm,
balancing low propagation loss and high finesse. To augment the coupling
strength at a wavelength of 775 nm, a pulley coupler is employed to convert
the mode from TM0 to TM2. This mode conversion is achieved by matching the
effective refractive indices between the waveguide and cavity mode [22].

The reproducible minimum waveguide thickness during the fabrication process
was 0.1 µm, and the cavity thickness needed to be 1.16 µm to achieve effective
refractive index matching with the TM2 mode, as shown in Fig. 1 d. Given that
the waveguide thickness for modal phase matching was 1.02 µm, the cavity was
designed with a thickness of 1.16 µm at the top for mode conversion and 1.02
µm at the bottom for phase matching. A linear waveguide taper of 25 µm was
used between these sections. The final device, shown in Fig. 2, has dimensions
of 580 µm x 520 µm (W x H).

Figure 2: Device used to generate squeezed light using modal phasematching

The fabrication process started with a $15\times 15\text{mm}^{2}$ diced chip
from a NANOLN wafer with a 500 nm thick Z-cut lithium niobate film bonded to
an 4.7 µm thick silicon dioxide insulating layer thermally grown on silicon.
After spin-coating with a negative-tone ArN 7520 resist, the chip underwent
electron beam lithography patterning. Subsequent development was performed
using an MF-319 solution, after which patterns were transferred onto the LN
thin films through a physical sputtering process using argon plasma in an
Oxford100 ICP-RIE etching tool with an etch depth of 0.40 µm. To address the
issue of material redeposition on the waveguide surfaces after etching, wet
etching (RCA-1 cleaning) was performed. Finally, to reduce propagation loss,
the sample was annealed at $400^{\circ}$C.

For light coupling onto the chip, negative angle grating couplers were used.
These grating couplers diffract incident light from an optical fiber into an
on-chip waveguide and vice versa. By optimising the structural periodicity of
the grating, we achieved phase-matching between the incoming light in the
fiber mode and the waveguide mode. The efficiency of the grating coupler was
highly wavelength-dependent.

Figure 3: a) Measured coupling efficiency for the signal. It exhibits an
efficiency of -2.2 dB at a wavelength of 1550 nm. b) Measured coupling
efficiency for the pump. At the optimal height, an efficiency of -3.67 dB was
measured at a wavelength of 775 nm. During sweeping of the wavelength,
instability in the laser power resulted in noise.

A measurement of the coupling efficiency is shown in Fig. 3. For the 1550 nm
light, a loss of 2.2 dB per facet was measured, while the pump light at 775 nm
exhibited a loss of 3.67 dB per facet. However, when using a fiber array to
simultaneously couple both wavelengths, the efficiency for the 775nm pump
light was measured to -5.8 dB at the maximum efficiency of the 1550 nm signal
light. This discrepancy is due to a difference in focusing height.

We designed the cavity to be double resonant. To maximise the squeezing, the
cavity was overcoupled for the signal light at 1550 nm, allowing as much
squeezed light to exit the cavity as possible. The Q-factor of this cavity was
$1.5\cdot 10^{5}$. To achieve maximal efficiency of the non-linear process, we
aimed for a critically coupled cavity for the 775 nm pump light, resulting in
a Q-factor of $7.1\cdot 10^{4}$.

Figure 4: a) Effect of power on the pump resonance. The photorefractive effect
transforms the resonance from a Lorentzian to a shark fin in addition to blue-
shifting the light. The data is taken at 0.5 nm/s. b) increasing the scan
speed decreases the photorefractive effect. At high scan speeds the Lorentzian
lineshape is recovered. The resonances are normalized and y-shifted.

Devices on a Lithium niobate integrated platform experience a strong
photorefractive effect [23], where charge migration due to optical
illumination causes an electro-optic effect[24, 25]. Due to the weak
interaction of the TM2 pump mode and the TE0 signal mode, a strong pump field
is required, which in turn induces a significant photorefractive effect, as
explored in Fig. 4. This effect results in a linear blue shift of the
resonance, as seen in Fig. 4a, with a slope of -17.4 nm/mW at a scan speed of
0.5 nm/s. Instead of a Lorentzian resonance shape, the photorefractive effect
produces a characteristic "shark fin" transmission spectrum, which becomes
more pronounced with increasing power. Up to 5 mW, this shark fin effect
becomes more distinct, but beyond 5 mW, the resonance becomes distorted and
wider.

The resonance stretching due to the photorefractive effect is inversely
proportional to the scan-speed of the laser wavelength [23]. In Fig. 4b, we
show the measured transmission spectrum of the pump when scanning the
wavelength around the resonance with a pump power of 0.9 mW on-chip. As seen
in the figure, the characteristic Lorentzian resonance shape is recovered at
high scan speeds, while the shark fin resonance is clearly visible for low
scan speeds, even at low powers, which is consistent with the findings of Shen
et. al. [23]. The effect on the 1550 nm signal mode is minimal and therefore
not shown in the figure.

##  3 Parametric Gain

Figure 5: a) Circuit for measuring parametric gain. b) Observed parametric
gain, the ripples are due to the pump light oscillating between being in and
out of phase with the seed light. The sharkfin pump envelope (blue) distorts
the gain. The sweep speed is 1 nm/s.

To characterize the squeezed light source, we first measured its parameteric
gain. The experimental setup for this measurement is shown in Fig. 5a. An NKT
Boostik E15 laser supplied 1550 nm light, part of which was up-converted to
775 nm light using an NKT Harmonik module. The chip was mounted on an aluminum
block with a Peltier element for temperature control. The light was monitored
with a Thorlabs PDA30B2 photodetector for the 1550 nm and a Thorlabs PDA36A2e
for the 775 nm path. By sweeping the laser frequency and laser power and
optimising the temperature of the chip, double resonance and phase matching
was achieved.

A typical measurement of the parametric gain is shown in Fig. 5b which was
obtained by amplifying a weak 1550 nm seed beam with a strong 775 nm pump
field. The gain was estimated by comparing the transmitted 1550 nm light with
and without the pump. When scanning the wavelength over a resonance, the pump
light interacts in and out of phase with the seed light, leading to the
ripples seen in the figure. The photorefractive effect, which causes the shark
fin envelope described above and shown in Fig. 4, distorts the parametric gain
envelope because it shifts the center resonance frequency for the pump light,
but not for the signal light. Consequently, the double resonance condition is
only fulfilled for specific optical powers.

With 10 mW of on-chip power, the centre of the 1550 nm resonance exhibited an
amplification factor of 3.15 and a deamplification factor of 0.5, taken as the
highest and lowest value of the ripples. This corresponds to a threshold power
of the OPO of around 50 mW of on-chip power.

##  4 Squeezing

Figure 6: a) Circuit for generating squeezed light. The photo diodes of the
homodyne detector have a quantum efficiency of $\eta=0.85$. b) Squeezing and
anti-squeezing at a sideband frequency of 5 MHz with an on-chip power of 6.9
mW. The resolution bandwidth is 1 MHz and the video bandwidth is 100 Hz. c)
Measured noise power relative to shot noise versus pump power (775 nm) at a
sideband frequency of 5 MHz. d) Measured noise power relative to shot noise
versus sideband frequency for an on-chip power of 6.9 mW at 775 nm.

We then used the experimental setup shown in Fig. 6a to measure the squeezing
generated by the thin film lithium niobate resonator. The 1550 nm light was
split in two parts: the majority was used to generate a strong 775 nm beam,
while the remainder was used as a local oscillator (LO) for homodyne
detection. The phase of the LO light was scanned at 0.5 Hz with respect to the
squeezed beam. The coupling efficiency was reduced from the -2.2 dB to -2.9 dB
compared to Fig. 3 due to damage to the devices.

Figure 6b shows a typical zero-span measurement of the noise power at 5 MHz
from the homodyne detector. With 6.9 mW of pump power on the chip, the
measured squeezing and anti-squeezing levels are -0.46 dB and 0.75 dB,
respectively. The shot noise level was determined by blocking the output from
the lithium niobate chip. From the squeezing and anti-squeezing measurement
results, we estimated the total efficiency of the setup to be approximately
$\eta\approx 0.24$.

Next, we tuned the 775 nm pump power, and the results of squeezing
measurements are shown in Fig. 6c. Due to the photorefractive effect on the
775 nm resonance, double resonance had to be individually achieved for each
data point. While the wavelength was fixed to achieve resonance for the 1550
nm light, the resonance for 775 nm was obtained by adjusting the temperature
of the circuit and finely tuning the pump power. This provided only a small
region of pump power where double resonance could be achieved; outside of the
region no squeezing was measured, explaining the sparse data points in the
figure. In addition, the stability of the coupling of pump light on chip was
crucial, as variations in the pump light shifted the pump resonance away from
the signal resonance due to the photorefractive effect. Each data point in the
figure represents an individual scanned LO measurement, where the value was
determined by taking the mean of 30 data points around the minimum (maximum)
of squeezing (anti-squeezing) of the signal.

At a lower on-chip pump power of 4.0 mW, compared to the previously discussed
6.9 mW, the squeezing and anti-squeezing levels decreased to -0.34 dB and 0.55
dB, respectively. We fitted a theoretical model to the measurement data. The
noise power of the squeezing, $S_{-}$, and anti-squeezing, $S_{+}$, are given
by
$S_{\pm}=1\pm\eta\frac{4\sqrt{\frac{P_{p}}{P_{p}^{th}}}}{\left(1\mp\sqrt{\frac{P_{p}}{P_{p}^{th}}}\right)^{2}+\left(\frac{f}{\bar{f}_{s}}\right)^{2}}$,
where the total efficiency is
$\eta=\eta_{QE}\mathcal{V}^{2}\eta_{opt}\eta_{esc}$, $P_{p}$ is the pump
power, $P_{p}^{th}$ is the oscillation threshold of the oscillator, $\omega$
is the side-band frequency, and $\bar{f}_{s}$ is the HWHM of the cavity
bandwidth of the signal in natural units. From the fit we obtained a total
efficiency $\eta=0.20$ and an on-chip pump threshold of $P_{th}\approx 200$
mW, higher than the 50 mW estimated from the parametric gain. This higher
number can be attributed to the difficulty achieving double resonance.
Clearly, the pump powers used in our experiment to measure squeezing are well
below the estimated threshold power.

Finally, we measured squeezing and anti-squeezing as a function of frequency
at a pump power of 6.9 mW. The results are shown in Fig. 6d. Since the LO
phase was not locked, we performed individual zero-span measurements for each
data point, similar to the measurements versus pump power. The relatively
large variation of the squeezing and anti-squeezing values can be attributed
to different coupling efficiencies during individual measurements. As
expected, a clear reduction of squeezing and anti-squeezing with increasing
sideband frequency is observed, due to the finite bandwidth of the cavity. In
comparison to the previous measurements at a 5 MHz sideband frequency, the
squeezing and anti-squeezing levels at 325 MHz decreased to -0.18 dB and 0.25
dB, respectively. The fit of the theoretical model to the measurement data is
shown in the figure with the following parameters: $P_{p}/P_{p}^{th}=0.02$,
total efficiency $\eta=0.23$, and cavity bandwidth $\bar{f}_{s}=310$ MHz. The
total efficiency $\eta$ is determined by the quantum efficiency of the
detectors $\eta_{QE}=0.85$, the visibility of the homodyne detector
$\mathcal{V}^{2}\approx 0.98$, and the optical efficiency $\eta_{opt}=0.45$.
Lastly, the escape efficiency is $\eta_{esc}\approx 0.55$

##  5 Discussion and outlook

In summary, we have developed a compact source of squeezed light by employing
modal phasematching on a thin film lithium niobate platform. By optimising the
geometry of the waveguide circuit, we enabled the TM2 pump mode to propagate
with the same effective index as the TE0 signal mode. This configuration
provided a gain coefficient of 3.16, allowing us to measure light squeezed
-0.46 dB below shot noise. This result matches the highest measured squeezing
generated in thin film lithium niobate [15, 18], achieved without the use of
periodic poling and by employing an all passive device.

The performance of our devices was limited by three factors: 1) coupling loss,
2) the photorefractive effect of the pump light, and 3) the difficulty of
simultaneously attaining resonance for both the pump and signal wavelengths.

Addressing the coupling loss, 1), inherent to thin film lithium niobate
circuits, is a significant engineering challenge. Grating couplers with a
negative angle, as used here, have the potential to reach coupling
efficiencies close to -0.1 dB if complemented with metal back-reflectors [26].
We estimate that the generated squeezing on-chip was -1.5 dB, significantly
higher than the measured off-chip value. The photorefractive effect, 2), is
intrinsic to lithium niobate. To mitigate this, the device’s operating
temperature can be increased [7], or MgO doping can be employed [27]. Another
option is to increase the length of the cavity, thereby decreasing the
resonance power buildup of the pump light, thereby decreasing the
photorefractive effect. For moderate increases in length, the extended phase-
matched region will compensate for the decrease in resonance, enhancing the
nonlinear efficiency. A decrease in photorefractive effect, 2), will give a
more gentle slope for the blue-shift of the resonance. Thus it can be used to
tune the device to double resonance, 3), making it possible to achieve
oscillation threshold power closer to the 50 mW estimated from the gain
measurement.

The limited tunability could also be addressed by adding a local heater or an
electro-optic modulator, mitigating the loss of tunability by increasing the
length of the resonator. We estimate that if we were able to pump with a power
close to the threshold value, -1.2 dB of squeezing could be measured off-chip
with the current grating couplers and -3.6 dB on chip.

Developing high quality squeezed light sources is a challenging task, but
worth the effort given the high prospects in terms of applications of on-chip
squeezed light sources. This work is a step towards this goal.

Funding The research was funded by the Danish National Research Foundation,
Center for Macroscopic Quantum States (bigQ, DNRF142), the Innovation Fund
Denmark (PhotoQ project, no. 1063-00046A), and from the EU projects CLUSTEC
(grant agreement no. 101080173) and EPIQUE (grant agreement no. 101135288).

Acknowledgments We thank Daniel A. Requena and Anders J. E. Bjerrum for
fruitful discussions.

Disclosures The authors declare no conflicts of interest.

Data Availability Statement All relevant data is included in the paper. The
data for the squeezing measurements can be found here:
https://figshare.com/s/cbf2154fa18e9201faab

## References

  * [1] C. Weedbrook, S. Pirandola, R. García-Patrón, _et al._ , Reviews of Modern Physics 84, 621–669 (2012).
  * [2] S. Pirandola, U. L. Andersen, L. Banchi, _et al._ , Advances in Optics and Photonics 12, 1012 (2020).
  * [3] S. Pirandola, B. R. Bardhan, T. Gehring, _et al._ , Nature Photonics 12, 724–733 (2018).
  * [4] A. Politi, J. C. Matthews, M. G. Thompson, and J. L. O’Brien, IEEE Journal of Selected Topics in Quantum Electronics 15, 1673 (2009).
  * [5] J. Wang, F. Sciarrino, A. Laing, and M. G. Thompson, Nature Photonics 14, 273–284 (2019).
  * [6] D. Zhu, L. Shao, M. Yu, _et al._ , Advances in Optics and Photonics 13, 242 (2021).
  * [7] R. Domeneguetti, M. Stefszky, H. Herrmann, _et al._ , Opt. Lett. 48, 2999 (2023).
  * [8] M. Stefszky, F. vom Bruch, M. Santandrea, _et al._ , Opt. Express 31, 34903 (2023).
  * [9] F. Lenzini, J. Janousek, O. Thearle, _et al._ , Science Advances 4 (2018).
  * [10] T. Kashiwazaki, T. Yamashima, N. Takanashi, _et al._ , Applied Physics Letters 119 (2021).
  * [11] T. Kashiwazaki, T. Yamashima, K. Enbutsu, _et al._ , Applied Physics Letters 122 (2023).
  * [12] A. Rahim, E. Ryckeboer, A. Z. Subramanian, _et al._ , Journal of Lightwave Technology 35, 639–649 (2017).
  * [13] Y. Zhang, M. Menotti, K. Tan, _et al._ , Nature Communications 12 (2021).
  * [14] C. Wang, C. Langrock, A. Marandi, _et al._ , Optica 5, 1438 (2018).
  * [15] H. S. Stokowski, T. P. McKenna, T. Park, _et al._ , Nature Communications 14 (2023).
  * [16] P.-K. Chen, I. Briggs, S. Hou, and L. Fan, Opt. Lett. 47, 1506 (2022).
  * [17] R. Nehra, R. Sekine, L. Ledezma, _et al._ , Science 377, 1333–1337 (2022).
  * [18] T. Park, H. Stokowski, V. Ansari, _et al._ , Science Advances 10, eadl1814 (2024).
  * [19] J.-Y. Chen, Y. M. Sua, H. Fan, and Y.-P. Huang, OSA Continuum 1, 229 (2018).
  * [20] L.-H. Hong, B.-Q. Chen, C.-Y. Hu, and Z.-Y. Li, Physical Review A 98 (2018).
  * [21] R. Luo, Y. He, H. Liang, _et al._ , Physical Review Applied 11 (2019).
  * [22] J. Lu, J. B. Surya, X. Liu, _et al._ , arXiv: Optics (2019).
  * [23] Y. Xu, M. Shen, J. Lu, _et al._ , Opt. Express 29, 5497 (2021).
  * [24] R. S. Weis and T. K. Gaylord, Applied Physics A Solids and Surfaces 37, 191–203 (1985).
  * [25] T. Hall, R. Jaura, L. Connors, and P. Foote, Progress in Quantum Electronics 10, 77 (1985).
  * [26] E. Lomonte, M. Stappers, L. Krämer, _et al._ , Scientific Reports 14 (2024).
  * [27] J. Fonseca-Campos, Y. Wang, W. Liang, _et al._ , “Comparison of photorefractive effects in undoped and mgo-doped ppln,” in _Photonics North 2007,_ (2007).


:
LEAVE UNSET 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H)
2023

# Enhancing Transformer-Based Segmentation for Breast Cancer Diagnosis using
Auto-Augmentation and Search Optimisation Techniques

Leon Hamnett<EMAIL_ADDRESS>
CARESAI1 Mary Adewunmi<EMAIL_ADDRESS>
University of Tasmania2, CARESAI1 Modinat Abayomi<EMAIL_ADDRESS>
Boston College3, CARESAI1 Kayode Raheem<EMAIL_ADDRESS>
COMSATS University Islamabad4, CARESAI1 Fahad Ahmed<EMAIL_ADDRESS>
Leibniz University5, CARESAI1

###### Abstract

Breast cancer remains a critical global health challenge, necessitating early
and accurate detection for effective treatment. This paper introduces a
methodology that combines automated image augmentation selection (RandAugment)
with search optimisation strategies (Tree-based Parzen Estimator) to identify
optimal values for the number of image augmentations and the magnitude of
their associated augmentation parameters, leading to enhanced segmentation
performance. We empirically validate our approach on breast cancer histology
slides, focusing on the segmentation of cancer cells. A comparative analysis
of state-of-the-art transformer-based segmentation models is conducted,
including SegFormer, PoolFormer, and MaskFormer models, to establish a
comprehensive baseline, before applying the augmentation methodology. Our
results show that the proposed methodology leads to segmentation models that
are more resilient to variations in histology slides whilst maintaining high
levels of segmentation performance, and show improved segmentation of the
tumour class when compared to previous research. Our best result after
applying the augmentations is a Dice Score of 84.08 and an IoU score of 72.54
when segmenting the tumour class. The primary contribution of this paper is
the development of a methodology that enhances segmentation performance while
ensuring model robustness to data variances. This has significant implications
for medical practitioners, enabling the development of more effective machine
learning models for clinical applications to identify breast cancer cells from
histology slides. Furthermore, the codebase accompanying this research will be
released upon publication. This will facilitate further research and
application development based on our methodology, thereby amplifying its
impact.

###### keywords:

breast cancer, histology slides, transformer, segmentation, auto-augmentation,
search optimisation

##  1 Introduction

Breast cancer is a significant global health issue, accounting for a
staggering one in eight of all cancer diagnoses worldwide, and the impact of
breast cancer will increase significantly by the year 2040 (cancer.net, ).
Breast cancer involves the growth of malignant cells forming a lesion within
the breast tissue. Early detection and treatment of these malignant cells has
been shown to have a significant impact on successful treatment of the cancer
and patient outcomes (Ott et al., 2009). Histopathological analysis (examining
cells and tissues under a microscope) (Roy and Gupta, 2020) is considered the
gold standard for detecting most cancers, as this reveals the microscopic
tissue structure, and can confirm the benign or malignant nature of a
suspected tumour (Kumar et al., 2017; Mills et al., 2012).

However, the manual evaluation of histopathological slides is time-consuming
and challenging due to the complex appearance of tissues and cells,
inconsistent staining and variation in illumination between different
laboratories, overlapped and clustered nuclei, and poorly fixed tissues (Aksac
et al., 2020). An accurate diagnosis of tumorous cells depends on the
pathologist’s experience and skill (Das et al., 2020), however, many
developing countries suffer from a shortage of trained pathologists
(Ziegenhorn and et al., 2020; Sayed et al., 2015) and this lack of skilled
pathologists presents an additional challenge for early detection and
successful treatment of breast cancer in these countries (Yan et al., 2020).
There has been growing interest in the development of Computer-aided
diagnostic systems (CAD) to increase the efficiency with which a pathologist
can examine histopathology slides. CAD systems for histopathological slides
typically involve image preprocessing, segmentation, feature extraction, and
classification of slide images (He et al., 2012; Demir and Yener, 2005). CAD
systems have begun to leverage machine learning and deep learning methods,
particularly neural networks, to enhance the accuracy and efficiency of slide
analysis (Madabhushi and Lee, 2016; Holzinger et al., 2020; Cohen, 2020).
Image segmentation is an essential component of many computer vision and
medical imaging systems, and this involves partitioning images into different
classes and predicting pixel-level classification by assigning each pixel to a
class category (Minaee et al., 2022a), e.g. tumorous cells and healthy cells.
Image segmentation models have recently shown remarkable performance
improvements and are increasingly used across many medical fields (Minaee et
al., 2022b; Moorthy and Gandhi, 2022). Applying these models to histopathology
images aids in identifying potential Regions of Interest (ROIs) (Rashmi et
al., 2021), highlighting possible tumorous cells for review by a trained
pathologist and speeding up pathologist efficiency. However, the time and
effort required to create a segmentation dataset from breast cancer histology
slides may have limited the adoption of image segmentation in the area of
breast cancer research.

Figure 1: Examples of patches taken from breast cancer histology slides

We see from existing research (Krithiga and Geetha, 2020), that image
segmentation has successfully been applied to a number of other areas of
cancer research, including lymphoma segmentation from histological images
(Tosta et al., 2017), nucleus detection in breast cancer histology slides
(Lakshmanan and Saravanakumar, 2018), renal cell cancer segmentation (Albayrak
and Bilgin, 2018), and colon cancer cell segmentation (Jia et al., 2017). A
recent study leveraged a crowd-sourced dataset to train a VGG16-FCN8
segmentation model for tumor segmentation in breast histology slides,
achieving a significant Dice Score of 85.1 for the cancer class (Amgad et al.,
2019). Recently, across many computer vision applications, transformer-based
models have shown increased research interest (pap, ) and seem to show better
performance than traditional CNNs for a wide variety of computer vision tasks
(Han et al., 2023). An extremely limited amount of previous research has been
undertaken into the task of using transformer-based models to segment breast
cancer histology images. Recent research (Lu and Zhu, 2022) has shown that the
use of transformer-based architectures is able to achieve significant results
for a similar segmentation task, by using a dual-path transformer model,
achieving a mean intersect over union (mIoU) score of 68.12 when segmenting
the cancer class. We note that, although impressive results have been achieved
on similar tasks for segmenting cancer cells from histology slides (Lu and
Zhu, 2022; Amgad et al., 2019), further increases in segmentation performance
may still be possible by implementing a more optimal choice of image
augmentations and associated augmentation parameters (Tellez et al., 2019).
There exists a wide range of possible image augmentations (Alomar et al.,
2023) which were not explored in the closest related works (Amgad et al.,
2019; Lu and Zhu, 2022). For example, in one related paper (Amgad et al.,
2019), only a limited selection of augmentations such as colour normalisation,
shift, and crop transforms were implemented. And in another related paper (Lu
and Zhu, 2022), only rotational augmentations were used. Applying a diverse
set of image augmentations is key to making segmentation models robust to
changes in input images and maintaining the ability to accurately identify
potential areas of tumorous cells. This robustness is particularly important
for models detecting cancerous cells in histology images as histology slides
may vary considerably between different labs and even within the same lab due
to different technicians’ slide-staining processes and variation between
slide-scanning machines (Bancroft and Layton, 2012). The authors hypothesise
that a combination of state-of-the-art segmentation models, particularly
transformer-based architectures, with a comprehensive and optimised selection
of image augmentations, can further enhance the accuracy of cancer cell
identification from breast cancer histology slides. To navigate the complex
task of selecting optimal image augmentations and associated parameters, we
propose a novel method to find optimal augmentation parameters which combines
the RandAugment auto-augmentation strategy with a tree-structured parzen
estimator optimisation strategy.

###  1.1 Research Questions

The following research questions gave rise to the project implementation:

Research Question 1:
Which image segmentation transformer model gives the best performance for
identifying tumorous regions from histopathology slides? This allows us to
start from a high-performing baseline and iterate further for better
performance.

Research Question 2:
Can a set of optimal image augmentations be discovered that improve
segmentation metrics and make more robust models? By applying a number of
different augmentations to training images, the model becomes more robust to
dataset-specific features and is better able to generalise when making
predictions on unseen data.

General research framework:
To address the above research questions, firstly the performance of
transformer-based models for segmenting cancerous cells is assessed without
image augmentation. Then the optimal number and magnitude of image
augmentations are found which are then applied to the best-performing models.
This research provides a framework for selecting optimum image augmentations,
a valuable resource for machine learning and medical researchers.

##  2 Methodology

Our general approach is to specify a dataset, explore model selection and then
apply the image augmentation optimisation process (including defining an
objective function related to improving segmentation performance as well as
defining a parameter search space) and then combining the best augmentations
and models and performing cross-validation.

###  2.1 Dataset preparation and preprocessing

In the original format, the image dataset (BCS, ) contained 151 images of
histology slides (each at resolution 0.25 microns/pixel) provided by a number
of different laboratories, and 151 corresponding segmentation masks (both
images and masks in .png image format). The dataset has a memory size of 4.59
GB, pixel widths ranging from 1222 to 9801 (mean of 4310) and pixel heights
ranging from 957 to 9248 pixels (mean of 3629). Each image corresponds to a
specific digitised histological slide (Amgad et al., 2019) (as shown in
Figures 1 and 2) and shows breast cancer cells and other biological tissues.
Data is downloaded using code provided by the researchers 111Cancer Research
with AI - CARESAI (BCS, ) and the data is prepared according to the
methodology presented in the research paper that originally generated the
dataset (Amgad et al., 2019).

To summarise the data preparation process, these steps involve collapsing the
segmentation masks down from 22 classes to 6 classes, and then splitting each
whole slide image (WSI) into overlapping patches of 800x800 pixels. The patch
size of 800x800 was chosen to allow for direct comparisons of performance
between models used by the researchers who initially generated the dataset and
the results from models that we developed. 11footnotetext: licensed under an
MIT licence

Figure 2: Example patch from histology slide (left) and 6 class segmentation
mask (right), with tumour cells shown in yellow

Each segmentation mask has the same height and width pixel dimensions as the
image it corresponds to, however instead of the RGB pixel values present in
the digitised slides, each segmentation mask contains an array of integer
values where each integer signifies a certain class is present at that
specific pixel location within the original image. Cancerous cells and healthy
cells show certain visual differences and it is these differences that
pathologists pay attention to when deciding if a ROI of the WSI shows
cancerous cells. However, these visual differences can be subtle and a
pathologist may need to spend a long period of time analysing a specific image
or region of interest to correctly determine if cancerous cells are present.
Following the methodology presented in the previous work (Amgad et al., 2019),
the initial provided ground truth masks of 22 classes were combined into 6
general classes as shown in Table 1.

Table 1: Segmentation mask class grouping Class |  Designation |  Items included in class
---|---|---
0 |  Out of scope |  Outside region of interest
1 |  Tumourous |  Tumour, Angioinvasion,
Ductal carcinoma in situ
2 |  Stroma |  Stroma
3 |  Inflammatory infiltration |  Inflammatory infiltration,
Lymphocytes,
Plasma cells,
Other immune infiltrates
4 |  Necrosis |  Necrosis
5 |  Other tissues |  Other biological tissue from all other classes

The large number of pixels present in WSIs makes it necessary to divide each
image and its corresponding segmentation mask into overlapping 800x800 pixel
patches, to facilitate easier loading into the machine learning model. This
process yields a total of 4812 Region of Interest (ROI) patches, each patch
comprising one RGB image and a corresponding mask. These ROIs are then
distributed into training, validation, and test subsets as per standard
machine learning methodologies. The allocation to subsets is based on the lab
where the slide was scanned, following the methodology used when the dataset
was first evaluated for segmentation tasks (Amgad et al., 2019; seg, ). Images
from labs coded as OL, LL, E2, EW, GM, and S3 were used in the original paper
for model performance evaluation. It is unclear if the authors used a separate
’hold-out’ testing set, so we chose ’E2’ lab images as an independent test
set. This allows us to better evaluate the model’s performance on new unseen
images after the completion of the research project and to simulate real-world
clinical applications. The allocation of image/mask pairs to each subset is
outlined in Table 2 and the pixel level class distributions per subset are
shown in Table 3. During a later stage of model comparisons, a number of
different image augmentations were included in the model training process with
the aim of making the machine learning model more generalisable and robust to
the specific distribution of images within the training data. Finally k-fold
cross validation is used to get an accurate measure of the models performance
and to make sure the model is generalising as expected.

Table 2: Training/validation/test image split Data split | Patch count | % of total
---|---|---
Training222College of Health & Medicine, School of Medicine, University of Tasmania (UTAS), Hobart, Australia | 3318 | 0.691
Validation333Boston College, Department of Biology, USA | 974 | 0.202
Testing444Chemical Biology and Drug Discovery Lab, Department of Bioscience, COMSATS University Islamabad, Pakistan | 520 | 0.108
44footnotetext: Training patch counts by lab - E9:66, GI:15, HN:9, D8:621, BH:603, C8:261, A7:273, A8:88, AC:132, AN:150, AO:274, AQ:28, AR:204, A1:55, A2:53944footnotetext: Validation patch counts by lab - OL:188, LL:62, EW:441, GM:249, S3:3444footnotetext: Test patch counts by lab - E2:520 Table 3: Distribution of pixels within each data subset belonging to each specific class Class |  Pixel dist. train (%) |  Pixel dist. val. (%) |  Pixel dist. test (%)
---|---|---|---
0 |  15.32 |  33.74 |  28.55
1 |  34.72 |  22.35 |  30.13
2 |  29.22 |  27.34 |  26.54
3 |  09.94 |  09.75 |  08.49
4 |  06.27 |  02.60 |  03.78
5 |  04.53 |  04.22 |  02.52

###  2.2 General model training

For each research question, a series of experiments were undertaken, assessing
how changing the variable(s) of interest (e.g. model type, type, number and
magnitude of image augmentations used, loss function etc.) would affect key
metrics relating to the image segmentation task. The metrics chosen for the
evaluation of each experiment are given as follows: Dice Score, IoU score
(Jaccard Index), and Accuracy. During model training, the performance of
tumour class segmentation was assessed via Dice Score (for comparison against
previous research (Amgad et al., 2019)) and mean IoU (mIoU) score (for
comparison against another previous research (Lu and Zhu, 2022)). The accuracy
metric was also included to highlight potential training issues, for example,
high accuracy paired with a low Dice score for the tumour class, may suggest
the model is over-fitting on the other non-tumour classes during the training
process. Training and validation data splits, as defined in Table 2, were
consistently maintained during model selection and augmentation comparisons in
order to control for potential variation in metrics if training and validation
sets were randomly generated before each run. It was only during the final
evaluation phase that these data splits were modified and we combine the
previous training and validation data and change to the k-fold cross
validation methodology.

We employed the mmsegmentation package555TiB - Leibniz Information Center for
Science and Technology, Hannover, Germany (Contributors, 2020; OpenMMLab,
Accessed 2023-05-12) for model training which is a PyTorch-based framework
with a large selection of segmentation architectures and pretrained models.
Model architectures, parameters, and hyperparameters are defined in a
configuration file and are loaded into a ’runner’ object which co-ordinates
data loading, model training and model evaluation. Each model was trained in a
virtual environment using a single Tesla G4 or V100 GPU (depending on GPU
availability) and 15GB of GPU RAM. 55footnotetext: licensed under the Apache
2.0 licence In addition, the Albumentations package666licensed under the
Apache 2.0 licence (Buslaev et al., 2020; Albumentations Team, Accessed
2023-05-12) was also used as it contains a large number of image augmentation
techniques (such as spatial transformations and colour manipulation). The
Optuna package777licensed under MIT license (Akiba et al., 2019; Optuna
Contributors, Accessed 2023-05-12) was used to search through the image
augmentation search space and optimise image augmentation parameters such as
number and magnitude of image augmentations.

###  2.3 Model architecture selection

Informed by existing image segmentation literature, we chose to explore
transformer-based models which have shown high performance on reference
segmentation tasks like the ADE20K and Cityscapes datasets. The inherent
attention mechanisms allow transformer-based segmentation models show improved
performance over traditional CNN models by capturing whole image information
and context (Li et al., 2023), which is crucial for accurate segmentation.
Minimal image transformations/augmentations were used during model comparison
experiments to control for the possible variation in model performance from
random/probabilistic augmentations. Image resizing and adjustment of mean and
standard deviation of image pixels on a per channel basis, facilitated
effective transfer learning (Zhuang et al., 2021) from the pretrained models.
We maintained a consistent training and validation split of images and masks
during initial model comparison, where each model was trained for 1000
iterations. The one-cycle learning rate scheduler (Smith and Topin, 2018) was
used with a maximum learning rate of 0.001, to prompt super-convergence whilst
model training, providing a performance approximation for a larger number of
training iterations. After 1000 iterations, we compared a model’s segmentation
predictions against the validation set, yielding per-class performance metrics
for Dice score, IoU score, and accuracy.

The model architectures which were tested during the initial evaluation are
given below:

SegFormer888Licensed under MIT license (Xie et al., 2021)

Mask2Former999Licensed under MIT license (Cheng et al., 2022)

PoolFormer101010Licensed under Apache 2.0 license (Yu et al., 2022)

ConvNext111111Licensed under MIT license (Liu et al., 2022)

FCN121212Unable to determine license conditions for this model (Long et al.,
2015)

As part of the secondary evaluation process, a number of models that showed
promise during the initial stage were selected for further evaluation. During
this stage, each model was trained for 2000 iterations, using cross-entropy
loss and using the one-cycle learning rate scheduler, again set to a maximum
learning rate of 0.001. We took into consideration model performance from the
initial stage as well as performance against common segmentation reference
datasets.

###  2.4 Image augmentation optimisation

We made use of the RandAugment methodology for selecting image augmentations
as this has been shown to give similar performance to more comprehensive
augmentation search strategies but considerably narrows the complexity of the
search space for possible image augmentation parameters (Cubuk et al., 2020).
A visualisation of some of the different image augmentations that were
explored can be seen in Appendix A. To search through the RandAugment search
space of augmentations, we employed the Optuna package (Akiba et al., 2019;
Optuna Contributors, Accessed 2023-05-12) and used a Tree-structured Parzen
Estimator (TPE) algorithm (Bergstra et al., 2011) to find values of ’M’
(relative strength of all image augmentations to be applied) and ’N’ (number
of augmentations to be applied) which would lead to a maximal Dice Score for
segmenting the tumour class. Upper and lower limits for augmentation
parameters are selected for each augmentation, and then between these lower
and upper values, 30 values are selected which each correspond to a specific M
value. During each optuna trial, values for M and N are generated, N augments
are randomly selected (with equal probability) from K total image
augmentations. Each trial, a model is trained using the randomly selected N
augmentations applied to the input training data with an augmentation strength
of M. For the next trial, M and N are adjusted via the Tree-based parzen
estimator search optimiser based on the previous results in order to attempt
to maximise the Dice Score for the tumour class. A baseline Dice Score is
obtained by training a model ten times without augmentations and taking the
mean of the Dice Scores for predictions against the validation set. The
performance of each individual augmentation is evaluated by using the final
value of ’M’ found in the earlier augmentation optimisation stage, training a
model five times, and taking the mean Dice Score. To streamline the
augmentation search process, each model was trained for 500 epochs with a
constant learning rate of 0.0001 and a resized patch size of 512x512. We
selected the SegFormer_B2 model for the optimisation process as it offers a
good balance between training and inference speed, ensuring the optimisation
process did not experience a bottleneck when the model makes predictions
against the validation dataset. During the comparison of image augmentation
techniques, we forced the model to use Automated Mixed Precision (AMP) during
training, to speed up the computation of loss gradients and training times
(nvi, ). Once values for M and N have been found that give the highest Dice
Score, we then perform an ablation study to identify the impact of individual
image augmentations. Ten trials without image augmentation are undertaken to
obtain a mean Dice Score baseline measure and then for each augmentation in
turn, only this single augmentation is applied, then a further ten trials are
undertaken to obtain a mean Dice Score when this augmentation is applied.

###  2.5 Loss function comparison

Loss functions play a crucial role in training machine learning models (Wang
et al., 2022), particularly in image segmentation tasks (Jadon, 2020).
Different loss functions can lead to variations in model behaviour during
training. In general, loss metrics tailored to image segmentation, like Dice
Score or Jaccard Index, tend to outperform more generic metrics like Cross-
Entropy loss (Eelbode et al., 2020). To compare the different loss functions,
the SegFormer_B5 model was used. We trained the model for 1000 iterations,
using a constant learning rate 0f 0.001, predicting against the validation set
at 500 and 1000 iterations and taking the mean Dice Score to account for
natural variations in Dice Score whilst training.

###  2.6 Combining best models and augmentations

Once we have found the optimal values for M and N from the augmentation search
process, we then examine the individual impact of the augmentations when each
is applied with magnitude M. The N best-performing individual augmentations
are selected. Next, we perform k-folds cross-validation with 10 folds. The
training and validation images are combined into a single dataset, the image
order is shuffled and then the images are randomly split into 10 folds. For
each of the best-performing models, we perform two sets of k-fold cross-
validation experiments. The first is to obtain a baseline for each model
without augmentations. We train a particular model 10 different times (one for
each fold) for 2000 epochs, and each time the model is trained, 9 of the folds
are used for training and the 10th fold is used for validation. The Dice Score
against the validation set is saved for each fold. Now we repeat the k-fold
cross-validation process and this time, each time a model is trained, the best
N augmentations with augmentation parameters of magnitude M are applied to the
training data. The validation data during each fold is left in its original
format without augmentations applied.

##  3 Discussion of results

###  3.1 Initial model evaluation

In the primary evaluation phase, a number of different model architectures and
subtypes are compared after being trained for 1000 iterations with the same
split of training and validation images. A summary of the initial model
comparison results is given in Table 4 with models grouped by model type
(detailed results can be found in the supplementary material).

Table 4: Initial model comparison results, grouped by model type Model group |  Mean Dice Score |  Std Dice Score |  Mean IoU |  Std IoU
---|---|---|---|---
Segformer |  67.50 |  0.71 |  50.94 |  0.81
Poolformer |  66.20 |  1.50 |  49.49 |  1.50
Mask2Former |  64.96 |  6.61 |  48.36 |  6.93
ConvNext |  62.00 |  2.27 |  44.95 |  2.39
FCN |  58.75 |  1.30 |  41.59 |  1.30

Table 4 shows summary statistics for the different model types during the
initial model evaluation experiments, including Dice and IoU scores for
predictions of tumour class segmentation compared against the validation
dataset. As we see from the table, the transformer-based models show increased
segmentation performance compared to the convolutional-based models. This is
because transformer-based models can capture global context for a patch via
the attention mechanism, allowing for dynamic focus on relevant image regions.
Transformers can learn hierarchical representations which allows features at
multiple scales to be captured, which is crucial for medical images where
cellular structures can vary in size and shape. This makes transformers adept
at handling the complexity and variability often seen in histology slides,
providing more robust and accurate segmentation compared to traditional
convolutional-based segmentation models.

Table 5: Best performing models - initial evaluation Model name | Val.Dice | Val.IoU
---|---|---
Mask2former_R51 | 69.03 | 52.76
Poolformer_FPN_S24 | 68.31 | 51.87
Mask2former_Swin-T | 68.02 | 51.52
Segformer_B3 | 67.87 | 51.37
Segformer_B5 | 67.82 | 51.33
Mask2former_Swin-S | 67.73 | 51.21
Poolformer_FPN_M36 | 66.41 | 49.71

###  3.2 Further model evaluation

After considering the results from the initial evaluation, we decided not to
continue experimenting with the ConvNext and FCN models due to their poor Dice
Scores compared to the other types of models. We selected the following models
for further evaluation: SegFormer_B5, PoolFormer_FPN_S36, and
PoolFormer_FPN_M36131313As part of the secondary evaluation process, it was
planned to experiment further with the Mask2Former models such as M48 but we
faced technical difficulties involving gradient instability and model
divergence with the training process for this model type . and the results for
these models are shown in Table 6.

Table 6: Results of the secondary evaluation experiments, showing Dice Score, IoU and accuracy after 2000 iterations Model Backbone | DS | IoU | Acc
---|---|---|---
PoolFormer_M36_FPN | 70.27 | 54.17 | 87.68
PoolFormer_S36_FPN | 69.92 | 53.75 | 89.99
SegFormer_B5 | 68.84 | 52.49 | 90.69

We have shown that the PoolFormer model using an M48 backbone and an FPN
decode head is able to achieve the best performance when segmenting the tumour
class from histology slides.

###  3.3 Comparison of loss types

Table 7: Comparison of different loss types, showing Dice Scores at 500 and 1000 iterations and the mean Dice Score Loss type | DS500 | DS1000 | Mean DS
---|---|---|---
Focal | 61.12 | 68.40 | 64.76
CE | 54.42 | 68.21 | 61.32
Lovasz | 47.99 | 63.49 | 55.74
Dice | 68.39 | 58.80 | 63.60
Tvarsky | 67.44 | 56.29 | 61.87

It is shown in Table 7 that when trained for 1000 iterations with no image
augmentations, the Segformer_B5 model shows the best performance when using a
Focal loss function, followed by Cross-Entropy loss function. As the
mmsegmentation package allows multiple losses to be used during back-
propagation, during the final evaluation of the models we attempted to create
a custom loss combining Focal and CE losses. We reasoned that two loss
functions working together would allow the model to generalise better as each
loss function would penalise different sources of error. However, when
attempting to train models using this custom loss function, models began to
exhibit divergent behaviour and so we could not investigate fully the effect
of combining these different loss types.

###  3.4 Individual augmentation comparison

After the image augmentation search, optimal RandAugment values were found to
be N=10 and M=27 (see Appendix B for a detailed table showing the progression
of the N and M values during the optimisation process). Individual
augmentations were then tested using this value of ’M’ to set each set of
augmentation parameters. Table 8 identifies the top five augmentations as
Sharpen, GridDropout, Emboss, ColorJitter, and HueSaturationValue. ColorJitter
and HueSaturationValue enhance colour variation in the training data,
simulating staining differences between different labs when processing the
histology slides. Sharpen and Emboss augmentations likely improve feature
emphasis for cancerous cells by emphasising specific features of the cells and
nucleus typically associated with cancerous cells, such as nucleus shape.
GridDropout forces the model to consider adjacent image regions, enhancing
visual context. GridDistortion and ElasticTransform augmentations are shown in
previous research to give increased performance, yet here surprisingly they
show decreased segmentation performance. This could be due to the high
augmentation magnitude (M), causing excessive image distortion which impedes
the training process.

Table 8: Comparison of specific image augmentations, showing Dice Score percentage difference vs the baseline as well as the p-value of the difference in means between the baseline and augmentation trials Augmentation |  Mean DS |  % Diff. in DS |  Std DS |  p-val
---|---|---|---|---
Sharpen |  67.78 |  1.51 |  0.34 |  0.023
GridDropout |  67.63 |  1.29 |  0.29 |  0.045
Emboss |  67.52 |  1.12 |  1.47 |  0.396
ColorJitter |  67.50 |  1.09 |  0.91 |  0.240
HueSaturationValue |  67.37 |  0.90 |  0.84 |  0.302
OpticalDistortion |  67.29 |  0.78 |  1.07 |  0.440
GaussianBlur |  67.23 |  0.69 |  1.47 |  0.591
CutOut |  66.84 |  0.11 |  0.80 |  0.896
CLAHE |  66.71 |  -0.09 |  1.09 |  0.927
GridDistortion |  66.71 |  -0.10 |  1.02 |  0.921
RandomContrast |  66.62 |  -0.23 |  2.06 |  0.893
ElasticTransform |  65.88 |  -1.34 |  1.14 |  0.223
RandomGamma |  60.28 |  -9.72 |  11.33 |  0.316

###  3.5 Combined best models and augmentation results

Table 9: Dice Score mean and standard deviations from k-fold cross-validation comparing baseline vs augmentations 151515(BL: baseline without augmentations, WA: with augmentations, p-value calculated for difference in means with and without augmentations)and p-value for difference in means Model |  BL Mean |  BL Std |  WA Mean |  WA Std |  p-val
---|---|---|---|---|---
PoolFormer_S36 |  81.07 |  1.37 |  81.05 |  1.56 |  0.98
PoolFormer_M36 |  80.84 |  1.35 |  81.10 |  1.14 |  0.36
SegFormer_B5 |  76.81 |  2.47 |  78.36 |  2.90 |  0.36

The best metrics achieved during the k-fold cross-validation with
augmentations were a Dice Score of 84.08 and an IoU score of 72.54 (achieved
by the PoolFormer_S36 model). A similar Dice Score is obtained compared to the
closest work (Amgad et al., 2019) which achieved a Dice Score of 85.1 As
indicated in Table 15, when comparing Dice Scores, the image augmentation
methodology did not yield any statistically significant improvements over the
baseline models . Although initially, this appears to be a negative result, it
offers a valuable insight: the Dice Scores remained stable even after the
introduction of ten different image augmentations, applied at near maximum
magnitude. Even though previous research (Lu and Zhu, 2022) uses a more
technically advanced model architecture, we demonstrated a higher IoU than the
value of 68.12 that the previous researchers achieved. This fact that the
segmentation metrics are similar or better than existing research suggests
that our methodology has resulted in models which have gained an enhanced
ability to generalize across diverse input images and have become more robust
to variations in input data, making it a versatile tool for segmenting
tumorous cells in histology images from multiple labs without compromising
segmentation performance. This resilience to data variability is a valuable
attribute, especially in medical applications involving histology slides where
data heterogeneity is often a challenge. Due to concerns regarding potential
over-fitting in our model, due to the repeated number of model trainings that
were undertaken during the augmentation optimisation process, we implemented
several strategies to mitigate this risk. Notably, we incorporated weight
decay, a form of L2 regularization, into our training process. This technique
penalizes larger weights in the model, encouraging simpler models that are
less likely to overfit. Furthermore, by employing the k-fold cross-validation
methodology, this allows us to evaluate the model’s performance across
different subsets of the data. Throughout this process, we closely monitored
the loss metrics for both training and validation sets. The absence of a
significant increase in validation loss compared to training loss during any
specific model training using a certain k-fold is a strong indicator of the
model’s generalization capabilities.

##  4 Limitations

This study relies on the assumption of accurate segmentation masks in the
dataset from previous research (Amgad et al., 2019). If these masks are
inaccurate, the trained model’s performance may be compromised. However, the
segmentation masks in the dataset were validated by multiple pathology
professionals which should ensure the annotations are accurate. Also, the
images within the dataset are assumed to be representative of histology slides
found in clinical settings. If the dataset is atypical in some manner, the
model may fail to make accurate predictions on clinical images. The image
augmentation optimisation process may cause overfitting on the validation set
due to repeated predictions against the validation set. However, the increased
variation of images within the training set after applying the augmentations
should counterbalance this potential overfitting as well as the use of k-folds
cross-validation during the final augmentation comparison process. Due to
hardware and time constraints, we were not able to test our augmentation
methodology in combination with larger models such as Beit (Bao et al., 2022)
and due to technical issues during model training, we were not able to fully
explore the performance of some Mask2Former models. It is possible that
employing an auxiliary segmentation head, in combination with the best-
performing models and augmentations may further increase segmentation
performance. Training a model on Whole Slide Images (WSIs) rather than patched
Regions of Interest (ROIs) could potentially improve segmentation performance,
but it introduces additional challenges due to the large image sizes.
Furthermore, testing the augmentation framework on models trained on a single
dataset limits the generalisability of our findings. However, there are an
extremely small number of annotated segmentation datasets relating to breast
cancer histology slides so at this time it is not a trivial matter to expand
our methodology to explore images from different datasets. Finally, employing
model pretraining with histology-specific datasets (Kawai et al., 2023) could
also enhance baseline performance and the impact of the augmentation
framework.

##  5 Conclusions

In conclusion, this research set out to develop an innovative image
augmentation methodology aimed at improving the performance of segmentation
models for identifying tumorous cells in histology slides. The methodology
yielded similar or better segmentation metrics than existing research and the
methodology resulted in more robust models without suffering a loss of
segmentation performance. This robustness to data variability is particularly
crucial in the medical imaging domain, where data can differ substantially due
to various factors such as equipment, preparation techniques, and human error.
Our methodology, therefore, contributes to the broader goal of creating more
reliable and versatile machine-learning models for medical applications,
specifically for segmenting histology slides from different sources. Overall,
this research offers valuable insights for the ongoing advancement of medical
image segmentation technologies and the developed methodology and provided
codebase will assist with the development of breast cancer cell segmentation
from histology slides.

##  6 Acknowledgements

We would like to thank our colleagues at Cancer Research with AI (CARESAI)
group for their assistance, support, and motivation in completing this
research project. Additionally, we would like to give thanks to Lucas Puvis de
Chavannes, Amina Chouigui, Abner Hernandez, Magnus Kallinger, and Andrii
Krutsylo for their time spent reviewing the paper prior to submission and
providing suggestions for improvements.

## References

  * (1) Bcss github repository.  https://github.com/PathologyDataScience/BCSS.  Accessed: 2023-05-14.
  * (2) NVIDIA Deep Learning Documentation - Mixed Precision Training.  NVIDIA Deep Learning Documentation.  URL https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html.  Accessed: [13/05/2023].
  * (3) Papers with code, vision transformers.  ’https://paperswithcode.com/method/vision-transformer.  Accessed: [12/04/23].
  * (4) Bcseg2015 grand challenge - baseline.  https://bcsegmentation.grand-challenge.org/Baseline/.  Accessed on May 13, 2023.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2019.
  * Aksac et al. (2020) A. Aksac, T. Ozyer, D. J. Demetrick, et al.  Cactus: cancer image annotating, calibrating, testing, understanding and sharing in breast cancer histopathology.  _BMC Res Notes_ , 13(1):13, 2020.  10.1186/s13104-019-4866-z.  URL https://doi.org/10.1186/s13104-019-4866-z.
  * Albayrak and Bilgin (2018) A. Albayrak and G. Bilgin.  Automatic cell segmentation in histopathological images via two-staged superpixel-based algorithms.  _Medical & Biological Engineering & Computing_, 57(3):653–665, Oct. 2018.  10.1007/S11517-018-1906-0.
  * Albumentations Team (Accessed 2023-05-12) Albumentations Team.  Albumentations: Fast and flexible image augmentations.  https://github.com/albumentations-team/albumentations, Accessed 2023-05-12.
  * Alomar et al. (2023) Khaled Alomar, Halil Ibrahim Aysel, and Xiaohao Cai.  Data augmentation in classification and segmentation: A survey and new strategies.  _Journal of Imaging_ , 9(2):46, 2023.  10.3390/jimaging9020046.  URL https://doi.org/10.3390/jimaging9020046.
  * Amgad et al. (2019) Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai A T Elsebaie, Lamia S Abo Elnasr, Rokia A Sakr, Hazem S E Salem, Ahmed F Ismail, Anas M Saad, Joumana Ahmed, Maha A T Elsebaie, Mustafijur Rahman, Inas A Ruhban, Nada M Elgazar, Yahya Alagha, Mohamed H Osman, Ahmed M Alhusseiny, Mariam M Khalaf, Abo-Alela F Younes, Ali Abdulkarim, Duaa M Younes, Ahmed M Gadallah, Ahmad M Elkashash, Salma Y Fala, Basma M Zaki, Jonathan Beezley, Deepak R Chittajallu, David Manthey, David A Gutman, and Lee A D Cooper.  Structured crowdsourcing enables convolutional segmentation of histology images.  _Bioinformatics_ , 35(18):3461–3467, Sep 2019\.  10.1093/bioinformatics/btz083.  URL https://doi.org/10.1093/bioinformatics/btz083.
  * Bancroft and Layton (2012) John D Bancroft and Christopher Layton.  The hematoxylins and eosin.  _Bancrofts theory and practice of histological techniques_ , 7:173–186, 2012.
  * Bao et al. (2022) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: BERT pre-training of image transformers.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Bergstra et al. (2011) James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl.  Algorithms for hyper-parameter optimization.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , 2011.
  * Buslaev et al. (2020) Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin.  Albumentations: Fast and flexible image augmentations.  _Information_ , 11(2), 2020.  ISSN 2078-2489.  10.3390/info11020125.  URL https://www.mdpi.com/2078-2489/11/2/125.
  * (15) cancer.net.  Breast cancer statistics.  https://www.cancer.net/cancer-types/breast-cancer/statistics.  Accessed: Date.
  * Cheng et al. (2022) Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  2022\.
  * Cohen (2020) S. Cohen.  _Artificial Intelligence and Deep Learning in Pathology_.  Elsevier Health Sciences, 2020.
  * Contributors (2020) MMSegmentation Contributors.  Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, July 2020.  License: Apache-2.0.
  * Cubuk et al. (2020) Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  _Computer Vision and Pattern Recognition - cs.CV conference_ , 2020\.
  * Das et al. (2020) Asha Das, Madhu S Nair, and S David Peter.  Computer-aided histopathological image analysis techniques for automated nuclear atypia scoring of breast cancer: a review.  _Journal of digital imaging_ , 33:1091–1121, 2020.
  * Demir and Yener (2005) C. Demir and B. Yener.  Automated cancer diagnosis based on histopathological images: a systematic survey.  Technical report, Rensselaer Polytechnic Institute, 2005.
  * Eelbode et al. (2020) Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B. Blaschko.  Optimization for medical image segmentation: Theory and practice when evaluating with dice score or jaccard index.  _IEEE Transactions on Medical Imaging_ , 39(11):3679–3690, 2020.  10.1109/TMI.2020.3002417.
  * Han et al. (2023) Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao.  A survey on vision transformer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(1):87–110, 2023.  10.1109/TPAMI.2022.3152247.
  * He et al. (2012) L. He, L.R. Long, S. Antani, and GR. Thoma.  Histology image analysis for carcinoma detection and grading.  _Computer Methods and Programs in Biomedicine_ , 107(3):538–556, 2012.
  * Holzinger et al. (2020) A. Holzinger, R. Goebel, M. Mengel, and H. Müller.  _Artificial Intelligence and Machine Learning for Digital Pathology: State-of-the-art and Future Challenges_ , volume 12090.  Springer Nature, 2020.
  * Jadon (2020) Shruti Jadon.  A survey of loss functions for semantic segmentation.  In _2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB)_ , pages 1–7, 2020.  10.1109/CIBCB48159.2020.9277638.
  * Jia et al. (2017) Z. Jia, X. Huang, E. I.-C. Chang, and Y. Xu.  Constrained deep weak supervision for histopathology image segmentation.  _IEEE Transactions on Medical Imaging_ , 36(11):2376–2388, Jul. 2017.  10.1109/TMI.2017.2724070.
  * Kawai et al. (2023) Masataka Kawai, Noriaki Ota, and Shinsuke Yamaoka.  Large-scale pretraining on pathological images for fine-tuning of small pathological benchmarks, 2023.
  * Krithiga and Geetha (2020) R. Krithiga and P. Geetha.  Breast cancer detection, segmentation and classification on histopathology images analysis: A systematic review.  _Archives of Computational Methods in Engineering_ , 28(4):2607–2619, Aug. 2020.  10.1007/S11831-020-09470-W.
  * Kumar et al. (2017) V. Kumar, A.K. Abbas, and J.C. Aster.  _Robbins Basic Pathology E-book_.  Elsevier Health Sciences, 2017.
  * Lakshmanan and Saravanakumar (2018) B. Lakshmanan and S. Saravanakumar.  Nucleus segmentation in breast histopathology images.  In _2018 International Conference on Communication and Computational Technologies (ICCTCT)_ , Mar. 2018.  10.1109/ICCTCT.2018.8550929.
  * Li et al. (2023) Jun Li, Junyu Chen, Yucheng Tang, Ce Wang, Bennett A Landman, and S Kevin Zhou.  Transforming medical imaging with transformers? a comparative review of key properties, current progresses, and future perspectives.  _Medical image analysis_ , page 102762, 2023.
  * Liu et al. (2022) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2015.
  * Lu and Zhu (2022) Xuan Lu and Xiaofeng Zhu.  Automatic segmentation of breast cancer histological images based on dual-path feature extraction network.  _Mathematical Biosciences and Engineering_ , 19(11):11137–11153, Aug 2022.  10.3934/mbe.2022519.
  * Madabhushi and Lee (2016) A. Madabhushi and G. Lee.  Image analysis and machine learning in digital pathology: Challenges and opportunities.  _Medical Image Analysis_ , 33:170–175, 2016.
  * Mills et al. (2012) S.E. Mills, D. Carter, J.K. Greenson, V.E. Reuter, and M.H. Stoler.  _Sternberg’s Diagnostic Surgical Pathology_.  Lippincott Williams & Wilkins, 2012.
  * Minaee et al. (2022a) S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos.  Image segmentation using deep learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(7):3523–3542, Jun. 2022a.  10.1109/TPAMI.2021.3051370.
  * Minaee et al. (2022b) Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(7):3523–3542, 2022b.  10.1109/TPAMI.2021.3059968.
  * Moorthy and Gandhi (2022) Jayashree Moorthy and Usha Devi Gandhi.  A survey on medical image segmentation based on deep learning techniques.  _Big Data and Cognitive Computing_ , 6(4), 2022.  ISSN 2504-2289.  URL https://www.mdpi.com/2504-2289/6/4/117.
  * OpenMMLab (Accessed 2023-05-12) OpenMMLab.  Openmmlab semantic segmentation toolbox.  https://github.com/open-mmlab/mmsegmentation, Accessed 2023-05-12.
  * Optuna Contributors (Accessed 2023-05-12) Optuna Contributors.  Optuna: A hyperparameter optimization framework.  https://optuna.org/, Accessed 2023-05-12.
  * Ott et al. (2009) J. J. Ott, A. Ullrich, and A. B. Miller.  The importance of early symptom recognition in the context of early detection and cancer survival.  _European Journal of Cancer_ , 45(16):2743–2748, Sep. 2009.  10.1016/j.ejca.2009.07.027.
  * Rashmi et al. (2021) R. Rashmi, K. Prasad, and CBK. Udupa.  Breast histopathological image analysis using image processing techniques for diagnostic purposes: A methodological review.  _J Med Syst_ , 46(1):7, Dec 2021.  10.1007/s10916-021-01786-9.
  * Roy and Gupta (2020) Bijoyeta Roy and Mousumi Gupta.  Macroscopic reconstruction for histopathology images: A survey.  In _Computer Vision and Machine Intelligence in Medical Image Analysis: International Symposium, ISCMM 2019_ , pages 101–112. Springer, 2020\.
  * Sayed et al. (2015) S. Sayed, R. Lukande, and K. A. Fleming.  Providing pathology support in low-income countries.  _JCO Global Oncology_ , 1(1):3–6, Sep. 2015.  10.1200/JGO.2015.000547.
  * Smith and Topin (2018) Leslie N. Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates, 2018.
  * Tellez et al. (2019) David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen van der Laak.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology.  _Medical Image Analysis_ , 58:101544, 2019.  ISSN 1361-8415.  https://doi.org/10.1016/j.media.2019.101544.  URL https://www.sciencedirect.com/science/article/pii/S1361841519300799.
  * Tosta et al. (2017) T. A. A. Tosta, L. A. Neves, and M. Z. do Nascimento.  Segmentation methods of h&e-stained histological images of lymphoma: A review.  _Informatics in Medicine Unlocked_ , 9:35–43, Jan. 2017\.  10.1016/J.IMU.2017.05.009.
  * Wang et al. (2022) Qi Wang, Yue Ma, Kun Zhao, and Yingjie Tian.  A comprehensive survey of loss functions in machine learning.  _Annals of Data Science_ , 9(2):187–212, April 2022.  10.1007/s40745-020-00253-5.
  * Xie et al. (2021) Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In _Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Yan et al. (2020) Rui Yan, Fei Ren, Zihao Wang, Lihua Wang, Tong Zhang, Yudong Liu, Xiaosong Rao, Chunhou Zheng, and Fa Zhang.  Breast cancer histopathological image classification using a hybrid deep neural network.  _Methods_ , 173:52–60, 2020.
  * Yu et al. (2022) Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10819–10829, 2022.
  * Zhuang et al. (2021) Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1):43–76, 2021\.  10.1109/JPROC.2020.3004555.
  * Ziegenhorn and et al. (2020) H.-V. Ziegenhorn and et al.  Breast cancer pathology services in sub-saharan africa: a survey within population-based cancer registries.  _BMC Health Services Research_ , 20(1):1–10, Oct. 2020.  10.1186/s12913-020-05726-0.

##  7 Appendices

##  Appendix A Visualisation of augmentations

##  Appendix B RandAugment Search results

Dice Scores against the validation set, found during the search process for
different RandAugment values of M and N

M | N | Final Dice Score
---|---|---
27 | 9 | 69.30
21 | 5 | 69.29
21 | 3 | 68.90
9 | 2 | 68.38
21 | 7 | 68.03
17 | 6 | 68.00
15 | 4 | 67.93
25 | 6 | 67.61
25 | 7 | 67.53
13 | 3 | 67.32
21 | 10 | 67.04
23 | 5 | 66.83
13 | 1 | 66.37
29 | 8 | 66.32
29 | 8 | 66.15
27 | 9 | 66.11
19 | 5 | 65.71
7 | 10 | 63.74
25 | 4 | 63.46

##  Appendix C RandAugment augmentation parameter generation corresponding to
various M values

Lower and upper limits for the augmentation parameters were chosen from
reviewing the existing literature and looking for commonly used values. For
each parameter, 30 values were generated between the lower and upper limits
(inclusive). These values were rounded to the nearest integer if the
augmentation parameter is unable to accept floats as inputs.

    augmentation_dict = {
    ’RandomGamma’: {
    ’gamma_limit_upper’:
    equally_spaced_nums(0, 120, 30,True)
    },
    ’CLAHE’: {
    ’clip_limit’:
    equally_spaced_nums(0, 10, 30, True),
    ’tile_grid_size’:
    equally_spaced_nums(8, 64, 30, True)
    },
    ’CutOut’: {
    ’num_holes’:
    equally_spaced_nums(0, 30, 30, True),
    ’max_h_size’:
    equally_spaced_nums(0, 20, 30, True),
    ’max_w_size’:
    equally_spaced_nums(0, 20, 30, True)
    },
    ’HueSaturationValue’: {
    ’hue_shift_limit’:
    equally_spaced_nums(0, 50, 30, True),
    ’sat_shift_limit’:
    equally_spaced_nums(0, 50, 30, True),
    ’val_shift_limit’:
    equally_spaced_nums(0, 50, 30, True)
    },
    ’ColorJitter’: {
    ’brightness’: equally_spaced_nums
    (0, 1.0, 30, False),
    ’contrast’:
    equally_spaced_nums (0, 1.0, 30, False),
    ’saturation’:
    equally_spaced_nums(0, 1.0, 30, False),
    ’hue’:
    equally_spaced_nums(0, 1.0, 30, False)
    },
    ’ElasticTransform’: {
    ’alpha’:
    equally_spaced_nums(0, 1000, 30, True),
    ’sigma’:
    equally_spaced_nums(0, 50, 30, True),
    ’alpha_affine’:
    equally_spaced_nums(0, 60, 30, True)},
    ’OpticalDistortion’: {
    ’distort_limit_high’:
    equally_spaced_nums(0, 0.5, 30, True),
    ’shift_limit_high’:
    equally_spaced_nums(0, 0.5, 30, True)
    },
    ’GridDistortion’: {
    ’num_steps’:
    equally_spaced_nums(4, 20, 30, True),
    ’distort_limit’:
    equally_spaced_nums(0, 1.0, 30, False)
    },
    ’RandomContrast’: {
    ’contrast_limit_upper’:
    equally_spaced_nums(0, 1.0, 30, False)
    },
    ’GridDropout’: {
    ’unit_size_min’:
    equally_spaced_nums(10, 210, 30, True),
    ’unit_size_max’:
    equally_spaced_nums(10, 210, 30, True),
    ’holes_num’:
    equally_spaced_nums(0, 100, 30, True),
    },
    ’GaussianBlur’: {
    ’blur_limit’:
    equally_spaced_nums(0, 10, 30, False),
    ’sigma_limit’:
    equally_spaced_nums(0.0, 5.0, 30, False)
    },
    ’Sharpen’: {
    ’alpha’: equally_spaced_nums(0.0, 1.0, 30, False),
    ’lightness’: equally_spaced_nums(0, 10, 30, False)
    },
    ’Emboss’: {
    ’alpha’: equally_spaced_nums(0.0, 1.0, 30, False),
    ’strength’: equally_spaced_nums(0.0, 2.0, 30, False)
    }
    }

##  Appendix D Final augmentations and parameters obtained from augmentation
search

    [{’type’: ’CLAHE’,
    ’clip_limit’: 9,
    ’tile_grid_size’: 60,
    ’p’: 1.0},
    {’type’: ’ColorJitter’,
    ’brightness’: 0.931,
    ’contrast’: 0.931,
    ’saturation’: 0.931,
    ’hue’: 0.931, ’p’: 1.0},
    {’type’: ’CutOut’,
    ’num_holes’: 28,
    ’max_h_size’: 19,
    ’max_w_size’: 19,
    ’fill_value’: 255,
    ’p’: 1.0},
    {’type’: ’Emboss’,
    ’alpha’: 0.931,
    ’strength’: 1.8621,
    ’p’: 1.0},
    {’type’: ’GaussianBlur’,
    ’blur_limit’: 9.3103,
    ’sigma_limit’: 4.6552,
    ’p’: 1.0},
    {’type’: ’GridDropout’,
    ’ratio’: 0.5,
    ’unit_size_min’: 0,
    ’unit_size_max’: 196,
    ’holes_number_x’: 93,
    ’holes_number_y’: 93,
    ’shift_x’: 0,
    ’shift_y’: 0,
    ’random_offset’: False,
    ’fill_value’: 0,
    ’mask_fill_value’: None,
    ’always_apply’: False, ’p’: 1.0},
    {’type’: ’HueSaturationValue’,
    ’hue_shift_limit’: 47,
    ’sat_shift_limit’: 47,
    ’val_shift_limit’: 47,
    ’p’: 1.0},
    {’type’: ’OpticalDistortion’,
    ’distort_limit’: 0.2,
    ’shift_limit’: 0.2, ’p’: 1.0},
    {’type’: ’Sharpen’,
    ’alpha’: 0.931,
    ’lightness’: 9.3103,
    ’p’: 1.0}]