# Deep Boosting Robustness of DNN-based Image Watermarking via DBMark

Guanhui Ye∗ Jiashi Gao∗ Wei Xie∘ Bo Yin⊗ Xuetao Wei∗
∗Southern University of Science and Technology ∘Hunan University ⊗Changsha
University of Science and Technology

###### Abstract

Image watermarking is a technique for hiding information into images that can
withstand distortions while requiring the encoded image to be perceptually
identical to the original image. Recent work based on deep neural networks
(DNN) has achieved impressive progression in digital watermarking. Higher
robustness under various distortions is the eternal pursuit of digital image
watermarking approaches. In this paper, we propose DBMark, a novel end-to-end
digital image watermarking framework to deep boost the robustness of DNN-based
image watermarking. The key novelty is the synergy of invertible neural
networks (INN) and effective watermark features generation. The framework
generates watermark features with redundancy and error correction ability
through the effective neural network based message processor, synergized with
the powerful information embedding and extraction abilities of INN to achieve
higher robustness and invisibility. The powerful learning ability of neural
networks enables the message processor to adapt to various distortions. In
addition, we propose to embed the watermark information in the discrete
wavelet transform (DWT) domain and design low-low ($LL$) sub-band loss to
enhance invisibility. Extensive experiment results demonstrate the superiority
of the proposed framework compared with the state-of-the-art ones under
various distortions such as dropout, cropout, crop, Gaussian filter, and JPEG
compression.

Figure 1: The architecture of DBMark. In the watermark encoding process, a
secret message $M$ is encoded into a cover image $I_{co}$ to generate an
encoded image $I_{en}$. In watermark decoding process, the encoded image
$I_{en}$ passing through the noise layer is fed in a reverse direction to
recover the secret message $M^{\prime}$. Here, we use dense block in our
$\phi(\cdot)$, $\rho(\cdot)$ and $\eta(\cdot)$ modules. The forward process
and the reverse process of INN share the same network parameters and modules.

##  1 Introduction

Digital watermarking has been widely used in the copyright protection of
multimedia products since its inception [29]. Digital watermarking hides the
watermark information with specific meanings in digital content, such as
images, videos, audio, documents, etc., through digital embedding. The
extraction and recovery of watermark information can be used to prove the
ownership and as evidence for identifying illegal infringement. The goal of
digital watermarking is to embed the secret message into the cover image in an
invisible way and to extract the accurate secret message in the case of
various distortions. In other words, digital watermarking requires high
robustness and high invisibility. Least significant bits (LSB) [29] was the
earliest research on image information hiding, which encodes the secret
message on the least significant bits of image pixels. However, statistical
measures [12, 11, 10] can easily detect the secret information hidden by LSB.
Furthermore, the researchers find that watermarking in the frequency domain is
more robust than the spatial ones. However, these traditional methods are
heavily dependent on shallow manual image features, which imply that they need
to be carefully designed and can not fully use the redundant information of
cover images, so the robustness of such methods is limited.

In recent years, with the upsurge of deep learning, many researchers have
applied deep neural networks (DNN) to digital image watermarking, which
significantly facilitates its development. These DNN-based methods [35, 22,
17] have shown advantages in robustness under various distortions compared
with traditional methods. Zhu et al. [35] proposed the first DNN-based method
named Hidden and demonstrated superior performance than most traditional
methods. Meanwhile, various subsequent DNN-based methods have adopted a
similar framework. Such a framework uses a separate encoder and decoder, which
treats the watermark encoding and decoding processes independently. Xu et al.
[32] simply applied invertible neural networks (INN) in image watermarking,
which did not achieve satisfactory performance. Since higher robustness is the
eternal pursuit of these DNN-based image watermarking approaches, our research
question is: how to deep boost the robustness of DNN-based image watermarking
under various distortions?

In this paper, we propose a novel end-to-end digital image watermarking
framework DBMark to deep boost the robustness of DNN-based image watermarking.
The key novelty is the synergy of invertible neural networks (INN) and
effective watermark features generation. The framework generates watermark
features with redundancy and error correction ability through the effective
neural network based message processor, synergized with the powerful
information embedding and extraction abilities of INN to achieve higher
robustness and invisibility. The powerful learning ability of neural networks
enables the message processor to adapt to various distortions. We embed and
extract watermark features by INN, in which the forward and inverse processes
of INN share the same parameters and modules. Instead of hiding information
directly in the spatial domain, we embed watermark information in the
frequency domain of the cover image through the DWT. Moreover, we embed most
of the watermark information into the high-frequency component through the
$LL$ sub-band loss to improve invisibility from the perspective of the human
visual system. Experiment results show that our DBMark framework outperforms
existing SOTA (State Of The Art) methods on robustness and invisibility
evidently.

In summary, the main contributions of this paper are:

  * •

We present a novel end-to-end digital image watermarking framework DBMark to
deep boost the robustness of DNN-based image watermarking, which is the
synergy of the invertible neural networks (INN) and effective watermark
features generation.

  * •

We propose the neural network based message processor to generate watermark
features with redundancy and error correction ability, significantly improving
robustness against various distortions simultaneously.

  * •

We propose low-low ($LL$) sub-band loss to embed more watermark information in
the high-frequency component of the discrete wavelet transform (DWT) domain,
which enhances the invisibility of our DBMark evidently.

  * •

We conduct extensive experiments to demonstrate that our framework DBMark
achieves higher robustness and invisibility than state-of-the-art approaches
under various distortions.

The rest of the paper is organized as follows. We review the related work
about digital watermarking and INN in Section 2. The details of the proposed
framework are described in Section 3. Extensive experiments are presented in
Section 4. Finally, we conclude our work in Section 5.

##  2 Related Work

###  2.1 Digital Watermarking

As a primary technology for copyright protection of content, digital
watermarking is a popular research area in a wide range of real-world
scenarios [5, 7, 14, 20, 26, 19]. The prior research of traditional methods
has mainly investigated pixel-level manipulation [29, 4] in spatial domain. In
order to improve the robustness, the traditional methods targeted the
frequency domain, such as discrete Fourier transform (DFT) [27] domain,
discrete cosine transform (DCT) [15] domain and discrete wavelet transform
(DWT) [13] domain.

In recent years, DNN-based methods have shown more advantages than traditional
methods in both invisibility and robustness against various distortions, which
is due to the powerful feature extraction ability of deep neural networks. Zhu
et al. [35] first proposed a DNN-based framework to jointly trained the
encoder and the decoder with a noise layer. Ahmadi et al. [1] introduced
domain transform in the DNN-based method and used the strength factor to
change the strength of the watermark presented in the image. Luo et al. [25]
proposed an attack network to simulate real distortions without any prior
knowledge on the type of distortion during training, which obtained comparable
results than other DNN-based methods and showed good generalization to unknown
distortions. To achieve high robustness to the anticipated distortions, adding
noise to encoded images during training is the most common and effective
approach. However, this approach can not work well for non-differentiable
distortions like JPEG compression. To overcome such limitation, many methods
have been proposed. Zhu et al. [35] and Shin et al. [28] proposed
differentiable JPEG-Mask and JPEG-SS respectively to approximate the real
JPEG. However, the gap between the JPEG simulation and the real JPEG results
in poor robustness against real JPEG. Liu et al. [22] proposed a two-stage
separable framework to solve the non-differentiable distortions problem. The
method jointly trains the encoder and the decoder without a noise layer in
stage one to obtain a powerful encoder. In stage two, the encoder is frozen
and the decoder gains robustness enhancement from noise layer. Jia et al. [17]
focused on JPEG compression and proposed a mixed training method that randomly
selects one from Identity, JPEG-Mask and real JPEG as noise layer during
training. Furthermore, Zhang et al. [34] proposed the forward attack
simulation layer method to improve the robustness against non-differentiable
distortions in an end-to-end framework. The previous work has demonstrated
that DNN have great potential in digital image watermarking.

###  2.2 Invertible Neural Networks

INN can be seen as a bijective function, which contains a forward mapping
$f_{\theta}(\cdot)$ and a corresponding inverse mapping
$f_{\theta}^{-1}(\cdot)$. Both mappings have a tractable Jacobian matrix,
allowing explicit computation of posterior probabilities. Therefore, for the
computational result that y generated by forward process $y=f_{\theta}(x)$, we
can recover x directly through $x=f^{-1}_{\theta}(y)$.

Since the Invertible Neural Network was first proposed by Nice [8], INN has
been applied for many computer vision tasks due to its outstanding
performance. Zhu et al. [36] first utilized the bidirectional mapping of the
INN to replace the cycle loss in CycleGan [2]. Ardizzone et al. [3] proposed
conditional INN for image generation and colorization. Xiao et al. [31]
introduced the invertible bijective transform of INN for image-rescaling
tasks. Jing et al. [18] and Lu et al. [24] applied INN to image steganography,
which achieved large capacity and high invisibility. In addition, INN are also
used for image-to-video synthesis [9], image compression [30], image denoising
[23] and video super-resolution [37]. Although INN have great potential in
information embedding and extraction, it shows weakness in robustness against
lossy data compression and other distortions, which are the key issues of
digital watermarking. Xu et al. [32] applied INN in image watermarking, but
the simple usage of INN can not achieve high robustness.

##  3 The Framework DBMark

In this section, we describe the proposed framework DBMark in detail. As shown
in Figure 1, the proposed model is an end-to-end framework including DWT
modules, message processors, INN blocks, and a noise layer. The following
subsections will describe these modules in detail. Table 1 presents the
definition of notations in this paper.

Table 1: Notation definition.

Notation | Definition
---|---
$M$ | secret message: the message to be hidden
$M^{\prime}$ | recovered message: the message recovered from encoded image
$I_{co}$ | cover image: the image to hide secret message
$I_{en}$ | encoded image: the image with secret message inside
$I_{no}$ | noised image: the encoded image with noise
$F_{co}$ | cover image features: the features of cover image
$F_{en}$ | encoded image features: the features of encoded image
$F_{no}$ | noised image features: the features of noised image
$F_{M}$ | watermark features: the features of secret message
$r$ | lost information: the information lost in encoding process
$z$ | auxiliary matrix: the matrix to help recover secret message

###  3.1 Network Architecture

Figure 1 shows the framework architecture of our DBMark. In the watermark
encoding process, the cover image and secret message are received as pair-wise
input. The cover image $I_{co}$ $\in$ $\mathbb{R}^{C\times H\times W}$ is
first transformed into frequency domain features $F_{co}$ $\in$
$\mathbb{R}^{4C\times H/2\times W/2}$ through DWT), in which $C$ is channel
number, $H$ is height, and $W$ is width. The secret message $M$ $\in$
$\left\\{0,1\right\\}^{L}$ needs to enter the message processor to generate
the watermark features $F_{M}$ $\in$ $\mathbb{R}^{C_{M}\times H/2\times W/2}$
with the same dimension as the frequency domain features $F_{co}$. $C_{M}$ is
the channel number of watermark features, which is set to equal $L$ by
default. Then, these two kinds of features go through a series of invertible
neural network blocks. The output of the last block contains the frequency
domain feature of encoded image $F_{en}$ and the lost information $r$. The
encoded image features $F_{en}$ produce the encoded image $I_{en}$ after
inverse discrete wavelet transform (IWT). The noise layer adds random noise to
the encoded image and outputs the noised image $I_{no}$. In the watermark
decoding process, the noised image features $F_{no}$ and an auxiliary matrix
$z$ pass through a series of invertible neural network blocks to extract the
watermark features $F_{M^{\prime}}$, which are then fed into the inverse
message processor to recover the secret message.

Discrete Wavelet Transform. DWT is a transform that decomposes the original
image into different sub-bands, namely low-low ($LL$), low-high ($LH$), high-
low ($HL$), and high-high ($HH$). The $LL$ sub-band contains no edge
information since it is the low-frequency component of the image. According to
the masking effect of the human sensory system, the digital watermark
information can be embedded in the area where the original carrier is not
easily perceptible, especially the high-frequency component in the DWT domain,
so that the digital watermark has strong invisibility. Here, we use the $Haar$
wavelet to implement our DWT algorithm, which is simple and effective. After
DWT, the feature map of the cover images with size $(C,H,W)$ is transformed
into a frequency domain with size $(4C,H/2,W/2)$.

Message Processor. In the watermark encoding process, the one-dimensional
secret message $M$ needs to be processed to two-dimensional watermark features
$F_{M}$ with the same spatial dimension as the frequency domain features of
cover images $F_{co}$. Here, we design a neural network to generate watermark
features instead of simply duplicating the message or using a manual coding
algorithm. $F_{M}$ contains much more bits than the original secret message,
making it redundant and error-correcting. Here, we design a message processor
to generate watermark features. First, the secret message $M$ $\in$
$\\{0,1\\}^{L}$ passes through a fully connected layer to increase the length
of the message to $L^{\prime}$. The motivation for this fully connected layer
is from [17], which uses the full connection layer to diffuse watermark
information to the whole image to resist distortions. Here, we utilize the
fully connected layer to diffuse the watermark features to the frequency
domain of the cover image. Then, the message is reshaped to a $3$-dimension
tensor $\\{0,1\\}^{1\times h\times w}$ where $L^{\prime}=h\times w$. It is fed
into a single convolution layer to increase the channel number to $C_{M}$. And
then, the shape of $F_{M}$ expanded to $C_{M}\times H/2\times W/2$ by several
2$\times$ upsampling layers. After that, the watermark features are sent to
several Squeeze-and-Excitation Network (SENet) [16] blocks and then pass
through INN concatenated with the cover image features.

Figure 2:  Visual comparisons of cover and encoded images of our DBMark. First
row: Cover image $I_{co}$. Second row: Encoded images $I_{en}$. Third row:
Corresponding noised images $I_{no}$ under six types of distortions . Fourth
row: The difference between $I_{co}$ and $I_{en}$, which is five times
magnified for visualization. PSNR = 39.98 and SSIM = 0.957.

In the watermark decoding process, the inverse message processor receives the
watermark features $F_{M^{\prime}}$ from the first block of INN and outputs
the recovered message $M^{\prime}$. The inverse message processor has a
similar network structure to the message processor but with a reverse
information flow direction. During the upsampling and downsampling step, the
number of upsampling or downsampling layers $n$ is decided by $L^{\prime}$,
$H$, and $W$. The formula is defined as follows,

$L^{\prime}=h\times w=(H/{2^{n-1}})\times(W/{2^{n-1}}).$ (1)

Forward and Inverse Process in INN. The INN has shown outstanding performance
in image steganography [18, 24]. In our proposed framework, we build the INN
blocks to embed and extract the watermark features to the image. The INN
consists of $N$ INN blocks with the same structure. As shown in Figure 1, the
forward and inverse processes in the INN share the same INN blocks but with
reverse data flow directions. In the forward process, the INN receives the
frequency domain features of the cover image $F_{co}$ and watermark features
$F_{M}$ as inputs. For the $i$-th INN block, we denote the inputs as
$b_{1}^{i}$ and $b_{2}^{i}$. The corresponding outputs $b_{1}^{i+1}$ and
$b_{2}^{i+1}$ are calculated as follows,

$b_{1}^{i+1}=b_{1}^{i}+\phi\left(b_{2}^{i}\right),$ (2)
$b_{2}^{i+1}=b_{2}^{i}\odot\exp\left(\alpha\left(\rho\left(b_{1}^{i+1}\right)\right)\right)+\eta\left(b_{1}^{i+1}\right),$
(3)

where $\alpha(\cdot)$ is a $sigmoid$ function multiplied by a constant,
$exp(\cdot)$ is an exponential function, and $\odot$ is the Hadamard product.
Here, $\phi(\cdot)$, $\rho(\cdot)$ and $\eta(\cdot)$ are arbitrary functions
and we apply the dense block for its simplicity and effectiveness. Finally,
the INN outputs $b_{1}^{N+1}$ and $b_{2}^{N+1}$, where $b_{1}^{N+1}$ are the
frequency domain features of encoded image $F_{en}$ and $b_{2}^{N+1}$ is lost
information $r$. $F_{en}$ is then processed by IWT to generate the encoded
image $I_{en}$.

Accordingly, the inverse process in the $i$-th block of INN is formulated as
follows,

$b_{1}^{i}=b_{1}^{i+1}-\phi\left(b_{2}^{i}\right),$ (4)
$b_{2}^{i}=\left(b_{2}^{i}-\eta\left(b_{1}^{i+1}\right)\right)\odot\exp\left(-\alpha\left(\rho\left(b_{1}^{i+1}\right)\right)\right).$
(5)

In order to keep the dimensions of the input and output of the INN consistent,
the input contains an auxiliary matrix $z$ in addition to the frequency domain
features $F_{no}$ generated by the noised image passing through the DWT
module. The auxiliary matrix $z$ is randomly sampled from a Gaussian
distribution. Specifically, for the $N$-th block, the inputs $b_{1}^{N+1}$ and
$b_{2}^{N+1}$ are the frequency domain features of the noised image $F_{no}$
and the auxiliary matrix $z$ correspondingly. Finally, the output of the first
block $b_{2}^{1}$ is fed into the inverse message processor as the watermark
feature $F_{M^{\prime}}$.

Noise Layer. To improve the robustness against various distortions, inserting
a noise layer after encoded images is the most common method in deep learning.
However, the standard noise layer can not gain enhanced robustness from non-
differentiable noise like JPEG. Zhang et al. [34] proposed a forward attack
simulation layer method to overcome such limitations. The method calculates
the difference between noised and encoded images and indicates this difference
as the pseudo-noise. Then, the pseudo-noise is added to the encoded image to
generate the pseudo-noised image. During the backward propagation, the pseudo-
noise does not participate in the gradient propagation. Therefore, the
gradient from the inverse blocks is directly back-propagated to the forward
process in INN without passing through the noise layer. In our DBMark, we
propose to apply the forward attack simulation layer method for non-
differentiable noise and standard noise layer for differentiable noise.

###  3.2 Loss Functions

Our DBMark has two main goals. The goal of the watermark encoding process is
to embed the secret message $M$ into the cover image $I_{co}$ to generate an
encoded image $I_{en}$. The encoded image is required to be as similar as
possible to the cover image. Therefore, the encoding loss function
$\mathcal{L}_{en}$ is calculated by mean square error (MSE),

$\mathcal{L}_{en}=MSE(I_{co},I_{en}).$ (6)

The watermark decoding process aims to extract and recover the secret message
from the encoded image and minimize the difference between the original secret
message $M$ and the recovered message $M^{\prime}$. According to this goal, we
define the decoding loss function as follows,

$\mathcal{L}_{de}=MSE(M,M^{\prime}).$ (7)

Since the human eye is less sensitive to noise in high-frequency sub-bands of
the image, Zebbiche et al. [33] proposed a perceptual masking model to embed
the hidden information into the high-frequency components in the DWT domain,
which enhances the performance significantly in terms of invisibility and
robustness. Inspired by this work, we propose a $LL$ sub-band loss
$\mathcal{L}_{LL}$ to enhance our DBMark’s invisibility. Suppose that
$\mathcal{F}(\cdot)_{LL}$ is a filter to extract the $LL$ sub-band of the
image. Here, we minimize the difference between the cover image and encoded
image in $LL$ sub-band to allow more watermark information to be embedded in
the high-frequency components of the encoded image. Therefore, the $LL$ sub-
band loss is defined as follows,

$\mathcal{L}_{LL}=MSE(\mathcal{F}(I_{co})_{LL},\mathcal{F}(I_{en})_{LL}).$ (8)

In summary, the total loss function $\mathcal{L}_{total}$ is formulated as:

$\mathcal{L}_{total}=\mathcal{\lambda}_{en}\mathcal{L}_{en}+\mathcal{\lambda}_{de}\mathcal{L}_{de}+\mathcal{\lambda}_{LL}\mathcal{L}_{LL},$
(9)

where $\mathcal{\lambda}_{en}$, $\mathcal{\lambda}_{de}$ and
$\mathcal{\lambda}_{LL}$ are the weights of the corresponding losses for
balancing different loss functions.

Table 2: Comparison with [35], [22], [32] and [17] trained by specified noise layer. SSIM is not reported in [35], [22] and [32]. PSNR is measured for RGB channels, except in [35], they use Y channel of YUV channels. Our DBMark achieves the highest PSNR/SSIM and the lowest BER. Method | Hidden [35] | TSDL [22] | IWN[32] | MBRS [17] | DBMark
---|---|---|---|---|---
Message size | 30 | 30 | 30 | 64 | 64
Noise Layer | JPEG-Mask | JPEG | JPEG-SS | Mixed(MBRS) | Forward ASL
PSNR | (for Y) 30.09 | 33.51 | 36.16 | 36.49 | 39.75
SSIM | - | - | - | 0.9137 | 0.9486
BER | 15% | 22.3% | 8.8% | 0.0092% | 0.0018%
Table 3: Comparison with [35], [22], [32] and [17] trained by combined noise
layer. Our DBMark achieves the lowest BER under six types of distortions.

Method | PSNR | Identity | Cropout ($p=0.3$) | Dropout ($p=0.3$) | Crop ($p=0.035$) | GF ($\sigma$ = 2) | JPEG ($Q=50$)
---|---|---|---|---|---|---|---
Hidden [35] | 33.5 | 0% | 6% | 7% | 12% | 4% | 37%
TSDL [22] | 33.5 | 0% | 2.7% | 2.6% | 11% | 1.4% | 23.8%
IWN [32] | 33.0 | 0.06% | 5.29% | 24.71% | 16.69% | 13.89% | 23.13%
MBRS [17] | 33.5 | 0% | 0.0027% | 0.0087% | 4.15% | 0.011% | 4.48%
DBMark | 34.8 | 0% | 0.002% | 0.0013% | 3.74% | 0% | 0%

###  3.3 Strength Factor

We can get the difference between the encoded image and cover image
$I_{diff}=I_{en}-I_{co}$, which is the watermark mask with the secret message
$M$. Therefore, we can adjust the watermarking strength by a strength factor
$S$ to balance robustness and invisibility:

$I_{en,S}=I_{co}+S*I_{diff}.$ (10)

In section 4.2.3, we will discuss the influence of different strength factors
on the performance of our framework.

##  4 Experiments

In this section, we conduct experiments to validate the efficacy and
robustness of our proposed framework DBMark. Performance against the baseline
and parameter sensitivity on performance are considered for effectiveness
verification. The experiments cover a wide range of complex image distortions
in terms of robustness.

###  4.1 Experimental Settings

Implementation Details. Our DBMark is trained under $10000$ images of COCO
[21] training data set, and the neural network is evaluated through $5000$
images of COCO test data set to ensure the generalization of the model. Our
model is implemented through PyTorch [6] and runs on NVIDIA A100 GPUs $\times
2$. In order to compare with previous work, we adjust the image’s resolution
to $128\times 128$. The secret message lengths are $30$ bits and $64$ bits for
combined training and specified training, where each bit is generated by
random sampling. Our DBMark contains $16$ invertible neural network blocks.
The parameters $\mathcal{\lambda}_{en}$, $\mathcal{\lambda}_{de}$ and
$\mathcal{\lambda}_{LL}$ are set to $0.1$, $100.0$ and $0.1$, respectively.
The mini-batch size is $16$, and the Adam optimizer is adopted with standard
parameters and an initial learning rate of $1\times 10^{-5.0}$. For specified
training, we add one specified distortion in the noise layer during the
training. For combined training, we choose one distortion from Identity,
Cropout ($p=0.3$), Dropout ($p=0.3$), Crop ($p=0.035$), Gaussian Filter (GF,
$\sigma=2$) and JPEG ($Q=50$) randomly and add it to encoded image in noise
layer. Since the input of the watermark decoding process is a fixed size
tensor, we pad the cropped images with pixels expressed as $(127,127,127)$ in
RGB channels.

Table 4: BER, PSNR, SSIM value under several strength factors. BER is tested under different distortions. As the factor increases, PSNR and SSIM values decrease steadily, but BER value decreases first and then increases. Strength factor | 0.5 | 0.75 | 1.0 | 1.25 | 1.5 | 1.75 | 2.0 | 2.25
---|---|---|---|---|---|---|---|---
BER | Identity | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00067% | 0.0087%
Cropout | 0.79% | 0.015% | 0.00067% | 0.00067% | 0.0013% | 0.0013% | 0.0013% | 0.0013%
Dropout | 1.67% | 0.051% | 0.0047% | 0.00% | 0.00% | 0.00067% | 0.00% | 0.00%
Crop | 25.6% | 12.5% | 5.68% | 2.92% | 1.96% | 1.62% | 1.56% | 1.56%
GF | 4.11% | 0.081% | 0.00% | 0.00% | 0.00067% | 0.002% | 0.0027% | 0.004%
JPEG | 8.43% | 0.175% | 0.0034% | 0.00067% | 0.00% | 0.00% | 0.00% | 0.0013%
PSNR | 44.34 | 40.81 | 38.32 | 36.38 | 34.79 | 33.46 | 32.30 | 31.27
SSIM | 0.9801 | 0.9584 | 0.9321 | 0.9033 | 0.8733 | 0.8430 | 0.8130 | 0.7838
Figure 3:  Samples of encoded and cover images for our DBMark with/without
$\mathcal{L}_{LL}$ loss. From left to right are: the cover image $I_{co}$, the
encoded image with $\mathcal{L}_{LL}$ loss $I_{en}$, the difference between
$I_{en}$ and $I_{co}$, the encoded image without $\mathcal{L}_{LL}$ loss
$I^{\prime}_{en}$, the difference between $I^{\prime}_{en}$ and $I_{co}$. The
differences are five times magnified for visualization.

Metrics. There are two objectives in digital image watermarking: robustness,
measured by the Bit Error Rate (BER) of the recovered message; and
invisibility, measured by both peak signal-to-noise ratio (PSNR) and
structural similarity (SSIM) between the cover and encoded image.

Benchmarks. To verify the performance of our proposed DBMark, we conduct the
comparison with four DNN-based methods[35], [22], [32] and [17], especially
with the state-of-the-art (SOTA) digital watermark method proposed in [17].
The architecture of [32] is based on INN, but the author does not open-source
their code. Although [35] and [22] open-sourced their code, we can not
reproduce the results as they reported. In order to respect the results they
have reported, we directly compare them with the results published in [35],
[22] and [32]. [17] open-sourced both their codes and their model, so we use
the pre-trained model for comparison.

###  4.2 Experimental Results

In this subsection, we show the comparison results between our DBMark and
previous methods. For JPEG compression, since the message length of each
method is different, we use a larger message length to train our model for a
fair comparison. Specifically, we use secret message length $L=64$, which is
larger than [35], [22] and [32]. For combined compression, we use secret
message length $L=30$. In terms of visual quality, our watermark mask is
distributed across the image like vertical stripes. The qualitative comparison
between the cover image and the encoded images, and the illustration of the
noised image under different distortions are shown in Figure 2.

####  4.2.1 JPEG Comparison

In this experiment, we mainly compare the JPEG robustness between the proposed
method and four other methods, [35], [22], [32] and [17]. Our model is trained
by the forward ASL method proposed by [34] with real JPEG. All the testing
processes are performed under real JPEG compression with quality factor
$Q=50$. As shown in Table 2, our DBMark achieves higher PSNR and SSIM and
lower BER, which indicates higher invisibility and robustness. Significantly,
the BER is only $0.0018\%$. Experimental results demonstrate that our DBMark
is more effective against JPEG compression than the previous methods.

####  4.2.2 Combined Comparison

To demonstrate that our DBMark is resistant to various distortions
simultaneously, we train a noise-resistant combined model by using a random
noise layer for each mini-batch during the training. The noise layers include
Identity, Cropout ($p=0.3$), Dropout ($p=0.3$), Crop ($p=0.035$), Gaussian
Filter (GF, $\sigma=2$) and JPEG ($Q=50$). We use forward ASL for JPEG since
it is a non-differentiable distortion. We adjust the strength factor to
achieve the best performance of our DBMark. When the strength factor is large,
with the increment of the strength factor, the robustness and invisibility of
the encoded image decrease simultaneously. Therefore, we do not use a large
strength factor to obtain similar PSNR value with different methods. As we can
see in Table 3, our DBMark can maintain higher PSNR and lower BER than
previous methods under each kind of distortion. In particular, our method can
achieve $0\%$ bit error rate for Gaussian filter and JPEG compression
distortions. Furthermore, our DBMark has a dramatic improvement in robustness
compared to [32], which is also implemented based on INN.

####  4.2.3 Strength Factor

We introduce a strength factor $S$ in Equation (10) to adjust the trade-off
between the quality of the encoded image and the BER of the recovered message.
In order to determine the best strength factor for our method, we test our
DBMark under different strength factor values from $0.5$ to $2.25$. As we can
see in Table 4, with the increment of strength factor, PSNR and SSIM values
decrease steadily. However, the BER values under different distortions
decrease first and then increase, which is different from the negative
correlation relationship between $S$ and BER under different quality factors
for JPEG compression in [17]. This observation indicates that the message
recovery in our DBMark depends not only on those changed pixels but also on
those unchanged pixels. Therefore, it demonstrates that our DBMark can better
utilize the redundant information of the image.

####  4.2.4 Effectiveness of $LL$ sub-band loss

The $\mathcal{L}_{LL}$ loss is designed to hide more watermark information in
the high-frequency component of the cover image so that the human eye can
hardly detect the presence of a watermark. Visual comparisons of encoded
images with and without $\mathcal{L}_{LL}$ loss are shown in Figure 3. As we
can see, the encoded images with $\mathcal{L}_{LL}$ loss maintain a higher
image quality rather than the one without $\mathcal{L}_{LL}$ loss, which
demonstrates that the $\mathcal{L}_{LL}$ loss can enhance the invisibility of
our method. In addition, the differences between $I_{co}$ and
$I^{\prime}_{en}$ are like the afterimage of the cover image. In contrast, we
can only see some sparse vertical artifacts in the differences between
$I_{co}$ and $I_{en}$.

##  5 Conclusion

In this paper, we have proposed a novel end-to-end digital image watermarking
framework DBMark to deep boost the robustness of DNN-based image watermarking.
We have proposed to generate robust watermark features through message
processing against distortions. In model training, the $LL$ sub-band loss has
been proposed to improve the visual quality of the encoded image. Our DBMark
has synergized the strengths of INN, discrete wavelet transform and the
forward attack simulation layer method to offer better robustness and
invisibility. Extensive experiments have demonstrated that our DBMark can
achieve digital image watermarking with higher robustness and invisibility,
outperforming other SOTA methods evidently.

## References

  * [1] Mahdi Ahmadi, Alireza Norouzi, Nader Karimi, Shadrokh Samavi, and Ali Emami.  Redmark: Framework for residual diffusion watermarking based on deep networks.  Expert Systems with Applications, 146:113157, 2020.
  * [2] Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville.  Augmented cyclegan: Learning many-to-many mappings from unpaired data.  In International Conference on Machine Learning, pages 195–204. PMLR, 2018.
  * [3] Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe.  Guided image generation with conditional invertible neural networks.  arXiv preprint arXiv:1907.02392, 2019.
  * [4] Abdullah Bamatraf, Rosziati Ibrahim, and Mohd Najib B Mohd Salleh.  Digital watermarking algorithm using lsb.  In 2010 International Conference on Computer Applications and Industrial Electronics, pages 155–159. IEEE, 2010.
  * [5] Walter Bender, Daniel Gruhl, Norishige Morimoto, and Anthony Lu.  Techniques for data hiding.  IBM systems journal, 35(3.4):313–336, 1996.
  * [6] Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet.  Torch7: A matlab-like environment for machine learning.  In BigLearn, NIPS workshop, number CONF, 2011.
  * [7] Ingemar J Cox, Matthew L Miller, Jeffrey Adam Bloom, and Chris Honsinger.  Digital watermarking, volume 53.  Springer, 2002.
  * [8] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  arXiv preprint arXiv:1410.8516, 2014.
  * [9] Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G Derpanis, and Bjorn Ommer.  Stochastic image-to-video synthesis using cinns.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3742–3753, 2021.
  * [10] Sorina Dumitrescu, Xiaolin Wu, and Zhe Wang.  Detection of lsb steganography via sample pair analysis.  In International workshop on information hiding, pages 355–372. Springer, 2002.
  * [11] Jessica Fridrich and Miroslav Goljan.  Practical steganalysis of digital images: state of the art.  security and Watermarking of Multimedia Contents IV, 4675:1–13, 2002.
  * [12] Jessica Fridrich, Miroslav Goljan, and Rui Du.  Detecting lsb steganography in color, and gray-scale images.  IEEE multimedia, 8(4):22–28, 2001.
  * [13] Huiping Guo and Nicolas D Georganas.  Digital image watermarking for joint ownership verification without a trusted dealer.  In 2003 International Conference on Multimedia and Expo. ICME’03. Proceedings (Cat. No. 03TH8698), volume 2, pages II–497. IEEE, 2003\.
  * [14] Mohamed Hamidi, Mohamed El Haziti, Hocine Cherifi, and Driss Aboutajdine.  A blind robust image watermarking approach exploiting the dft magnitude.  In 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA), pages 1–6. IEEE, 2015.
  * [15] Mohamed Hamidi, Mohamed El Haziti, Hocine Cherifi, and Mohammed El Hassouni.  Hybrid blind robust image watermarking technique based on dft-dct and arnold transform.  Multimedia Tools and Applications, 77(20):27181–27214, 2018.
  * [16] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [17] Zhaoyang Jia, Han Fang, and Weiming Zhang.  Mbrs: Enhancing robustness of dnn-based watermarking by mini-batch of real and simulated jpeg compression.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 41–49, 2021.
  * [18] Junpeng Jing, Xin Deng, Mai Xu, Jianyi Wang, and Zhenyu Guan.  Hinet: deep image hiding by invertible network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4733–4742, 2021.
  * [19] Umair Khadam, Muhammad Munwar Iqbal, Muhammad Awais Azam, Shehzad Khalid, Seungmin Rho, and Naveen Chilamkurti.  Digital watermarking technique for text document protection using data mining analysis.  IEEE Access, 7:64955–64965, 2019.
  * [20] Eelandula Kumaraswamy, G Mahesh Kumar, Kommabatla Mahender, Krishnaveni Bukkapatnam, and Ch Rajendra Prasad.  Digital watermarking: State of the art and research challenges in health care & multimedia applications.  In IOP Conference Series: Materials Science and Engineering, volume 981, page 032031. IOP Publishing, 2020.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [22] Yang Liu, Mengxi Guo, Jian Zhang, Yuesheng Zhu, and Xiaodong Xie.  A novel two-stage separable deep learning framework for practical blind watermarking.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 1509–1517, 2019.
  * [23] Yang Liu, Zhenyue Qin, Saeed Anwar, Pan Ji, Dongwoo Kim, Sabrina Caldwell, and Tom Gedeon.  Invertible denoising network: A light solution for real noise removal.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13365–13374, 2021.
  * [24] Shao-Ping Lu, Rong Wang, Tao Zhong, and Paul L Rosin.  Large-capacity image steganography based on invertible neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10816–10825, 2021.
  * [25] Xiyang Luo, Ruohan Zhan, Huiwen Chang, Feng Yang, and Peyman Milanfar.  Distortion agnostic deep watermarking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13548–13557, 2020.
  * [26] Zhaoxiong Meng, Tetsuya Morizumi, Sumiko Miyata, and Hirotsugu Kinoshita.  Design scheme of copyright management system based on digital watermarking and blockchain.  In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), volume 2, pages 359–364. IEEE, 2018.
  * [27] JJKO Ruanaidh, WJ Dowling, and Francis M Boland.  Phase watermarking of digital images.  In Proceedings of 3rd IEEE International Conference on Image Processing, volume 3, pages 239–242. IEEE, 1996.
  * [28] Richard Shin and Dawn Song.  Jpeg-resistant adversarial images.  In NIPS 2017 Workshop on Machine Learning and Computer Security, volume 1, page 8, 2017.
  * [29] Ron G Van Schyndel, Andrew Z Tirkel, and Charles F Osborne.  A digital watermark.  In Proceedings of 1st international conference on image processing, volume 2, pages 86–90. IEEE, 1994.
  * [30] Yaolong Wang, Mingqing Xiao, Chang Liu, Shuxin Zheng, and Tie-Yan Liu.  Modeling lost information in lossy image compression.  arXiv preprint arXiv:2006.11999, 2020.
  * [31] Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu.  Invertible image rescaling.  In European Conference on Computer Vision, pages 126–144. Springer, 2020.
  * [32] Hong-Bo Xu, Rong Wang, Jia Wei, and Shao-Ping Lu.  A compact neural network-based algorithm for robust image watermarking.  arXiv preprint arXiv:2112.13491, 2021.
  * [33] Khalil Zebbiche and Fouad Khelifi.  Efficient wavelet-based perceptual watermark masking for robust fingerprint image watermarking.  IET Image Processing, 8(1):23–32, 2014.
  * [34] Chaoning Zhang, Adil Karjauv, Philipp Benz, and In So Kweon.  Towards robust deep hiding under non-differentiable distortions for practical blind watermarking.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 5158–5166, 2021.
  * [35] Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei.  Hidden: Hiding data with deep networks.  In Proceedings of the European conference on computer vision (ECCV), pages 657–672, 2018.
  * [36] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [37] Xiaobin Zhu, Zhuangzi Li, Xiao-Yu Zhang, Changsheng Li, Yaqi Liu, and Ziyu Xue.  Residual invertible spatio-temporal network for video super-resolution.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5981–5988, 2019.


possible for the case $\gamma=1/2$, which plays here the role of an algebraic
critical point separating short-range spin liquid phases with no divergence-
free polarization tensor.

####  V.4.2 The case $\gamma=-1$

Figure 27: Fluxes structure allowing to build a Gauss Law for corner-sharing
hexagons lattice with $\gamma=-1$. Each bond of the dice lattice is oriented
as depicted with arrows. A flux made of two spins taken with opposites signs
is attached to each bond.

In this case, we can use the kagome dual lattice depicted in Fig. 27 to place
fluxes. Each oriented bond is attached with a flux composed of two spins
belonging to the link at the right of the arrow, see Fig. 27 where the spins
attached to a bond are highlighted with the corresponding bond color. The two
spins come with signs $+$ or $-$, in such a way that the projection of the
dipole on the bond points in the bond direction. In this way we obtain

$\sum_{i\in\hexagon}\bm{\Pi}_{i}=\bm{\mathcal{S}}_{p}=0$ (84)

when summing around a hexagonal plaquette. For vertices surrounded by a
triangle, the sum of outgoing fluxes gives

$\sum_{i\in\triangle}\bm{\Pi}_{i}=\sum_{i\in\triangle}\left(\mathbf{S}_{i}-\mathbf{S}_{i}\right)=0,$
(85)

meaning the sum of the fluxes is zero for each vertex of the dual lattice. The
Polarization tensor $\bm{\Pi}_{i}$ constructed from these fluxes following Eq.
(58) is thus divergence-free in the ground state manifold, as expected.

##  VI Conclusion and perspectives

In summary, our research provides an extensive investigation of three distinct
families of classical spin liquids derived from cluster Hamiltonians,
employing a combination of complementary analytical and numerical techniques.
On the analytical side, the key ingredient is the definition of the constraint
vector $\mathbf{L}_{\mathbf{q}}$, which serves as the building block of the
LTA and the Projective Analysis of the structure factors and, in some cases,
whose topological properties allow for a classification of the different kind
of spin liquid phases. On the numerical side, we used extensive MC simulations
which turn out to corroborate the analytical results but also account for the
entropic effect at non-zero temperature.

The first two families of spin liquids that we investigated, defined on the
checkerboard and kagome lattices, exhibit algebraic correlations, and a real
space Gauss Law is explicitly derived for all parameter values in the
Hamiltonian. The analytical and MC analysis also shows that for some critical
values of the parameters, higher rank gauge fields emerge. These two models
nevertheless differ in the fact that the second one clearly shows OBD
phenomena and a subsequent selection within the ground state manifold.

The final example that we analyzed reveals a distinct qualitative behavior,
the system is predominantly in a short-range spin liquid phase for most
microscopic parameter values. There are however different short-range phases
separated by critical points where the system has algebraic correlations. For
two of these points with algebraic behavior, we provided an explicit
construction of a divergence-free flux field, and the LTA analysis makes us
believe that it should be the case also for the third point.

In conclusion, our study highlights the effectiveness of complementary
analytical and numerical techniques in the investigation of classical spin
liquids. By analyzing three distinct families of cluster Hamiltonians, which
we can consider as tailored benchmark models, we have shown that we can
encompass what we believe is the vast majority of the zoology that one can
encounter in the study of two-dimensional classical spin liquids, such as OBD
phenomena, vector and higher rank tensor gauge fields, and its associated
multi-arm pinch points, and the co-existence in the phase diagram of short-
range and algebraic spin liquid states. Our results are also of particular
importance from the experimental perspective. Indeed, for example in material
candidates for kagome spin liquids such as polymorphs herbertsmithite and
kapellasite Mendels  _et al._ (2007); Olariu  _et al._ (2008); Fåk  _et al._
(2012), there are competing interactions and it is interesting to see how
their corresponding ab-initio Hamiltonians can approach one of the categories
of the model studied here. Our results are also of course of first importance
for the elaboration of artificial spin liquid materials (see refs. Skjærvø
_et al._ , 2020; Rougemaille and Canals, 2019 for a compelling review of
different geometries), where the control on the design of the setup can allow
to stick to different families of spin liquids presented here. Furthermore,
our analysis is not limited to two-dimensional systems, and its extension to
three-dimensional models built from extended plaquettes holds considerable
promise.

##  Acknowledgments

The authors thank the CNRS International Research Project COQSYS for their
support. F. A. G. A. and H. D. R. thank the Laboratoire de Physique Théorique,
Toulouse, France, for their hospitality during the scientific stay when this
project was initiated. F. A. G. A. and H. D. R. acknowledge financial support
from CONICET (PIP 2021-11220200101480CO, PIBAA 2872021010 0698CO), Agencia
I+D+i (PICT 2020 Serie A 03205), and SECyT-UNLP (I+D X893).

Note: Recently, another independent work regarding the classification of
classical spin liquids appeared as a preprint by Yan et al. Yan  _et al._
(2023).

## References

  * Wannier (1950) G. H. Wannier, Phys. Rev. 79, 357 (1950).
  * Isakov  _et al._ (2004) S. V. Isakov, K. Gregor, R. Moessner, and S. L. Sondhi, Phys. Rev. Lett. 93, 167204 (2004).
  * Henley (2005) C. L. Henley, Phys. Rev. B 71, 014424 (2005).
  * Anderson (1956) P. W. Anderson, Phys. Rev. 102, 1008 (1956).
  * Youngblood and Axe (1981) R. W. Youngblood and J. D. Axe, Phys. Rev. B 23, 232 (1981).
  * Garanin and Canals (1999) D. A. Garanin and B. Canals, Phys. Rev. B 59, 443 (1999).
  * Villain, J.  _et al._ (1980) Villain, J., Bidaux, R., Carton, J.-P., and Conte, R., J. Phys. France 41, 1263 (1980).
  * Rehn  _et al._ (2017) J. Rehn, A. Sen, and R. Moessner, Phys. Rev. Lett. 118, 047201 (2017).
  * Benton and Moessner (2021) O. Benton and R. Moessner, Phys. Rev. Lett. 127, 107202 (2021).
  * Luttinger and Tisza (1946) J. M. Luttinger and L. Tisza, Phys. Rev. 70, 954 (1946).
  * Luttinger (1951) J. M. Luttinger, Phys. Rev. 81, 1015 (1951).
  * Kaplan and Menyuk (2007) T. Kaplan and N. Menyuk, Philosophical Magazine 87, 3711 (2007).
  * Yan  _et al._ (2018) H. Yan, R. Pohle, and N. Shannon, Phys. Rev. B 98, 140402 (2018).
  * Prem  _et al._ (2018) A. Prem, S. Vijay, Y.-Z. Chou, M. Pretko, and R. M. Nandkishore, Phys. Rev. B 98, 165140 (2018).
  * Pretko (2017a) M. Pretko, Phys. Rev. B 95, 115139 (2017a).
  * Pretko (2017b) M. Pretko, Phys. Rev. B 96, 035119 (2017b).
  * Yan  _et al._ (2023) H. Yan, O. Benton, R. Moessner, and A. H. Nevidomskyy, (2023), arXiv:2305.00155 [cond-mat.str-el] .
  * Yan  _et al._ (2020) H. Yan, O. Benton, L. D. C. Jaubert, and N. Shannon, Phys. Rev. Lett. 124, 127203 (2020).
  * Gómez Albarracín and Rosales (2016) F. A. Gómez Albarracín and H. D. Rosales, Phys. Rev. B 93, 144413 (2016).
  * Chalker  _et al._ (1992) J. T. Chalker, P. C. W. Holdsworth, and E. F. Shender, Phys. Rev. Lett. 68, 855 (1992).
  * Moessner and Chalker (1998a) R. Moessner and J. T. Chalker, Phys. Rev. B 58, 12049 (1998a).
  * Rehn  _et al._ (2016) J. Rehn, A. Sen, K. Damle, and R. Moessner, Phys. Rev. Lett. 117, 167201 (2016).
  * Gómez Albarracín and Pujol (2018) F. A. Gómez Albarracín and P. Pujol, Phys. Rev. B 97, 104419 (2018).
  * Gómez Albarracín and Rosales (2021) F. A. Gómez Albarracín and H. D. Rosales, Journal of Physics: Condensed Matter 33, 185801 (2021).
  * Benton  _et al._ (2016) O. Benton, L. D. C. Jaubert, H. Yan, and N. Shannon, Nature Communications 7, 11572 (2016).
  * Hornreich  _et al._ (1975) R. M. Hornreich, M. Luban, and S. Shtrikman, Phys. Rev. Lett. 35, 1678 (1975).
  * Moessner and Chalker (1998b) R. Moessner and J. T. Chalker, Phys. Rev. Lett. 80, 2929 (1998b).
  * Zhitomirsky (2002) M. E. Zhitomirsky, Phys. Rev. Lett. 88, 057204 (2002).
  * Zhitomirsky (2008) M. E. Zhitomirsky, Phys. Rev. B 78, 094423 (2008).
  * Okumura  _et al._ (2010) S. Okumura, H. Kawamura, T. Okubo, and Y. Motome, Journal of the Physical Society of Japan 79, 114705 (2010).
  * Mohylna  _et al._ (2022) M. Mohylna, F. A. Gómez Albarracín, M. Žukovič, and H. D. Rosales, Phys. Rev. B 106, 224406 (2022).
  * Essafi  _et al._ (2016) K. Essafi, O. Benton, and L. D. C. Jaubert, Nature Communications 7, 10297 (2016).
  * Rosales  _et al._ (2023) H. D. Rosales, F. A. G. Albarracín, P. Pujol, and L. D. C. Jaubert, Phys. Rev. Lett. 130, 106703 (2023).
  * Mendels  _et al._ (2007) P. Mendels, F. Bert, M. A. de Vries, A. Olariu, A. Harrison, F. Duc, J. C. Trombe, J. S. Lord, A. Amato, and C. Baines, Phys. Rev. Lett. 98, 077204 (2007).
  * Olariu  _et al._ (2008) A. Olariu, P. Mendels, F. Bert, F. Duc, J. C. Trombe, M. A. de Vries, and A. Harrison, Phys. Rev. Lett. 100, 087202 (2008).
  * Fåk  _et al._ (2012) B. Fåk, E. Kermarrec, L. Messio, B. Bernu, C. Lhuillier, F. Bert, P. Mendels, B. Koteswararao, F. Bouquet, J. Ollivier, A. D. Hillier, A. Amato, R. H. Colman, and A. S. Wills, Phys. Rev. Lett. 109, 037208 (2012).
  * Skjærvø  _et al._ (2020) S. H. Skjærvø, C. H. Marrows, R. L. Stamps, and L. J. Heyderman, Nature 2, 13 (2020).
  * Rougemaille and Canals (2019) N. Rougemaille and B. Canals, The European Physical Journal B 92, 62 (2019).