# Scale dependant layer for
self-supervised nuclei encoding ††thanks: Citation: Peter Naylor, Yao-Hung
Hubert Tsai, Marick Laé and Makoto Yamada. Scale dependant layer for self-
supervised nuclei encoding. ArXiv 2022. DOI:000000/11111.

Peter Naylor
RIKEN AIP
Kyoto, Japan
<EMAIL_ADDRESS>Yao-Hung Hubert Tsai
Carnegie Mellon University
Pittsburgh, Pennsylvania, USA
<EMAIL_ADDRESS>
Marick Laé
Centre Henri Becquerel - Unicancer
Rouen, France
<EMAIL_ADDRESS>
Makoto Yamada
RIKEN AIP – Kyoto University
Kyoto, Japan
<EMAIL_ADDRESS>
###### Abstract

Recent developments in self-supervised learning give us the possibility to
further reduce human intervention in multi-step pipelines where the focus
evolves around particular objects of interest. In the present paper, the focus
lays in the nuclei in histopathology images. In particular we aim at
extracting cellular information in an unsupervised manner for a downstream
task. As nuclei present themselves in a variety of sizes, we propose a new
Scale-dependant convolutional layer to bypass scaling issues when resizing
nuclei. On three nuclei datasets, we benchmark the following methods:
handcrafted, pre-trained ResNet, supervised ResNet and self-supervised
features. We show that the proposed convolution layer boosts performance and
that this layer combined with Barlows-Twins allows for better nuclei encoding
compared to the supervised paradigm in the low sample setting and outperforms
all other proposed unsupervised methods. In addition, we extend the existing
TNBC dataset to incorporate nuclei class annotation in order to enrich and
publicly release a small sample setting dataset for nuclei segmentation and
classification.

##  1 Introduction

Histopathology is the study of diseased tissue under a microscope, these image
data correspond to tissue slides encompassing the tumour and the surrounding
tissue. Digital pathology emerged in 60’s and since, supported clinicians and
informatics pipelines for the analysis of histopathology samples. Prior to
2012, automatic algorithms and classical machine learning (in contrast to deep
learning) have left the field relatively untapped. Due to the inherent
complexity of histopathology specimens, most studies were usually highly
specific and performed on homogeneous small cohorts [25]. The advent of deep
learning, where automated algorithms reached unprecedented results in computer
vision, have fuelled investments and research in this field [28]. The field of
digital pathology topical has become topical through the ever growing number
of challenges, scientific papers and publicly available datasets [32, 3, 5,
14].

Histopathology studies can be divided into “basic” and “advanced” tasks [12].
Basic tasks range from tumour detection and grading to sub-type
classification, i.e. relatively known and formalized tasks. For these tasks
deep learning research is mostly oriented to help pathologists in their every
day work by simplifying workflows [45]. In contrast, advanced tasks range from
survival, mutation to treatment response prediction and go beyond the usual
reporting asked from pathologists. In order to leverage the power of deep
learning in the field of histopathology, it is essential to build explainable
and interpretable models that study biological elements of interest. At the
moment, there are two classes of algorithms: those that aim at predicting
clinical variables directly from the image data [9], and those which seek to
profile the images in terms of biologically meaningful features [40]. These
latter algorithms involve, typically, the segmentation of an object of
interest, like the nuclei. Then a quantitative description of their phenotypes
and context, i.e. density information, morphological properties and spatial
disposition. This step is followed by specimen classification based on the
previous extracted attributes. The introduction of this intermediate step is
the most logical way of imposing biological interpretability as the tissue
slides are described in terms of cellular and tissular phenotypes [40, 35]. In
Fig. 1 we show an example of such a pipeline.

Figure 1: Feature extraction pipeline. We first performed instance
segmentation, followed by feature extraction for each object. In the present
article, we focused on step 2.

This study fits in a larger framework presented in Fig. 1 and we focus on the
second step in the present article. In particular, we study the encoding of a
meaningful element in histopathology analysis, the nuclei. The nuclei is the
most important element, and is the most commonly studied element in the field
[40, 23, 25, 35, 5]. The nuclei class and status, as well as its abundance,
hold valuable information for the elaboration of diagnosis and treatment for
many diseases [46]. Therefore building a multi-step model where we boil down
the analysis through the nuclei is highly relevant. If successful, these
multi-step models allow for apriori, via the segmentation and quantification
of nuclei before sample prediction, and aposteriori analysis, by back-
propagating the prediction back to the nuclei. Such analysis can be a powerful
tool for assisting current and fuelling future biological and medical
research.

For the second step, there exist accurate nuclei detection algorithms that
have been extended to Whole Slide Imaging [15, 2], and thus the success of the
pipeline is to encode nuclei accurately. In the past decades, researchers have
developed useful manual features for classifying nuclei and use machine
learning algorithms such as the support vector machines (SVM) [11, 40].
However, designing manual features is time-consuming, error prone and sub-
optimal for a given task. An alternative approach is to use deep learning
models such as the residual networks [22] where end-to-end learning is adopted
and the feature extractor module is optimized according to a given task.
However, histopathology data, depending on the tissue and disease, presents a
fairly heterogeneous collection of cells, i.e., the size, morphology and
texture of each detected nuclei is different. More specifically, cell type
also depends on the context and the organ in which the cell is present. When
describing nuclei and visualizing them, we do not expect clearly
distinguishable clusters but rather a continuum between cell class due to the
wide range and variability found in their phenotypes.

Moreover, since only trained researchers/technicians can annotate nuclei
classes, it is challenging to create a large scaled dataset for nuclei
classification. One approach is to use machine learning algorithms to annotate
nuclei images [13, 2, 14], however, annotations are usually restricted to a
small number of classes and do not cover the whole spectrum of nuclei classes.
Indeed for every tissue type, nuclei class can be refined into many more
classes [13]. Thus, there exist a small number of publicly available accurate
annotated datasets for nuclei classification.

In this paper, to address the heterogeneity in nuclei size, we propose the
Scale Dependant dilation convolution. The proposed operation takes in input a
resized object and the original object size to allow for a meaningful and
comparable feature computation from different image scales. Thus this layer
allows different scales to be combined while retaining the advantages of a
homogeneous image size, highly valued in deep neural networks. Then to study
the small sample case, we extended the Triple Negative Breast Cancer (TNBC)
dataset [38] by adding cell type to the dataset. This is, currently, the only
hand segmented and annotated TNBC available dataset. Moreover, we propose a
self-supervised training framework for nuclei classification with a benchmark
of methods. Through experiments, we demonstrate that our proposed Scale
Dependant convolution layer achieves better results with the Barlows-Twins
(BT) compared to manual models, supervised and other state of the art self-
supervised models.

The contribution of this paper is summarized as follows:

  * •

We extend the TNBC dataset by adding cell type to the dataset. In particular,
$4,056$ nuclei are annotated in $9$ classes.

  * •

A scale dependant convolution layer that can be used to extract features from
objects at different scales.

  * •

We demonstrate the utility of the scale dependant layer by conducting a
benchmark on three datasets that shows that it has a positive impact the
performance. In particular, we compare manual, pre-trained, self-supervised
and supervised features.

##  2 Related work

In this section, we review existing datasets for nuclei classification,
feature extraction, and scale invariant methods, respectively. Then, we
discuss the potential problems in the current nuclei classification methods.

Annotated nuclei datasets: Due to its essential role in the automatic
interpretation of stained tissue sections, segmentation of nuclei has been
addressed by many authors with a variety of traditional approaches, see also
[25] for an extensive reviews. The aim of this field is to quantify by nuclei
class the tumour micro environment, and to use this information as a profiling
tool, diagnosis or in-depth analysis of tumour subtypes [1, 26] Today, deep
neural networks are considered the best tool for this task [38, 30, 14]. These
models, based on supervised learning, come hand in hand with manually
annotated datasets, and depending on the number of annotators, typically range
in the thousands to tens of thousands of annotated nuclei [14]. Lately,
datasets have reached hundreds of thousands of annotated nuclei [2, 14] with
the help of automated algorithms. They propose a data generation pipeline
where they leverage an already trained network to generate segmentation for
unseen data. The new data is then checked and iteratively refined through
semi-automatic and manual procedures. We summarize other existing datasets in
Table 1, in this work we extend the TNBC dataset by including cell annotation.

Dataset | Nuclei | Labels | Organs | Semi-
---|---|---|---|---
|  |  |  | automatic
TNBC [38] | 4,056 | ✓ | Breast | ✗
CPM-17 [47] | 7,570 | ✗ | Multiple | ✗
MoNuSeg [29] | 21,623 | ✗ | Multiple | ✗
CoNSep [15] | 24,319 | ✓ | Colon | ✗
MoNuSac [48] | 46,909 | ✓ | Multiple | ✗
PanNuke [13] | 189,744 | ✓ | Multiple | ✓
NuCLS [2] | 222,396 | ✓ | Breast | ✓
Lizard [14] | 495,179 | ✓ | Colon | ✓

Table 1: Nuclei segmentation datasets.

The current trend is to build models like the HoVer-Net [15] where
segmentation and cell class prediction is done in a single pass. The work
presented in the current article differs from these in two manners. Firstly,
we aim to build an unsupervised feature extraction pipeline to allow the
quantification of tissue that does not rely on hard nuclei labelling but
rather on a soft representation. Secondly, because the spectrum of nuclei
class is huge and dependant on the tissue and disease, we aim to quantify
every nuclei in a general manner to not restrict the downstream analysis to
fixed labels but rather to these soft representations that allow the whole
spectrum of nuclei classes to be represented.

Moreover, cell types prediction can be challenging. In Hematoxylin and Eosin
(H&E) staining, similar nuclei morphologies does not necessarily imply same
class. For instance, epithelial cells can appear small and dense like
lymphocytes. Glial cells found in brain tissue are also similar to
lymphocytes. Macrophages can be mistaken for other cells when they are
digesting and cell type is hard to distinguish for mitotic events, in these
situations context can shed information about the type.

Feature extraction: Image encoding or representation learning in its most
primal state can be a shallow classifier where the prediction is based on a
continuous hidden variable, such as the distance to the hyperplane. The
underlying hope is that this hidden variable acts as a compact representation
from the original high-dimensional data. Prior to deep learning, hand designed
features were commonly used in computer vision. In particular, in image
analysis pipelines where one had to manually design a feature extractor for an
underlying task, including SIFT [34] and SURF [4]. Now, the scientific
community has moved to deep networks to learn non-manual-designed features.
The first famous example was trained on ImageNet [10] and this dataset is
still used today for pre-training the networks. Indeed the large number of
classes forces the feature extractor to cope with a large heterogeneity of
images and labels making it general. Pre-training has mostly been used when
the number of samples is low, even when ImageNet and the target dataset differ
in a number of ways [42].

Training from scratch, i.e. end to end training, is a means to overcome the
biases of the pre-trained network. In particular, recent advances show that
self-supervised learned features perform on a par with supervised learned
features [6, 36]. We can categorize self-supervised learning approaches into
three groups: contrastive learning approaches [21, 6], predictive learning
approaches [20], and others [51, 18]. For the first two groups: contrastive
learning approaches aim to learn similar features for positively-paired
samples and dissimilar features for negatively-paired samples, and predictive
learning approaches aim to reconstruct a partial of inputs given the rest part
of the inputs. In this paper, we consider Barlows-Twins [51] (BT) method,
which considers to maximize cross-correlation matrix between augmented batch
of inputs, as our self-supervised learning approach, and it belongs to the
“others” category in self-supervised learning.

For histopathology data, self-supervised learning has been used in multi-step
approaches to classify large images as end-to-end learning is impractical. The
multi-step approach consists of encoding tiles followed by the classification
of the whole image [9, 31, 39]. Prior to self-supervised learning, the most
common approach was to use pre-trained networks on ImageNet to encode the
tiles [42]. To the best of our knowledge, only one other paper has studied the
encoding of nuclei in histopathology data [14], which uses contrastive
learning for nuclei representation. To account for the difference in size they
crop with a large window, and mask the surrounding during training. They
report the Cohen’s kappa agreement measure between trained pathologist and
their prediction, found via hierarchical clustering of the contrastive
representation. However no accuracy measure on the classification of nuclei is
reported.

Scale invariant neural network: The most straightforward manner to impose
invariance in a neural network is to randomly augment the input data. For
scale invariance, input images are randomly rescaled in order to render the
network invariant to the scale of the object to a certain degree. To address
the topic further, two approaches have been adopted. The first involves
building models inherently invariant to scaling [49, 27] or by using
invariance properties to constrain the parameters. For instance, multi-branch
networks have been proposed, where each branch has its own filter and scale,
leading to a model where different scales are handle and combined for the
final prediction [49]. Multi-scale pyramidal models with skip layers to allow
the combination of large and small scale information have been successful [44,
33] and used for general object detection tasks with the RCNN [43]. Moreover,
invariances can induce symmetries that are efficient methods to reduce the
number of parameters [27]. The second type of approach focuses on leveraging a
previously successful scale invariant model to become scale aware [16, 17].

Dilated convolutions, and more generally deformable convolutions [50, 8, 37,
19] allow a model to better adapt to the local geometry found in the dataset.
In particular, it can adapt itself by finding the best window, or scale for
image feature extraction. The work we present in this article uses
parametrized deformable convolution, and in contrast to previous work, does
not aim to learn the best spacing. Specifically, we see the dilated
convolution as a parametrize and input-dependant function. The aim is to
extract features while preserving the initial shape of the object by using
deformable convolutions.

##  3 Dataset

We introduce the datasets used in this paper. Moreover, we extend the TNBC
dataset by adding cell type annotation to the existing dataset.

TNBC: The Triple Negative Breast Cancer (TNBC) dataset published by the Curie
Institute consists of 50 annotated H&E stained histology images at $40\times$
magnification. In particular $4,056$ nuclei boundaries were annotated. This
dataset has been complemented and the nuclei types are now also publicly
available, see Fig. 2(a) for some samples. Each nuclei was annotated into one
of the $9$ classes. The nuclei state mitosis is not a true nuclei type per say
but a nuclei state. We decided on adding this notation as the localisation and
detection of mitosis is an important topic and is involved in quantifying
cancer growth. In addition, any nuclei type can undergo mitosis and can look
morphologically similar. The annotation is summarized in Table 2.

Slide number | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | Total
---|---|---|---|---|---|---|---|---|---|---|---|---
Number of patches | 7 | 3 | 5 | 8 | 4 | 3 | 3 | 4 | 6 | 4 | 3 | 50
Adipocyte | 0 | 1 | 0 | 32 | 0 | 0 | 0 | 0 | 5 | 1 | 0 | 39
Cancerous | 243 | 150 | 133 | 443 | 306 | 110 | 48 | 187 | 178 | 192 | 141 | 2131
Endothelial | 12 | 0 | 0 | 7 | 0 | 3 | 6 | 35 | 3 | 7 | 8 | 81
Epithelial | 22 | 0 | 0 | 0 | 0 | 0 | 34 | 0 | 0 | 0 | 0 | 56
Fibroblast | 136 | 8 | 95 | 92 | 105 | 85 | 155 | 5 | 1 | 9 | 97 | 788
Lympho/Plasmocyte | 43 | 29 | 58 | 20 | 8 | 6 | 307 | 212 | 103 | 105 | 6 | 897
Mitosis | 2 | 0 | 1 | 2 | 0 | 0 | 0 | 6 | 11 | 2 | 2 | 26
Myoepithelial | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 0 | 0 | 4
Necrosis | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1
Total | 458 | 188 | 287 | 596 | 419 | 204 | 555 | 445 | 301 | 316 | 254 | 5150

Table 2: Detailed account of nuclei type within each sample of the TNBC [38]
dataset.

The nuclei type annotations were added via the software CellCognition [23].
CellCognition is usually used for annotating fluorescent microscopy images and
can also be used for any image and object type annotation. With this tool, we
annotated cell types which were all checked by an expert pathologist.

CoNSep: The colorectal nuclear segmentation and phenotypes dataset (CoNSep)
published by the department of pathology at University Hospitals Coventry and
Warwickshire consists of 41 annotated H&E stained histology images at
$40\times$ magnification. In particular $24,909$ nuclei boundaries and nuclei
types were annotated into 7 classes, see Fig. 2(b) for some samples. As with
the original paper, we combine classes 3 and 4 into the epithelial class and
5,6 and 7 into the spindle-shaped class.

(a) TNBC

(b) ConSep

Figure 2: Dataset samples (in columns) with overlaid semantic segmentation.

PanNuke: The PanNuke [13] dataset was also published by the department of
pathology at University Hospitals Coventry and Warwickshire. It consists of
$189,744$ annotated nuclei in a semi-automated manner from multiple tissue
divided into 5 classes.

Dataset preparation and manual feature extraction: For each nucleus, we
extract a total of $68$ features. We measure the nucleus size, width, height,
elongation, circularity. We add color information by measuring the average and
standard deviation of the intensity of the nuclei on each color bandwidth and
its greyscale transformation. To include edge information as well as texture
information, we include local binary patterns (LBP) [41] and granulometric
features [7]. LBP are operators that extract edge information by sliding a
window along the image, this results into a distribution that we numerically
quantify with quantiles ranging from $10\%$ to $90\%$, moreover we apply this
operator with a radius equal to $1$ and $3$ pixels. We apply granulometric
features on the greyscale image at sizes ranging from $1$ to $5$. Finally, in
order to allow the incorporation of surrounding information, we apply the same
set of features to the dilated version of the nuclei. We use a dilation of
factor of $4$ and we do not extract the width, height, elongation and
circularity of the dilated nucleus.

Size information, shape, intensity as well as texture information is valuable
when differentiating between cell types. Lymphocyte tend to be smaller, darker
and have a color distribution that is relatively homogeneous, see Fig. 3(a).
Cancer nuclei are usually larger, irregularly shaped, with irregular
intensities and tend to have a lighter color, see Fig. 3(b). Adipocyte,
similarly to lymphocyte, tend to be small, darker and homogeneous. They differ
only in their shape and context, indeed, they are usually compressed by fat
and are much more likely to be elongated, see Fig. 3(c).

From this semantic segmentation dataset, we crop each connected component with
a $5$ pixel margin on each side and resize all nuclei to a fixed $32\times$
$32$ size.

(a) Cancerous

(b) Lymphocyte

(c) Adipocyte

Figure 3: Nucleus type example

##  4 Proposed Method

We will compare the previously described manual features to learned
representation. In particular, to a pre-trained ResNet on ImageNet, to a
benchmark of supervised models and self-supervised models, MoCo [21] and BT
[51].

###  4.1 Problem formulation

Let us denote an input image by ${\bm{X}}_{i}\in\mathbb{R}^{h_{i}\times
w_{i}\times 3}$ and the corresponding output class labels
$y\in\\{1,2,\ldots,L\\}$, where the image input size is sample dependant, i.e
the height $h_{i}$ and the width $w_{i}$ can be different and $L$ is the total
number of classes. For nuclei data, as the size of the nuclei are different,
this setup is reasonable. Then, we resize ${\bm{X}}_{i}$ into
$\widetilde{{\bm{X}}_{i}}\in\mathbb{R}^{32\times 32\times 3}$ and construct
the set of samples
$\\{(\widetilde{{\bm{X}}_{i}},{\bm{s}}_{i},y_{i})\\}_{i=1}^{n}$, where
${\bm{s}}_{i}=(h_{i},w_{i})^{\top}\in\mathbb{R}_{+}^{2}$ is the size
information of the original image ${\bm{X}}_{i}$.

The goal of this paper is to build meaningful nuclei encoding that represent
the wide heterogeneity found in histopathology data. We check the efficiency
of the encoding via two steps In the first, we encode the samples via one of
the proposed methods. Then, we apply a classification model on top of the
encoding and report two metrics. Specifically, we use relatively simple
classification models such as a single layer neural network and a nearest
neighbour algorithm.

###  4.2 Backbone model

The supervised and self-supervised model both rely on a backbone model. We
implement a modified and adapted version of the 34-layer ResNet for smaller
images, named SResNet. In particular we replace the initial $7\times 7$ and 64
feature convolutional block and max-pooling layer by a $3\times 3$ and 32
features convolutional layer with batch normalisation and ReLu activation. We
then stack three residual blocks consisting of $3$, $4$ and $3$ residual
layers each and $32$, $64$ and $128$ features respectively. We reduce the size
of the feature maps with a striding of two after the second and third residual
blocks. We then use an average pooling layer to reduce the spatial resolution
to $1$ to produce the image encoding. In the supervised setting, we apply a
final fully connected layer to produce a probability vector with respect to
the number of classes.

###  4.3 Dealing with resizing

Size injection: As we presume that size information is valuable and absent
from the backbone, we simply inject the original width and height of the input
image to the encoding in the latter layers of the network. Moreover, in order
to allow the backbone to adjust the representation, we add an extra fully
connected layer with batch normalization and ReLu. In particular the layer
prior to the final probability layer is the new encoding, see Fig. 4.

Figure 4: SResNet with size concatenation.

Adjustable dilation: The main issue with resizing the images prior to their
analysis happens in the first layer. Indeed, if the images are resized, we are
extracting and comparing features from different scales. However, resizing
allows for computational efficiently and batching of the samples. Therefore
all input are resized to a fixed $32\times 32$ size.

In the current backbone, the first convolutional layer extract and compares
features that ultimately have different scales. This issue is more important
for the first layers where the networks learn basic features, such as edges,
corners and color gradients. We oppose basic features to more abstract
features as they have been defined in other works [52], it has been
empirically shown that latter layers hold more abstract concepts. We decide to
use the original height and width to adjust the dilation in the first layer to
compensate for the different scales and allow for a fairer and more scale
invariant representation in the first layer. We do not need to adjust the
scale in the following layers as the inner representation should already be
invariant to the scale.

We give below the equation for our parametrized dilated convolution for
computing the resulting feature map $m$ at pixel $(i_{0},j_{0})$ for a one
channel input image:

$f_{m}^{sd}(I)[i_{0},j_{0}]=\sum_{i,j=-k_{w},-k_{h}}^{k_{w},k_{h}}I[i_{0}+i*d_{w}(I),j_{0}+j*d_{h}(I)]\times
W_{i,j},$

where $W$ is the learnable kernel with width $w_{k}$ and height $h_{k}$. For
the sake of clarity and conciseness, we assume the height and width of $W$ are
odd and measured from the center pixel $W_{0,0}$. For a $3\times 3$ kernel, we
have $w_{k}=h_{k}=1$. In particular we define $d_{a}(I)=\text{max}(\lfloor
u_{f}\times 32/a_{I}\rfloor,1)$ which computes the dilation factor over axis
$a$ (equal to the height or width axis), $a_{I}$ corresponds to the original
length of $I$ over axis $a$. $u_{f}$ corresponds to the up-scaling factor
applied to the image $I$ prior to the convolution, set to $3$. In larger
images, the up-scaling factors allow to reduce the dilation factor further.
Finally, max-pooling is applied to reduce the image back to the original size
of the feature map, we give below the whole function:

$F^{sd}(I)=\text{max-pooling}_{3}\circ f^{sd}\circ\text{Upsample}_{3}(I),$

where $\text{Upsample}_{3}$ up-scales by factor $3$ and $\text{max-
pooling}_{3}$ downsizes by a factor $3$ with a max-pooling operation.

We name this layer, SD-CL, for Scale Dependant Convolutional Layer. With no
loss of generality, we can also apply SD-CL to 3D objects.

Figure 5: SD-CL: Scale Dependant convolutional layer with adjustable dilation.
The kernel weights are shared.

###  4.4 Models

Pre-trained model:  We encode each nuclei with a pre-trained ResNet on
ImageNet. In particular, for each image, we resize it to $224\times 224$ and
extract the $2048$ encoding.

Baseline supervised method: As another baseline approach for the learning
paradigm we propose a supervised framework. We compare SResNet, with and
without size concatenation to the same models with the SD-CL.

Momentum Contrast (MoCo) [21]: During training we infer queries and match
these to a set of keys, that correspond to encoded samples. In particular, it
is assumed that there is a single key $k_{+}$ that matches a query $q$. We
minimize the following loss:

$\mathcal{L}_{q}=-\log\frac{\exp\left(q\cdot
k_{+}/\tau\right)}{\sum_{i=0}^{K}\exp\left(q\cdot k_{i}/\tau\right)},$

where $K$ denotes the number of negative samples and $\tau$ the temperature.
During training, a positive match corresponds to a given image and its
augmented version. Negative keys correspond to the other encoded samples.

Barlows-twins [51]:  Barlows-twins is a self-supervised method, where given a
model $f_{\theta}$ and set of random transformations (or data augmentation)
function $\mathcal{T}$ we minimize the following loss
$\mathcal{L}_{\mathcal{BT}}$:

$\mathcal{L}_{\mathcal{BT}}=\sum_{i}\left(1-\mathcal{C}_{ii}\right)^{2}+\lambda\quad\sum_{i}\sum_{j\neq
i}\mathcal{C}_{ij}{}^{2},$

where
$\mathcal{C}_{ij}=\frac{\sum_{b}z_{b,i}^{A}z_{b,j}^{B}}{\sqrt{\sum_{b}\left(z_{b,i}^{A}\right)^{2}}\sqrt{\sum_{b}\left(z_{b,j}^{B}\right)^{2}}}$
and $z^{A}$ and $z^{B}$ are random transformations of the same input feed to
the model $f_{\theta}$. $\lambda$ is a hyper-parameter to balance between
invariance and the redundancy reduction term. The underlying idea with
contrastive learning is that two transformed input images should have a
similar embedding in the feature space. It was shown in the original paper
that it is better when applying neural networks to add a projection head to
the backbone. Like the supervised paradigm, SResNet is used as the backbone
and we experiment with the SD-CL and size injection.

##  5 Experiments and results

In this section, we evaluate our proposed algorithms using our newly
constructed TNBC datasets and two other datasets, CoNSep and PanNuke.

###  5.1 Experimental details

Data splits: For all datasets, we split the dataset into a training and test
sets. For TNBC, we use the samples folds 1 and 9, for the test set in order to
have sufficient labelled samples in each class. Moreover we remove the minor
classes Myoepithelial and Necrosis, resulting in $3,259$ train samples and
$759$ test samples. For CoNSep, we use the split given by the authors that
allows us to constitute a dataset of $15,555$ train samples and $8,777$ test
samples. Similarly for PanNuke, we use the pre-defined folds and use fold 1
and 2 for training and fold 3 for testing. For this dataset we have $122,117$
train samples $67,627$ test samples. During model selection and depending if
the method requires tuning, we further split the training dataset into train
and validation set in order to perform model selection in an unbiased manner.

Training details: To refine the manually designed features we perform feature
selection in a similar fashion to the forward and backwards stepwise selection
[24]. In the forward method, we add features if this improves the negative
log-likelihood on the validation set. Similarly, in the backward method, we
remove features if this improves the negative log-likelihood minus some
heuristic scalar (set to $0.05$) on the validation set. Finally, we compare
the forward selection, the backward selection, the intersection and the union
of the selected features on the validation set and retain the best performing
set of features.

For the learning paradigms we use an Adam optimizer and optimize the learning
rate, weight decay and $\lambda$ for BT on the validation set. We use a batch
size of $128$ for TNBC and CoNSeP. For PanNuke we use a batch size of $512$.
We set the number of epochs to $100$ except for PanNuke where the number of
training iterations is reduced to $50$. We use the following set of
transformations, all applied with probability $p=0.5$: rotation, vertical and
horizontal flip, color jitter, resized crop and greyscale transformations. For
the rotation and resized crop we modify the injected height and width
accordingly. In particular, a rotation of an angle $\theta$ leads to a new set
of heights and widths defined as:

$\displaystyle h_{\theta}$
$\displaystyle=h.\text{cos}\theta+w.\text{sin}\theta,~{}~{}w_{\theta}=h.\text{sin}\theta+w.\text{cos}\theta.$

Metrics: We use an accuracy score based on the training of a one layer Softmax
classifier on top of the embedding given by the selected model, we name this
metric the linear accuracy metric. In addition, we use a nearest neighbour
classifier with $k=50$ on the given encoding, we name this the kNN accuracy
metric. We prefer to maximise the nearest neighbour score as this one is more
relevant as it shows that the embedding space holds semantic information. In
particular, we select the model that maximises the nearest neighbour
validation accuracy score.

###  5.2 Application to nuclei encoding

|  | TNBC | CoNSep | PanNuke
---|---|---|---|---
| $L$ | 7 | 4 | 5
| Training set size | $3,259$ | $15,555$ | $122,117$
| Test set size | $759$ | $8,777$ | $67,627$
| Accuracy metric | Linear | kNN | Linear | kNN | Linear | kNN
No training | Manual | $55.5\pm 3.7$ | $55.6\pm 0.0$ | $52.8\pm 10.7$ | $17.6\pm 15.2$ | $43.5\pm 5.8$ | $42.4\pm 0.0$
Pre-trained ResNet | $55.6\pm 0.0$ | $51.3\pm 0.0$ | $37.1\pm 1.1$ | $37.8\pm 0.0$ | $42.3\pm 0.1$ | $42.4\pm 0.0$
Pre-trained ResNet + Size | $55.6\pm 0.1$ | $20.0\pm 0.0$ | $37.1\pm 1.6$ | $37.9\pm 0.0$ | $42.3\pm 0.1$ | $17.1\pm 0.0$
Supervised | SRN | $57.3\pm 1.2$ | $52.6\pm 8.2$ | $73.9\pm 0.7$ | $71.3\pm 1.7$ | $71.1\pm 0.4$ | $64.1\pm 1.6$
SRN + Size | $58.1\pm 1.4$ | $39.0\pm 3.5$ | $74.1\pm 0.8$ | $58.6\pm 2.5$ | $\mathbf{73.1\pm 0.3}$ | $45.3\pm 2.2$
SRN + SD-CL | $\mathbf{58.8\pm 0.9}$ | $\mathbf{58.9\pm 2.3}$ | $\mathbf{74.4\pm 0.7}$ | $\mathbf{71.6\pm 1.5}$ | $72.2\pm 0.3$ | $\mathbf{64.9\pm 1.6}$
SRN + SD-CL + Size | $58.4\pm 0.9$ | $46.7\pm 3.4$ | $74.0\pm 0.4$ | $61.4\pm 2.7$ | $\mathbf{73.1\pm 0.4}$ | $47.0\pm 3.1$
Self-supervised | BT + SRN | $59.6\pm 1.3$ | $56.7\pm 1.4$ | $66.9\pm 0.6$ | $66.1\pm 0.5$ | $58.3\pm 0.3$ | $57.6\pm 0.3$
BT + SRN + Size | $63.3\pm 0.8$ | $63.7\pm 0.7$ | $68.3\pm 0.8$ | $65.8\pm 0.4$ | $\mathbf{59.9\pm 0.3}$ | $\mathbf{59.1\pm 0.2}$
BT + SRN + SD-CL | $61.8\pm 2.6$ | $60.4\pm 2.1$ | $67.3\pm 0.9$ | $\mathbf{66.7\pm 0.5}$ | $58.5\pm 0.7$ | $59.0\pm 0.5$
BT + SRN + SD-CL + Size | $\mathbf{64.3\pm 1.1}$ | $\mathbf{64.2\pm 1.4}$ | $\mathbf{68.4\pm 0.5}$ | $65.7\pm 0.5$ | $\mathbf{59.9\pm 0.3}$ | $59.0\pm 0.2$
MoCo + SRN | $49.9\pm 7.9$ | $53.5\pm 5.7$ | $36.0\pm 1.8$ | $10.8\pm 5.0$ | $35.4\pm 8.0$ | $37.0\pm 4.0$
MoCo + SRN + Size | $55.3\pm 0.3$ | $39.9\pm 10.7$ | $37.3\pm 0.8$ | $26.5\pm 3.9$ | $40.8\pm 1.8$ | $30.6\pm 4.0$
MoCo + SRN + SD-CL | $55.2\pm 4.4$ | $38.6\pm 8.0$ | $37.2\pm 1.5$ | $26.7\pm 3.7$ | $41.2\pm 1.5$ | $32.1\pm 3.9$
MoCo + SRN + SD-CL + Size | $54.7\pm 2.0$ | $38.3\pm 9.5$ | $37.5\pm 0.9$ | $25.8\pm 4.2$ | $39.5\pm 2.3$ | $31.6\pm 3.1$

Table 3: Comparison of different methods for nuclei encoding on TNBC, CoNSep
and PanNuke datasets. We bold the best score in its category, self-supervised
learning and supervised. Our proposed convolution layer is named SD-CL. SRN is
a modified 34-ResNet adapted for small images [22]. MoCo [21] and BT [51] have
not yet been applied to nuclei encoding in histopathology images. $L$ is the
number of classes, $n_{train}$ and $n_{test}$ are the number of samples in the
train and test sets.

Results We run our experiments on a High-performance computing cluster and
repeat each method $20$ times. We display the average accuracy and standard
deviations for the different methods in Table 3.

The manual designed features achieve scores at least as well as the pre-
trained ResNet except for the kNN accuracy on CoNSeP. We notice that injecting
the size to the pre-trained ResNet feature is detrimental for the encoding as
the kNN accuracy loses more than $20\%$ for TNBC and PanNuke. It is possible
that the scale of the added feature penalises the encoding, the scale would
not penalise the linear accuracy as the feature can be weight and the scales
can be learned and adapted to the situation. Generally speaking the models
where no training occur under-perform compared to the learning paradigms. We
can however imagine a situation, when the number of data points is very low,
that these features work reasonably well.

When the number of samples is low, i.e. for the TNBC dataset, the self-
supervised framework outperforms the supervised one with a relative difference
of $5.5\%$ for the linear accuracy and $5.3\%$ for the kNN accuracy. In
comparison, on the CoNSep and PanNuke dataset, the supervised methods
outperform the self-supervised method by $6.0\%$ and $13.2\%$ on the linear
accuracy and by $4.9\%$ and $5.8\%$ on the kNN accuracy. This behaviour is
expected as the supervised models need more data in order to reach a correct
representation of the nuclei.

Injecting the size information into the latter layers prior to the encoding
generally boosts the linear score and lowers the kNN score. Similarly to the
pre-trained ResNet this could simply be a question of scale and correct
weighting. Apart for the CoNSep data, the BT model is the only one where both
linear and kNN scores increase with the size injection. In particular, the
best performing self-supervised models are reached when the size is present,
except for CoNSep on kNN score.

For self-supervised nuclei encoding, BT should be prioritised over MoCo.
Indeed the simpler formulation and the difference in linear and kNN score
proves the effectiveness of BT over MoCo in these datasets. In addition, to
boost MoCo’s performance, we increased the batch size to $1024$ as well as
increased the learning rate range during the model selection phase.

Finally, the SD-CL achieves at least as well as the standard convolution
layer. For most datasets and most situations, i.e. supervised, self-supervised
and with/without size, the effect of the SD-CL layer is positive. The
difference in performance is more acute when the sample size is small.
Theoretically, it should allow features to be extracted better from the
images. However, with enough samples, we suspect that the model can learn
clues regarding the original scale of the image. As images are resized using a
bi-linear interpolation, smaller images will show smooth color gradients
whereas larger images will be sharper.

Weight visualisation:  In Fig. 6, we visualise the weights of the first layer
of SResNet and of SResNet + SDCL model. For visualisation purposes, we perform
the min-max normalisation on the whole set of kernels and rescale the range to
$[0,255]$ and plot the resulting RGB weights. From the figure, both sets of
weights are reasonable and conform with the literature [52], such as the
presence of a homogeneous colour or colour gradients. Expectedly, we notice
the presence of purple and pink filters matching the H&E staining.

(a) Standard convolution

(b) SD-CL

Figure 6: Visualisation of the weights learned weights for the supervised
model trained on PanNuke.

##  6 Conclusion

We compare hand-designed features, pre-trained models, supervised models and
self-supervised models on three nuclei type datasets that range from small to
large samples size. In this benchmark, we show the effectiveness of
differently produced encodings. The self-supervised learning Barlows-twins
model ahead in the low sample setting and the supervised learning ahead in the
large sample setting. Moreover, we presented a new convolutional layer for
extracting features dependant on the scale. In addition to accounting for the
scale, this layers impacts positively the performances and shows similar
weight introspection to the standard convolution layer. In the spirit of open
science and reproducibility, the cell type extension dataset for TNBC and the
code are made publicly available.

## Data and code availability

The TNBC segmentation dataset is available at
https://zenodo.org/record/1175282. The extended TNBC classification dataset is
available at https://zenodo.org/record/3552674. The CoNSeP segmentation and
classification dataset is available at
https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/. The PanNuke
segmentation and classification dataset is available at
https://warwick.ac.uk/fac/cross_fac/tia/data/pannuke. The code for MoCo used
the lightly package https://docs.lightly.ai/index.html and the code for BT was
inspired from https://github.com/yaohungt/Barlow-Twins-HSIC. In addition
methodological developments, for the sake of reproducible research and open
science, we make the code for all experiments publicly available in the
following Github repository
https://github.com/PeterJackNaylor/ScaleDependantCNN.

## Acknowledgement

We thank Dr. Thomas Walter for his support in the project and in particular
for the insightful discussions, help with the annotations and use of the
software. M.Y. was supported by MEXT KAKENHI 20H04243 and partly supported by
MEXT KAKENHI 21H04874. We are also thankful for the RAIDEN computing system
and its support team at the RIKEN AIP, which we used for our experiments.

## References

  * [1] David Ahmedt-Aristizabal, Mohammad Ali Armin, Simon Denman, Clinton Fookes, and Lars Petersson.  A survey on graph-based deep learning for computational histopathology.  Computerized Medical Imaging and Graphics, page 102027, 2021.
  * [2] Mohamed Amgad, Lamees A Atteya, Hagar Hussein, Kareem Hosny Mohammed, Ehab Hafiz, Maha AT Elsebaie, Ahmed M Alhusseiny, Mohamed Atef AlMoslemany, Abdelmagid M Elmatboly, Philip A Pappalardo, et al.  Nucls: A scalable crowdsourcing, deep learning approach and dataset for nucleus classification, localization and segmentation.  arXiv preprint arXiv:2102.09099, 2021.
  * [3] Guilherme Aresta et al.  Bach: Grand challenge on breast cancer histology images.  Medical image analysis, 2019.
  * [4] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In European conference on computer vision, pages 404–417. Springer, 2006.
  * [5] Juan C Caicedo, Allen Goodman, Kyle W Karhohs, Beth A Cimini, Jeanelle Ackerman, Marzieh Haghighi, CherKeng Heng, Tim Becker, Minh Doan, Claire McQuin, et al.  Nucleus segmentation across imaging experiments: the 2018 data science bowl.  Nature methods, 16(12):1247–1253, 2019.
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [7] Yidong Chen and Edward R Dougherty.  Gray-scale morphological granulometric texture classification.  Optical Engineering, 33(8):2713–2722, 1994.
  * [8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  * [9] Olivier Dehaene, Axel Camara, Olivier Moindrot, Axel de Lavergne, and Pierre Courtiol.  Self-supervision closes the gap between weak and strong supervision in histology.  arXiv preprint arXiv:2012.03583, 2020.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [11] Harris Drucker, Christopher J Burges, Linda Kaufman, Alex Smola, and Vladimir Vapnik.  Support vector regression machines.  Advances in neural information processing systems, 9, 1996.
  * [12] Amelie Echle, Niklas Timon Rindtorff, Titus Josef Brinker, Tom Luedde, Alexander Thomas Pearson, and Jakob Nikolas Kather.  Deep learning in cancer pathology: a new generation of clinical biomarkers.  British journal of cancer, 124(4):686–696, 2021.
  * [13] Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot.  Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification.  In European congress on digital pathology, pages 11–19. Springer, 2019.
  * [14] Simon Graham, Mostafa Jahanifar, Ayesha Azam, Mohammed Nimir, Yee-Wah Tsang, Katherine Dodd, Emily Hero, Harvir Sahota, Atisha Tank, Ksenija Benes, et al.  Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 684–693, 2021.
  * [15] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot.  Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.  Medical Image Analysis, 58:101563, 2019.
  * [16] Mara Graziani, Thomas Lompech, Henning Müller, Adrien Depeursinge, and Vincent Andrearczyk.  Interpretable cnn pruning for preserving scale-covariant features in medical imaging.  In Interpretable and Annotation-Efficient Learning for Medical Image Computing, pages 23–32. Springer, 2020.
  * [17] Mara Graziani, Thomas Lompech, Henning Müller, Adrien Depeursinge, and Vincent Andrearczyk.  On the scale invariance in state of the art cnns trained on imagenet.  Machine Learning and Knowledge Extraction, 3(2):374–391, 2021.
  * [18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [19] Ismail Khalfaoui Hassani et al.  Dilated convolution with learnable spacings.  arXiv:2112.03740, 2021.
  * [20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [23] Michael Held, Michael HA Schmitz, Bernd Fischer, Thomas Walter, Beate Neumann, Michael H Olma, Matthias Peter, Jan Ellenberg, and Daniel W Gerlich.  Cellcognition: time-resolved phenotype annotation in high-throughput live cell imaging.  Nature methods, 7(9):747–754, 2010.
  * [24] Ronald R Hocking.  A biometrics invited paper. the analysis and selection of variables in linear regression.  Biometrics, pages 1–49, 1976.
  * [25] Humayun Irshad, Antoine Veillard, Ludovic Roux, and Daniel Racoceanu.  Methods for nuclei detection, segmentation, and classification in digital histopathology: a review—current status and future potential.  IEEE reviews in biomedical engineering, 7:97–114, 2013.
  * [26] Guillaume Jaume, Pushpak Pati, Behzad Bozorgtabar, Antonio Foncubierta, Anna Maria Anniciello, Florinda Feroce, Tilman Rau, Jean-Philippe Thiran, Maria Gabrani, and Orcun Goksel.  Quantifying explainers of graph neural networks in computational pathology.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8106–8116, 2021.
  * [27] Angjoo Kanazawa, Abhishek Sharma, and David Jacobs.  Locally scale-invariant convolutional neural networks.  arXiv preprint arXiv:1412.5104, 2014.
  * [28] C Kaushal, S Bhat, D Koundal, and A Singla.  Recent trends in computer assisted diagnosis (cad) system for breast cancer diagnosis using histopathological images.  IRBM, 2019.
  * [29] Neeraj Kumar, Ruchika Verma, Deepak Anand, Yanning Zhou, Omer Fahri Onder, Efstratios Tsougenis, Hao Chen, Pheng-Ann Heng, Jiahui Li, Zhiqiang Hu, et al.  A multi-organ nucleus segmentation challenge.  IEEE transactions on medical imaging, 39(5):1380–1391, 2019.
  * [30] Andrew Lagree, Majidreza Mohebpour, Nicholas Meti, Khadijeh Saednia, Fang-I Lu, Elzbieta Slodkowska, Sonal Gandhi, Eileen Rakovitch, Alex Shenfield, Ali Sadeghi-Naini, et al.  A review and comparison of breast tumor cell nuclei segmentation performances using deep convolutional neural networks.  Scientific Reports, 11(1):1–11, 2021.
  * [31] Tristan Lazard, Guillaume Bataillon, Peter Naylor, Tatiana Popova, François-Clément Bidard, Dominique Stoppa-Lyonnet, Marc-Henri Stern, Etienne Decencière, Thomas Walter, and Anne Vincent Salomon.  Deep learning identifies new morphological patterns of homologous recombination deficiency in luminal breast cancers from whole slide images.  bioRxiv, 2021.
  * [32] Geert Litjens et al.  1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset.  GigaScience, 2018.
  * [33] Dan Liu, Dawei Du, Libo Zhang, Tiejian Luo, Yanjun Wu, Feiyue Huang, and Siwei Lyu.  Scale invariant fully convolutional network: detecting hands efficiently.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4344–4351, 2019.
  * [34] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [35] Claire McQuin, Allen Goodman, Vasiliy Chernyshev, Lee Kamentsky, Beth A Cimini, Kyle W Karhohs, Minh Doan, Liya Ding, Susanne M Rafelski, Derek Thirstrup, et al.  Cellprofiler 3.0: Next-generation image processing for biology.  PLoS biology, 16(7):e2005970, 2018.
  * [36] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [37] Alhassan Mumuni and Fuseini Mumuni.  Cnn architectures for geometric transformation-invariant feature representation in computer vision: a review.  SN Computer Science, 2(5):1–23, 2021.
  * [38] Peter Naylor, Marick Laé, Fabien Reyal, and Thomas Walter.  Segmentation of nuclei in histopathology images by deep regression of the distance map.  IEEE transactions on medical imaging, 38(2):448–459, 2018.
  * [39] Peter Naylor, Tristan Lazard, Guillaume Bataillon, Marick Laé, Anne-Sophie Hamy, Fabien Reyal, and Thomas Walter.  Prediction of treatment response in triple negative breast cancer from whole slide images.  Frontiers in Signal Processing, page 26, 2022.
  * [40] Beate Neumann et al.  Phenotypic profiling of the human genome by time-lapse microscopy reveals cell division genes.  Nature, 2010.
  * [41] Timo Ojala, Matti Pietikainen, and Topi Maenpaa.  Multiresolution gray-scale and rotation invariant texture classification with local binary patterns.  IEEE Transactions on pattern analysis and machine intelligence, 24(7):971–987, 2002.
  * [42] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio.  Transfusion: Understanding transfer learning for medical imaging.  Advances in neural information processing systems, 32, 2019.
  * [43] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [45] Nikita Shvetsov, Morten Grønnesby, Edvard Pedersen, Kajsa Møllersen, Lill-Tove Rasmussen Busund, Ruth Schwienbacher, Lars Ailo Bongo, and Thomas K Kilvaer.  A pragmatic machine learning approach to quantify tumor infiltrating lymphocytes in whole slide images.  arXiv preprint arXiv:2202.06590, 2022.
  * [46] W Fraser Symmans, Florentia Peintinger, Christos Hatzis, Radhika Rajan, Henry Kuerer, Vicente Valero, Lina Assad, Anna Poniecka, Bryan Hennessy, Marjorie Green, et al.  Measurement of residual breast cancer burden to predict survival after neoadjuvant chemotherapy.  Journal of Clinical Oncology, 25(28):4414–4422, 2007.
  * [47] Kurc Tahsin, Zhao Tianhao, Gupta Rajarsi, Kalpathy-Cramer Jayashree, Farahani Keyvan, and Saltz Joel.  MICCAI 2018 Workshop and Challenges in Computational Precision Medicine, 2018.
  * [48] Ruchika Verma, Neeraj Kumar, Abhijeet Patil, Nikhil Cherian Kurian, Swapnil Rane, and Amit Sethi.  Multi-organ nuclei segmentation and classification challenge 2020.  IEEE transactions on medical imaging, 39(1380-1391):8, 2020.
  * [49] Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang.  Scale-invariant convolutional neural networks.  arXiv preprint arXiv:1411.6369, 2014.
  * [50] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [51] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [52] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.


# The dark energy phenomenon from backreacation effect

Yan-Hong Yao<EMAIL_ADDRESS>Xin-He Meng<EMAIL_ADDRESS>Department of Physics, Nankai University, Tianjin 300071, China

###### Abstract

In this paper, we interpret the dark energy phenomenon as an averaged effect
caused by small scale inhomogeneities of the universe with the use of the
spatial averaged approach of Buchert. Two models are considered here, one of
which assumes that the backreaction term ${\cal Q}_{\cal D}$ and the averaged
spatial Ricci scalar $\left\langle{\cal R}\right\rangle_{\cal D}$ obey the
scaling laws of the volume scale factor $a_{\cal D}$ at adequately late times,
and the other one adopts the ansatz that the backreaction term ${\cal Q}_{\cal
D}$ is a constant in the recent universe. Thanks to the effective geometry
introduced by Larena et. al. in their previous work, we confront these two
backreaction models with latest type Ia supernova and Hubble parameter
observations, coming out with the results that the constant backreaction model
is slightly favoured over the other model, and within $1\sigma$ confidence
interval of the parameter $n$ in the scaling backreaction model, $\mid{\cal
Q}_{\cal D}\mid$ decreases with the increase of the volume scale factor
$a_{\cal D}$ at adequately late times. Also, the numerical simulation results
show that the constant backreaction model predicts a smaller expansion rate
and decelerated expansion rate than the other model does at redshifts higher
than about 1, and both backreaction terms begin to accelerate the universe at
a redshift around 0.6. In addition, by confronting the standard cosmological
model against the same datasets, we find that the effective geometry tends to
push the constraint toward a smaller cosmological constant term.

##  I Introduction

According to recent observations of type Ia supernovae, the universe is in a
state of accelerated expansion riess1998 ; perlmutter1999measurements . The
simplest scenario to account for these observations is a positive cosmological
constant in Einstein’s equations (The most well known cosmology model
including such constant is the so called Lambda cold dark matter $(\Lambda
CDM)$ model.), which is assumed to be an effect of quantum vacuum
fluctuations. However, because of the huge discrepancy between the theoretical
expected value and the observed one, other alternative scenarios have been
proposed, including scalar field models such as
quintessenceCaldwell1998Cosmological , phantomCaldwell1999A ,
dilatonicPiazza2004Dilatonic , tachyonPadmanabhan2002Accelerated and
quintomBo2006Oscillating  etc. and modified gravity models such as braneworlds
maartens2010brane , scalar-tensor gravity esposito2001scalar , higher-order
gravitational theories capozziello2005reconciling ; das2006curvature . Since
the so called fitting problem that how well is our universe described by a
standard Friedmann-Lemaître-Robertson-Walker(FLRW) model is not solved yet,
recently a third alternative has been considered to explain the dark energy
phenomenon as an averaged effect caused by small scale inhomogeneities of the
universe rasanen2004dark ; kolb2006cosmic .

In order to consider cosmology model without assuming a FLRW background, it is
necessary to answer a longstanding question that how to average a general
inhomogeneous model. To date the macroscopic gravity (MG) approach
zalaletdinov1992averaging ; zalaletdinov1997averaging ;
zalaletdinov1993towards ; mars1997space  is probably the most well known
attempt at averaging in space-time. Although it is the only approach that
gives a prescription for the correlation functions which emerge in an
averaging of the Einstein’s field equations, so far it required a number of
assumptions about the correlation functions which make the theory less
convictive. Therefore, in this paper we adopt another averaged approach which
is put forward by Buchert buchert2000average ; buchert2001average , in despite
of its foliation dependent nature, such approach is quite simple and hence
becomes the most well studied theoretical framework of averaged models. Since
the averaged field equations in such approach do not form a closed set, one
needs to make some assumptions about the backreaction term appeared in the
averaged equations. In buchert2006correspondence , by taking the assumption
that the backreaction term ${\cal Q}_{\cal D}$ and the averaged spatial Ricci
scalar $\left\langle{\cal R}\right\rangle_{\cal D}$ obey the scaling laws of
the volume scale factor $a_{\cal D}$, Buchert proposes a simple backreaction
model. To confront such model with observations, Larena et. al. present the
effective geometry with the introduction of a template metric that is only
compatible with homogeneity and isotropy on large scales of FLRW cosmology
instead of on all scales larena2009testing . As was pointed out by Larena et.
al., the scaling solution cannot be expected to fully represent the realistic
backreaction effect throughout the whole history of the universe since we
expect that the realistic backreaction term will change considerably at
redshift $z_{\cal D}\sim 10$. However, since we only use the datasets of type
Ia supernova and observational Hubble parameter in this paper, we merely
concern the behavior of the backreaction term at adequately late times, i.e.
$z_{\cal D}\lesssim O(1)$, which means that although we assume ${\cal Q}_{\cal
D}$ obeys scaling laws of $a_{\cal D}$ in such redshift range , it can behave
very differently at higher redshifts, particularly, such term encounters rapid
change when $z_{\cal D}\sim 10$ because of the structure formation effects,
and becomes negligible when $z_{\cal D}\gtrsim 1000$, which is reasonable
because of the consistence between perturbation theory predictions and CMB
observations. Nevertheless, we still doubt that the scaling solution is a
prime description of the late-time backreaction term, so we propose another
parameterization of ${\cal Q}_{\cal D}$ by simply setting it as a constant at
late times, and it turn out that such model is preferred by observations. We
also point out that such model is similar to the $o-\Lambda CDM$ model
(i.e.the $\Lambda CDM$ model added by a non-zero spatial curvature
$\Omega_{k}$) except its geometry background, to understand the effects coming
from the effective geometry, we confront the $o-\Lambda CDM$ model with the
same datasets and the fitting results suggest that such geometry background
tends to push the constraint toward a smaller cosmological constant term.

The paper is organized as follows. In Section II, the spatial averaged
approach of Buchert is demonstrated with presentation of the averaged
equations for the volume scale factor $a_{\cal D}$. In Section III, we
introduce the template metric, which is a necessary tool to test the
theoretical preditions with observations, and computation of observables. In
Section IV, we apply a simple likelihood analysis of two backreaction models
by confronting them with latest type Ia supernova and Hubble parameter
observations. After analysis of the results in Section IV, we summarize our
results in the last section.

##  II The backreaction models

In buchert2000average , Buchert considers a universe filled with irrotational
dust with energy density $\varrho$. By foliating space-time with the use of
Arnowitt-Deser-Misner(ADM) procedure and defining an averaging operator that
acts on any spatial scalar $\Psi$ function as

$\left\langle\Psi\right\rangle_{\cal D}:=\frac{1}{V_{{\cal D}}}\int_{{\cal
D}}\Psi Jd^{3}X,$ (1)

where $V_{{\cal D}}:=\int_{{\cal D}}Jd^{3}X$ is the domain’s volume, he
obtains two averaged equations here we need, the averaged Raychaudhuri
equation

$3\frac{{\ddot{a}}_{\cal D}}{a_{\cal D}}+4\pi
G\left\langle\varrho\right\rangle_{\cal D}={{\cal Q}}_{\cal D},$ (2)

and the averaged Hamiltonian constraint

$3\left(\frac{{\dot{a}}_{\cal D}}{a_{\cal D}}\right)^{2}-8\pi
G\left\langle\varrho\right\rangle_{\cal D}=-\frac{\left\langle{\cal
R}\right\rangle_{\cal D}+{{\cal Q}}_{\cal D}}{2}.$ (3)

In these two equations, $a_{\cal D}(t)=\left(\frac{V_{\cal D}(t)}{V_{{{\cal
D}_{\rm\bf 0}}}}\right)^{1/3}$ is the volume scale factor, where $V_{{{\cal
D}_{\rm\bf 0}}}=|{{{\cal D}_{\rm\bf 0}}}|$ denotes the present value of the
volume, and ${\cal Q}_{\cal D}$, $\left\langle{\cal R}\right\rangle_{\cal D}$
represent the backreaction term and the averaged spatial Ricci scalar
respectively, which are related by the following integrability condition

$\frac{1}{a_{\cal D}^{6}}\partial_{t}\left(\,{{\cal Q}}_{\cal D}\,a_{\cal
D}^{6}\,\right)\;+\;\frac{1}{a_{\cal
D}^{2}}\;\partial_{t}\left(\,\left\langle{\cal R}\right\rangle_{\cal D}a_{\cal
D}^{2}\,\right)\,=0\;.$ (4)

One can then obtain a specific backreaction model with an extra ansatz about
the form of ${\cal Q}_{\cal D}$ and $\left\langle{\cal R}\right\rangle_{\cal
D}$. A popular choice is to assume that

${\cal Q}_{{\cal D}}={\cal Q}_{{{\cal D}_{\rm\bf 0}}}a_{{\cal
D}}^{p};\left\langle{\cal R}\right\rangle_{\cal D}=\left\langle{\cal
R}\right\rangle_{{\cal D}_{\rm\bf 0}}a_{{\cal D}}^{n}$ (5)

where $n$ and $p$ are real numbers, while ${\cal Q}_{{{\cal D}_{\rm\bf 0}}}$
and $\left\langle{\cal R}\right\rangle_{{\cal D}_{\rm\bf 0}}$ represent the
present value of the backreaction term and the averaged spatial Ricci scalar
respectively. There are two types of solutions found in
buchert2006correspondence . The first type, with $n=-2$ and $p=-6$, is less
important since at late times it corresponds to a quasi-Friedmannian model in
which the backreaction effect can be neglected. The second type, which demands
$n=p$, has the explicit expression as:

$\displaystyle\left\langle{\cal R}\right\rangle_{\cal D}$ $\displaystyle=$
$\displaystyle\left\langle{\cal R}\right\rangle_{{\cal D}_{\rm\bf 0}}a_{{\cal
D}}^{n},$ (6) $\displaystyle{\cal Q}_{{\cal D}}$ $\displaystyle=$
$\displaystyle-\frac{n+2}{n+6}\left\langle{\cal R}\right\rangle_{{\cal
D}_{\rm\bf 0}}a_{{\cal D}}^{n}.$ (7)

As mentioned above, we only assume such parameterization of the backreaction
term to be valid in the recent universe.

By introducing the following dimensionless parameters:

$\displaystyle\Omega_{m}^{{\cal D}}:$ $\displaystyle=$
$\displaystyle\frac{8\pi G}{3H_{{\cal D}}^{2}}\langle\varrho\rangle_{\cal D},$
(8) $\displaystyle\Omega_{{\cal R}}^{{\cal D}}:$ $\displaystyle=$
$\displaystyle-\frac{\left\langle{\cal R}\right\rangle_{\cal D}}{6H_{{\cal
D}}^{2}},$ (9) $\displaystyle\Omega_{{\cal Q}}^{{\cal D}}:$ $\displaystyle=$
$\displaystyle-\frac{{{\cal Q}}_{{\cal D}}}{6H_{{\cal D}}^{2}},$ (10)
$\displaystyle\Omega_{X}^{\cal D}:$ $\displaystyle=$
$\displaystyle\Omega_{{\cal R}}^{{\cal D}}+\Omega_{{\cal Q}}^{{\cal D}},$ (11)

one can express the volume Hubble parameter $H_{\cal D}:={\dot{a}}_{\cal
D}/a_{\cal D}$ and the volume deceleration parameter $q^{\cal
D}:=-\frac{{\ddot{a}}_{\cal D}}{a_{\cal D}}\frac{1}{H_{{\cal D}_{\rm\bf
0}}^{2}}$ as:

$\displaystyle H_{{\cal D}}(a_{{\cal D}})$ $\displaystyle=$ $\displaystyle
H_{{{\cal D}_{\rm\bf 0}}}\left(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}}a_{{\cal
D}}^{-3}+\Omega_{X}^{{{\cal D}_{\rm\bf 0}}}a_{{\cal
D}}^{n}\right)^{\frac{1}{2}},$ (12) $\displaystyle q^{\cal D}(a_{{\cal D}})$
$\displaystyle=$ $\displaystyle\frac{1}{2}\Omega_{m}^{{{\cal D}_{\rm\bf
0}}}a_{{\cal D}}^{-3}-\frac{1}{2}(n+2)\Omega_{X}^{{{\cal D}_{\rm\bf
0}}}a_{{\cal D}}^{n}.$ (13)

In this paper, we propose another backreaction model with the assumption that
the backreaction term is a constant at late times of the universe, which
means, by using the integrability condition, ${\cal Q}_{\cal D}$ and
$\left\langle{\cal R}\right\rangle_{\cal D}$ have the following expression:

$\displaystyle\left\langle{\cal R}\right\rangle_{\cal D}$ $\displaystyle=$
$\displaystyle-3{\cal Q}_{{{\cal D}_{\rm\bf 0}}}+(3{\cal Q}_{{{\cal D}_{\rm\bf
0}}}+\left\langle{\cal R}\right\rangle_{{\cal D}_{\rm\bf 0}})a_{{\cal
D}}^{-2},$ (14) $\displaystyle{\cal Q}_{{\cal D}}$ $\displaystyle=$
$\displaystyle{\cal Q}_{{{\cal D}_{\rm\bf 0}}},$ (15)

from which one can obtain the volume Hubble parameter $H_{{\cal D}}$ and the
volume deceleration parameter $q^{\cal D}$ in this backreaction model as
follow with the use of the averaged equations

$\displaystyle H_{{\cal D}}(a_{{\cal D}})$ $\displaystyle=$ $\displaystyle
H_{{{\cal D}_{\rm\bf 0}}}(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}}a_{{\cal
D}}^{-3}-2\Omega_{{\cal Q}}^{{{\cal D}_{\rm\bf 0}}}+(\Omega_{{\cal R}}^{{{\cal
D}_{\rm\bf 0}}}+3\Omega_{{\cal Q}}^{{{\cal D}_{\rm\bf 0}}})a_{{\cal
D}}^{-2})^{\frac{1}{2}},$ (16) $\displaystyle q^{\cal D}(a_{{\cal D}})$
$\displaystyle=$ $\displaystyle\frac{1}{2}\Omega_{m}^{{{\cal D}_{\rm\bf
0}}}a_{{\cal D}}^{-3}+2\Omega_{{\cal Q}}^{{{\cal D}_{\rm\bf 0}}}.$ (17)

##  III Effective geometry

###  III.1 The template metric

In larena2009testing , a template metric was proposed by Larena et. al. as
follows,

${}^{4}{\bf g}^{\cal D}=-dt^{2}+L_{{H_{\rm\bf 0}}}^{2}\,a_{\cal
D}^{2}\gamma^{\cal D}_{ij}\,dX^{i}\otimes dX^{j}\;\;,$ (18)

where $L_{{H_{\rm\bf 0}}}=1/H_{{{\cal D}_{\rm\bf 0}}}$ is the present size of
the horizon introduced so that the coordinate distance is dimensionless, and
the domain-dependent effective three-metric reads:

$\gamma^{\cal D}_{ij}\,dX^{i}\otimes dX^{j}=\frac{dr^{2}}{1-\kappa_{{\cal
D}}(t)r^{2}}+r^{2}d\Omega^{2}$ (19)

with $d\Omega^{2}=d\theta^{2}+\sin^{2}(\theta)d\phi^{2}$, this effective
three-template metric is identical to the spatial part of a FLRW metric at any
given time, but its scalar curvature $\kappa_{{\cal D}}$ can vary from time to
time. As was pointed out by Larena et. al., $\kappa_{{\cal D}}$ cannot be
arbitrary, more precisely, they argue that it should be related to the true
averaged scalar curvature $\left\langle{\cal R}\right\rangle_{\cal D}$ in the
way that

$\left\langle{\cal R}\right\rangle_{\cal D}=\frac{\kappa_{{\cal
D}}(t)|\left\langle{\cal R}\right\rangle_{{\cal D}_{\rm\bf 0}}|a_{{{\cal
D}_{\rm\bf 0}}}^{2}}{a_{{\cal D}}^{2}(t)}$ (20)

, which is taken as one of the assumptions for two models considered in this
paper.

###  III.2 Computation of observables

The computation of effective distances along the light cone defined by the
template metric is very different from that of distances in FLRW models.
Firstly, let us introduce an effective redshift $z_{{\cal D}}$ defined by

$1+z_{{\cal D}}:=\frac{(g_{ab}k^{a}u^{b})_{S}}{(g_{ab}k^{a}u^{b})_{O}}\mbox{
,}$ (21)

where the letters O and S denote the evaluation of the quantities at the
observer and at the source respectively, $g_{ab}$ in this expression
represents the template metric, while $u^{a}$ is the four-velocity of the dust
which satisfies $u^{a}u_{a}=-1$, $k^{a}$ the wave vector of a light ray
travelling from the source S towards the observer O with the restrictions
$k^{a}k_{a}=0$. Then, by normalizing this wave vector such that
$(k^{a}u_{a})_{O}=-1$ and introducing the scaled vector $\hat{k}^{a}=a_{{\cal
D}}^{2}k^{a}$, we have the following equation:

$1+z_{{\cal D}}=(a_{{\cal D}}^{-1}\hat{k}^{0})_{S}\mbox{ ,}$ (22)

with $\hat{k}^{0}$ obeying the null geodesics equation
$k^{a}\nabla_{a}k^{b}=0$ which leads to

$\frac{1}{\hat{k}^{0}}\frac{d\hat{k}^{0}}{da_{{\cal D}}}=-\frac{r^{2}(a_{{\cal
D}})}{2(1-\kappa_{{\cal D}}(a_{{\cal D}})r^{2}(a_{{\cal
D}}))}\frac{d\kappa_{{\cal D}}(a_{{\cal D}})}{da_{{\cal D}}}\mbox{ .}$ (23)

As usual, the coordinate distance can be derived from the equation of radial
null geodesics:

$\frac{dr}{da_{{\cal D}}}=-\frac{H_{{{\cal D}_{\rm\bf 0}}}}{a_{{\cal
D}}^{2}H_{{\cal D}}(a_{{\cal D}})}\sqrt{1-\kappa_{{\cal D}}(a_{{\cal
D}})r^{2}}$ (24)

Solving these two equations with the initial condition
$\hat{k}^{0}(0)=1,r(0)=0$ and then plugging $\hat{k}^{0}(a_{\cal D})$ into Eq.
(22), one finds the relation between the redshift and the scale factor. With
these results, we can determine the volume Hubble parameter $H_{{\cal
D}}(z_{{\cal D}})$ and the luminosity distance $d_{L}(z_{{\cal D}})$ of the
sources defined by the following formula

$\displaystyle d_{L}(z_{{\cal D}})$ $\displaystyle=$
$\displaystyle\frac{1}{H_{{{\cal D}_{\rm\bf 0}}}}(1+z_{{\cal D}})^{2}a_{{\cal
D}}(z_{{\cal D}})r(z_{{\cal D}}).$ (25)

Having computed these two observables , it is then possible to compare the
backreaction model predictions with type Ia supernova and Hubble parameter
observations.

##  IV Constraints from supernovae data and OHD

In this section, we perform a simple likelihood analysis on the free
parameters of two backreaction model mentioned above with the combination of
datasets from type Ia supernova and Hubble parameter observations.

The best-fit values of the model parameters $(\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)$ (Here $o$ represents $n$ in the case of scaling backreaction model
and $\Omega_{{\cal R}}^{{{\cal D}_{\rm\bf 0}}}$ in the case of constant
backreaction model respectively.) from the recently released
Union2.1Suzuki2012The  compilation with 580 data points are determined by
minimizing

$\chi_{SNIa}^{2}(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)=R-\frac{S^{2}}{T}$ (26)

here $R$, $S$ and $T$ are defined as

$\displaystyle R$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{580}\frac{\left(5\log_{10}\left[H_{{{\cal D}_{\rm\bf
0}}}d_{L}({z_{{\cal D}}}_{i})\right]-\mu_{obs}({z_{{\cal
D}}}_{i})\right)^{2}}{\sigma_{\mu}^{2}({z_{{\cal D}}}_{i})},$ (27)
$\displaystyle S$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{580}\frac{\left(5\log_{10}\left[H_{{{\cal D}_{\rm\bf
0}}}d_{L}({z_{{\cal D}}}_{i})\right]-\mu_{obs}({z_{{\cal
D}}}_{i})\right)}{\sigma_{\mu}^{2}({z_{{\cal D}}}_{i})},$ (28) $\displaystyle
T$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{580}\frac{1}{\sigma_{\mu}^{2}({z_{{\cal D}}}_{i})}.$
(29)

where $\mu_{obs}$ represents the observed distance modulus and $\sigma_{\mu}$
denotes its statistical uncertainty.

For the observed Hubble parameter dataset in Table 1, the best-fit values of
the parameters $(H_{{{\cal D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)$ can be determined by a likelihood analysis based on the calculation
of

$\chi^{2}_{H}(H_{{{\cal D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)=\sum_{i=0}^{30}\frac{(H_{{\cal D}}({z_{{\cal D}}}_{i};H_{{{\cal
D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)-H_{obs}({z_{{\cal
D}}}_{i}))^{2}}{\sigma_{H}^{2}({z_{{\cal D}}}_{i})}.$ (30)

As Ma et. al.ma2011power  stated, the marginalized probability density
function determined by integrating $\rm e^{-\frac{\chi^{2}_{H}(H_{{{\cal
D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)}{2}}$ over $H_{{{\cal
D}_{\rm\bf 0}}}$ from $x$ to $y$ with a uniform prior reads

${\rm e^{-\frac{\chi^{2}_{H}(\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)}{2}}}=\frac{U(x,C,D)-U(y,C,D)}{\sqrt{C}}{\rm e}^{\frac{D^{2}}{C}}$
(31)

where

$C=\sum_{i=0}^{30}\frac{H_{{\cal D}}^{2}({z_{{\cal D}}}_{i};H_{{{\cal
D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)}{2H_{{{\cal D}_{\rm\bf
0}}}^{2}\sigma_{H}^{2}({z_{{\cal D}}}_{i})},D=\sum_{i=0}^{30}\frac{H_{{\cal
D}}({z_{{\cal D}}}_{i};H_{{{\cal D}_{\rm\bf 0}}},\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)H_{obs}({z_{{\cal D}}}_{i})}{2H_{{{\cal D}_{\rm\bf
0}}}\sigma_{H}^{2}({z_{{\cal D}}}_{i})},$ (32)

and

$U(x,\alpha,\beta)={\rm erf}(\frac{\beta-x\alpha}{\sqrt{\alpha}}),$

$[x,y]$ is taken as $[50,90]$, and erf represents the error function.

Finally, the total $\chi^{2}(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)$ for the
combined observational dataset are given by $\chi^{2}(\Omega_{m}^{{{\cal
D}_{\rm\bf 0}}},o)=\chi_{SNIa}^{2}(\Omega_{m}^{{{\cal D}_{\rm\bf
0}}},o)+\chi_{H}^{2}(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},o)$.

The fitting results attained from analyzing $\chi^{2}(\Omega_{m}^{{{\cal
D}_{\rm\bf 0}}},o)$ are presented in Fig.1, Table 2 for the scaling
backreaction model and Fig.2, Table 3 for the constant backreaction model. The
comparison of $\chi^{2}_{min}$ in these two tables show that the constant
backreaction model is slightly favoured over the other model by current
observations, confirming the correctness of our speculation that the scaling
solution is not a prime description of the late-time backreaction term. Also,
the result in Table 2 suggests that, within $1\sigma$ confidence interval of
the parameter $n$ in the scaling backreaction model, $\mid{\cal Q}_{{\cal
D}}\mid$ decreases with the increase of volume scale factor $a_{{\cal D}}$ at
adequately late times, indicating the existence of a maximum of $\mid{\cal
Q}_{{\cal D}}\mid$ at large redshift in consideration of the structure
formation effects which lead to a rapid change of the backreaction term at
redshift $z_{\cal D}\sim 10$. This behavior of $\mid{\cal Q}_{{\cal D}}\mid$
is different with that in the constant backreaction model in which $\mid{\cal
Q}_{{\cal D}}\mid$ tends to be a constant at late times.

Noting from the fitting results that the best-fit value of the matter density
parameter in the scaling backreaction model is bigger than that in the other
model, indicating that this model predicts a larger expansion rate and
decelerated expansion rate at high redshifts. Such departure is shown in
Fig.3, which also reveals that the universes described by two models with
their best-fit parameters share the almost same expansion rate and decelerated
expansion rate(accelerated expansion rate) once $z_{{\cal D}}$ drops below
about 1, and enter a stage of an accelerated expansion with a redshift around
0.6.

One may be aware that by defining $\Omega_{k}^{{{\cal D}_{\rm\bf
0}}}=\Omega_{{\cal R}}^{{{\cal D}_{\rm\bf 0}}}+3\Omega_{{\cal Q}}^{{{\cal
D}_{\rm\bf 0}}}$ and $\Omega_{\Lambda}^{{{\cal D}_{\rm\bf 0}}}=-2\Omega_{{\cal
Q}}^{{{\cal D}_{\rm\bf 0}}}$, we can rewrite $H_{{\cal D}}(a_{{\cal D}})$ in
the constant backreaction model as

$H_{{\cal D}}(a_{{\cal D}})=H_{{{\cal D}_{\rm\bf 0}}}(\Omega_{m}^{{{\cal
D}_{\rm\bf 0}}}a_{{\cal D}}^{-3}+\Omega_{\Lambda}^{{{\cal D}_{\rm\bf
0}}}+\Omega_{k}^{{{\cal D}_{\rm\bf 0}}}a_{{\cal D}}^{-2})^{\frac{1}{2}}$ (33)

, which is formally identical to the the Hubble parameter in the $o-\Lambda
CDM$ model. However, such two models in fact are quite different, firstly, the
parameters of two models have different physical meaning, this is obvious
since $\Omega_{\Lambda}$ in the $o-\Lambda CDM$ model denotes the vacuum
energy density parameter, while $\Omega_{\Lambda}^{{{\cal D}_{\rm\bf 0}}}$ is
assumed as an averaged effect caused by small scale inhomogeneities of the
universe. Secondly, two models are considered in different geometry
background, which leads to different fitting results, in fact, by confronting
the $o-\Lambda CDM$ model with the same dataset we use above, we obtain the
best-fit result $\chi^{2}_{min}=-1334.52,\Omega_{m}=0.30,\Omega_{k}=0.04$.
Clearly, $\Omega_{m}>\Omega_{m}^{{{\cal D}_{\rm\bf
0}}}=0.25,\Omega_{k}<\Omega_{k}^{{{\cal D}_{\rm\bf 0}}}=0.27$, so the
effective geometry tends to push the constraint toward smaller value for
$\Omega_{\Lambda}^{{{\cal D}_{\rm\bf 0}}}$ compared to $\Omega_{\Lambda}$. We
note that $\chi^{2}_{min}$ in the $o-\Lambda CDM$ model is smaller than that
in the constant backreaction model, this is reasonable since the datasets we
use are obtained in the way that already depends on the FLRW geometry
assumption.

$z$ | $H(z)$ | Ref.
---|---|---
$0.0708$ | $69.0\pm 19.68$ | Zhang et al. (2014)-Zhang2014
$0.09$ | $69.0\pm 12.0$ | Jimenez et al. (2003)-Jimenez2003
$0.12$ | $68.6\pm 26.2$ | Zhang et al. (2014)-Zhang2014
$0.17$ | $83.0\pm 8.0$ | Simon et al. (2005)-Simon2005
$0.179$ | $75.0\pm 4.0$ | Moresco et al. (2012)-Moresco2012
$0.199$ | $75.0\pm 5.0$ | Moresco et al. (2012)-Moresco2012
$0.20$ | $72.9\pm 29.6$ | Zhang et al. (2014)-Zhang2014
$0.27$ | $77.0\pm 14.0$ | Simon et al. (2005)-Simon2005
$0.28$ | $88.8\pm 36.6$ | Zhang et al. (2014)-Zhang2014
$0.352$ | $83.0\pm 14.0$ | Moresco et al. (2012)-Moresco2012
$0.3802$ | $83.0\pm 13.5$ | Moresco et al. (2016)-Moresco2016
$0.4$ | $95\pm 17.0$ | Simon et al. (2005)-Simon2005
$0.4004$ | $77.0\pm 10.2$ | Moresco et al. (2016)-Moresco2016
$0.4247$ | $87.1\pm 11.2$ | Moresco et al. (2016)-Moresco2016
$0.4497$ | $92.8\pm 12.9$ | Moresco et al. (2016)-Moresco2016
$0.4783$ | $80.9\pm 9.0$ | Moresco et al. (2016)-Moresco2016
$0.48$ | $97.0\pm 62.0$ | Stern et al. (2010)-Stern2010
$0.593$ | $104.0\pm 13.0$ | Moresco et al. (2012)-Moresco2012
$0.68$ | $92.0\pm 8.0$ | Moresco et al. (2012)-Moresco2012
$0.875$ | $125.0\pm 17.0$ | Moresco et al. (2012)-Moresco2012
$0.88$ | $90.0\pm 40.0$ | Stern et al. (2010)-Stern2010
$0.9$ | $117.0\pm 23.0$ | Simon et al. (2005)-Simon2005
$1.037$ | $154.0\pm 20.0$ | Moresco et al. (2012)-Moresco2012
$1.3$ | $168.0\pm 17.0$ | Simon et al. (2005)-Simon2005
$1.363$ | $160.0\pm 33.6$ | Moresco (2015)-Moresco2015
$1.43$ | $177.0\pm 18.0$ | Simon et al. (2005)-Simon2005
$1.53$ | $140.0\pm 14.0$ | Simon et al. (2005)-Simon2005
$1.75$ | $202.0\pm 40.0$ | Simon et al. (2005)-Simon2005
$1.965$ | $186.5\pm 50.4$ | Moresco (2015)-Moresco2015
Table 1:  The current available OHD dataset. model parameters |  | scaling backreaction model
---|---|---
$\Omega_{m}^{{{\cal D}_{\rm\bf 0}}}$ |  | $0.31_{-0.05}^{+0.05}$
$n$ |  | $-0.70_{-0.30}^{+0.35}$
$\chi^{2}_{min}$ |  | $-1329.32$
Table 2:  The fitting results of the parameters ($\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},n$) with 1$\sigma$ region in the scaling backreaction model, $\chi^{2}_{min}$ is corresponding to $(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},n)=(0.30,-0.68)$. model parameters |  | constant backreaction model
---|---|---
$\Omega_{m}^{{{\cal D}_{\rm\bf 0}}}$ |  | $0.25_{-0.07}^{+0.06}$
$\Omega_{{\cal R}}^{{{\cal D}_{\rm\bf 0}}}$ |  | $0.99_{-0.03}^{+0.05}$
$\chi^{2}_{min}$ |  | $-1331.84$
Table 3:  The fitting results of the parameters ($\Omega_{m}^{{{\cal
D}_{\rm\bf 0}}},\Omega_{{\cal R}}^{{{\cal D}_{\rm\bf 0}}}$) with 1$\sigma$
region in the constant backreaction model, $\chi^{2}_{min}$ is corresponding
to $(\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},\Omega_{{\cal R}}^{{{\cal D}_{\rm\bf
0}}})=(0.25,0.99)$.

Figure 1: The $1\sigma$, $2\sigma$ and $3\sigma$ confidence regions of the
free parameters $\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},n$ for the scaling
backreaction model, along with their own probability density function.

Figure 2: The $1\sigma$, $2\sigma$ and $3\sigma$ confidence regions of the
free parameters $\Omega_{m}^{{{\cal D}_{\rm\bf 0}}},\Omega_{{\cal R}}^{{{\cal
D}_{\rm\bf 0}}}$ for the constant backreaction model, along with their own
probability density function.

Figure 3: The evolution of $\frac{H_{{\cal D}}}{H_{{{\cal D}_{\rm\bf 0}}}}$
and $q_{{\cal D}}$ with respect to $z_{{\cal D}}$. Here the blue line and the
orange line corresponding to that of the scaling backraction model and the
constant backreaction model with best-fit parameters.

##  V Conclusion and discussion

In this paper, the dark energy phenomenon has been interpreted as an averaged
effect caused by small scale inhomogeneities of the universe. In order to
understand the averaged evolutional behavior of the universe within the
approach of Buchert, we have considered two backreaction models, one of which
assumes that the backreaction term ${\cal Q}_{\cal D}$ and the averaged
spatial Ricci scalar $\left\langle{\cal R}\right\rangle_{\cal D}$ obey the
scaling laws of the volume scale factor $a_{\cal D}$ at adequately late times,
and the other one adopts the ansatz that ${\cal Q}_{\cal D}$ is a constant in
the recent universe. With the aid of the effective geometry introduced by
Larena et. al. in their previous work, we have confronted these two
backreaction models with latest type Ia supernova and Hubble parameter
observations, and found that the constant backreaction model is preferred by
current observations. Also, the analysis result of the probability density
function of the parameter $n$ suggests that the $\mid{\cal Q}_{{\cal D}}\mid$
increases toward the past at adequately late times, indicating the existence
of a maximum of $\mid{\cal Q}_{{\cal D}}\mid$ at large redshift predicted by
the scaling model in consideration of the structure formation effects.
Moreover, as is shown by the results of numerical simulation, the constant
backreaction model predicts a smaller expansion rate and decelerated expansion
rate than the other model does at redshifts higher than about 1 and both
backreaction terms begin to accelerate the universe at a redshift around 0.6.
In addition, by confronting the $o-\Lambda CDM$ model against the same
datasets, we find that the effective geometry tends to push the constraint
toward a smaller cosmological constant term.

Although we only make assumptions about the specific form of the backreaction
term at late times throughout the paper, a complete backreaction model must
consider the specific behavior of the backreaction term at arbitrary redshift.
Nevertheless, parameterization of the late-time backreaction term is helpful
and necessary for searching a complete backreaction model that is also
favoured by observations at high redshifts.

## Acknowledgments

The paper is supported by the Natural Science Foundation of China.

## References

  * (1) A.G. Riess, A.V. Filippenko, P. Challis, A. Clocchiatti, A. Diercks, P.M. Garnavich, R.L. Gilliland, C.J. Hogan, S. Jha, R.P. Kirshner, et al., The Astronomical Journal 116(3), 1009 (1998)
  * (2) S. Perlmutter, G. Aldering, G. Goldhaber, R. Knop, P. Nugent, P. Castro, S. Deustua, S. Fabbro, A. Goobar, D. Groom, et al., The Astrophysical Journal 517(2), 565 (1999)
  * (3) R.R. Caldwell, R. Dave, P.J. Steinhardt, Physical Review Letters 80(8), 1582 (1998)
  * (4) R.R. Caldwell, Phys.lett 45(3), 549 (1999)
  * (5) F. Piazza, S. Tsujikawa, Journal of Cosmology and Astroparticle Physics 2004(07), 004 (2004)
  * (6) T. Padmanabhan, Physical Review D Particles Fields 66(2), 611 (2002)
  * (7) B. Feng, M. Li, Y.S. Piao, X. Zhang, Physics Letters B 634(2), 101 (2006)
  * (8) R. Maartens, K. Koyama, Living Reviews in Relativity 13(1), 5 (2010)
  * (9) G. Esposito-Farese, D. Polarski, Physical Review D 63(6), 063504 (2001)
  * (10) S. Capozziello, V.F. Cardone, A. Troisi, Physical Review D 71(4), 043503 (2005)
  * (11) S. Das, N. Banerjee, N. Dadhich, Classical and Quantum Gravity 23(12), 4159 (2006)
  * (12) S. Räsänen, Journal of Cosmology and Astroparticle Physics 2004(02), 003 (2004)
  * (13) E.W. Kolb, S. Matarrese, A. Riotto, New Journal of Physics 8(12), 322 (2006)
  * (14) R.M. Zalaletdinov, General Relativity and Gravitation 24(10), 1015 (1992)
  * (15) R.M. Zalaletdinov, arXiv preprint gr-qc/9703016 (1997)
  * (16) R.M. Zalaletdinov, General relativity and gravitation 25(7), 673 (1993)
  * (17) M. Mars, R.M. Zalaletdinov, Journal of Mathematical Physics 38(9), 4741 (1997)
  * (18) T. Buchert, General Relativity and Gravitation 32(1), 105 (2000)
  * (19) T. Buchert, General Relativity and Gravitation 33(8), 1381 (2001)
  * (20) T. Buchert, J. Larena, J.M. Alimi, Classical and Quantum Gravity 23(22), 6379 (2006)
  * (21) J. Larena, J.M. Alimi, T. Buchert, M. Kunz, P.S. Corasaniti, Physical Review D 79(8), 083011 (2009)
  * (22) N. Suzuki, D. Rubin, C. Lidman, G. Aldering, R. Amanullah, K. Barbary, L. Barrientos, J. Botyanszki, M. Brodwin, N. Connolly, et al., The Astrophysical Journal 746(1), 85 (2012)
  * (23) C. Ma, T.J. Zhang, The Astrophysical Journal 730(2), 74 (2011)
  * (24) C. Zhang, H. Zhang, S. Yuan, S. Liu, T.J. Zhang, Y.C. Sun, Research in Astronomy and Astrophysics 14(10), 1221 (2014)
  * (25) R. Jimenez, L. Verde, T. Treu, D. Stern, The Astrophysical Journal 593(2), 622 (2003)
  * (26) J. Simon, L. Verde, R. Jimenez, Physical Review D 71(12), 123001 (2005)
  * (27) M. Moresco, L. Verde, L. Pozzetti, R. Jimenez, A. Cimatti, Journal of Cosmology and Astroparticle Physics 2012(07), 053 (2012)
  * (28) M. Moresco, L. Pozzetti, A. Cimatti, R. Jimenez, C. Maraston, L. Verde, D. Thomas, A. Citro, R. Tojeiro, D. Wilkinson, Journal of Cosmology and Astroparticle Physics 2016(05), 014 (2016)
  * (29) D. Stern, R. Jimenez, L. Verde, M. Kamionkowski, S.A. Stanford, Journal of Cosmology and Astroparticle Physics 2010(02), 008 (2010)
  * (30) M. Moresco, Monthly Notices of the Royal Astronomical Society: Letters 450(1), L16 (2015)