# Color Equivariant Network

Felix O’Mahony1 Yulong Yang2 Christine Allen-Blanchette2
1University of Oxford, Oxford, UK 2Princeton University, Princeton, USA
<EMAIL_ADDRESS>{yulong.yang<EMAIL_ADDRESS>
###### Abstract

Group equivariant convolutional neural networks have been designed for a
variety of geometric transformations from 2D and 3D rotation groups, to semi-
groups such as scale. Despite the improved interpretability, accuracy and
generalizability afforded by these architectures, group equivariant networks
have seen limited application in the context of perceptual quantities such as
hue and saturation, even though their variation can lead to significant
reductions in classification performance. In this paper, we introduce
convolutional neural networks equivariant to variations in hue and saturation
by design. To achieve this, we leverage the observation that hue and
saturation transformations can be identified with the 2D rotation and 1D
translation groups respectively. Our hue-, saturation-, and fully color-
equivariant networks achieve equivariance to these perceptual transformations
without an increase in network parameters. We demonstrate the utility of our
networks on synthetic and real world datasets where color and lighting
variations are commonplace.

##  1 Introduction

The tremendous progress of image classification in the last decade can be
readily attributed to the development of deep convolutional neural networks
[1; 2; 3]. The highly nonlinear mapping and large parameter space does not
lend itself to interpretation easily, but even in early networks,
representations of geometry and color were observed and recognized for their
importance in representation learning [1; 4]. While a relatively large body of
literature has worked to improve the robustness of convolutional neural
networks to geometric transformations [5; 6; 7; 8; 9; 10], improving their
robustness to perceptual variation has garnered considerably less attention.

A commonly used heuristic for improving network robustness to color variation
is to perform mean subtraction and normalization on training set examples.
This approach can work well when the training and testing datasets are drawn
from the same distribution; however, for data that are collected at different
points in time, or with different sensors, this not likely to be the case.
Consider, for example, the case of medical imaging where images of tissue
samples collected from different labs (or from the same lab at different
points in time) may have different characteristics due to variability in data
collection protocols or imaging processes [11]. This variability presents a
significant challenge for convolutional neural networks which have been found
to be sensitive to color variations even on the level of individual blocks
[12]. Moreover, when presented with color perturbations, conventional networks
exhibit a significant drop in classification performance [13]. One approach to
mitigate the effect of color variation is to ignore color entirely by
enforcing color invariance. This can be done by converting input images to
grayscale, or enforcing representation similarity across color as is done in
[14]. This approach has its own challenges, however, since in many domains,
color is an important cue for classification. Another approach is to use
dataset augmentation, a technique that can improve robustness in the presence
of known transformations [15]. This approach, however, requires extended
training times and is not likely to offer improved interpretability.

In this work, we address the challenge of neural network sensitivity to color
variation by leveraging the geometric structure of color in a group
equivariant network. Geometric deep learning has gained strong interest in
recent years due to its ability to capture information in interpretable and
generalizable representations; by using a group equivariant architecture, we
inherit these characteristics for the context of color variation. Using this
approach, our network retains color information in a structured representation
throughout the processing pipeline, thus allowing color information to be used
and/or discarded with intentionality.

We represent color in the hue saturation luminance color space, and leverage
the insight that the hue, and saturation can be modeled with geometric group
structure. Specifically, we identify variations in hue with the 2D rotation
group, and variations in saturation with the 1D translation group. With this
identification, we propose a group equivariant network where variations in hue
and saturation are represented as a geometric transformations.

To summarize, our key contributions are:

  * •

We introduce networks that are equivariant to shifts in hue and saturation,
without requiring an increase in the number of network parameters.

  * •

Our networks perform on par or better than conventional architectures in the
presence of unseen color combinations on synthetic, and real world datasets.

  * •

Our networks produce interpretable feature representations that are useful for
downstream tasks such as color based sorting.

##  2 Related Work

##### Group Equivariant Networks

The strong generalization performance of convolutional neural networks on
image processing tasks is attributed, in part, to the equivariance of planar
convolution to 2D translations. This insight has garnered considerable
attention and led to a strong interest in the development of convolutional
neural networks equivariant to other symmetry groups [16; 17]. Previous works
introduce group equivariant convolutional networks for finite and continuous
groups. A framework for finite group convolution was developed in [6], and
demonstrated for the 2D rotation, and reflection groups. The authors achieve
equivariance to specific symmetry groups by convolving input images and
feature maps with the group orbit of a learned filter bank. In [18], the
authors design for equivariance to the continuous 2D rotation group by
constraining filter representations to have circular harmonic structure.

Group equivariant networks have also been designed for use in the context of
scale symmetry. In [19], the authors represent input images in log-polar
coordinates where 2D rotation and scale transformations present as 2D
translations. To navigate the semi-group structure of scale transformations,
the authors in [20] approximate the scale space as finite and use dilated
convolutions in a group equivariant architecture. The SREN network proposed in
[21] designs for equivariance to the continuous 2D rotation and isotropic
scaling group by constraining filter representations to be expressed as the
linear combination of a windowed Fourier basis.

Group equivariant networks have also been designed for groups acting in higher
dimensions. For 3D rotational symmetry, [22] and [23] use a spherical
harmonics based representation with 3D point clouds and spherical image inputs
respectively; and [24] introduce an equivariant network for transformations of
the Euclidean group in 3D. In [25], the authors introduce a framework for
general Lie groups whose exponential map is surjective, and [26] introduces a
framework for more general manifolds. By designing for known symmetries in the
task, these works provide improved interpretability, training efficiency and
generalizability over conventional convolutional networks.

Other works encourage transformation equivariance using a soft penalty term on
representation dissimilarity [4; 27]. These methods are separate from the
group equivariant network literature, and different from what we propose. Our
model leverages the geometric structure of hue and saturation to design a hue-
saturation equivariant network bringing the benefits of group equivariant
networks to perceptual transformations.

##### Color Invariance

Several prior works attempt to mitigate the effect of color variation in image
processing tasks, for example, in [28], the authors represent images in a
color space where pixel values are invariant to changes in luminance for
luminance invariant image segmentation, and in [14], the authors penalize
changes in their latent representations due to color variation for color
invariant skin lesion identification. While color invariance resolves the
effect of color variation, color has been identified as an important cue in
neural network representation learning [12; 13] which is discordant with the
goal of color invariance. In contrast to these approaches, our work leverages
the geometric structure of hue, and saturation to construct a convolutional
neural network equivariant to variations in these quantities by design.

Most similar to ours is the model proposed in [29], a hue equivariant network
for improved network performance in the presence of color variation. Our color
equivariant networks differ from the work proposed in [29] in three important
ways. First, our networks are fully equivariant, in contrast to the hybrid
networks proposed in [29]. Second, we lift the input image instead of the
filters of the first layer which circumvents the issue of invalid hue
rotations suffered by the network in [29]. Finally, we introduce networks
equivariant to hue and saturation, which together provide full color
equivariance and is unique to our approach.

##  3 Preliminaries

In this section we describe our notation, and review definitions of the group
action, equivariance, and group convolution.

##### Notation.

We use $f^{l}$ to denote the $l$-th feature map ($f^{0}$ to denote an input
image), and $f^{l}_{j}$ to denote the $j$-th channel of that feature map. We
use $\psi_{i}^{l}$ to denote the $i$-th filter of the $l$-th layer, and
$\psi_{i,j}^{l}$ to denote the $j$-th channel of that filter.

##### Group action.

Adapted from [30]. Given a set $X$ and a group $G$, the action of the group
$G$ on $X$ is a function $\varphi:G\times X\rightarrow X$ satisfying the
following:

  1. 1.

For all $g,h\in G$ and all $x\in X$

$\varphi(g,\varphi(h,x))=\varphi(gh,x)$ (1)

  2. 2.

For all $x\in X$

$\varphi(1,x)=x$ (2)

where $1\in G$ is the identity element of G.

The set $X$ is called a (left) $G$-set.

##### Equivariance.

Adapted from [30]. Given two $G$-sets $X$ and $Y$, and group actions
$\varphi:G\times X\rightarrow X$ and $\phi:G\times Y\rightarrow Y$, a function
$f:X\rightarrow Y$ is said to be equivariant, if and only if for all $x\in X$,
and $g\in G$,

$f(\varphi(g,x))=\phi(g,f(x)).$ (3)

When $f$ is invariant to the action of $G$, the group action $\phi$ is the
identity map and we write

$f(\varphi(g,x))=f(x).$ (4)

The equivariance of our network to hue shifts is illustrated in Figure 1(a).

##### Group convolution.

In a conventional CNN, the input to convolution layer $l$, denoted
$f^{l}:\mathbb{Z}^{2}\rightarrow\mathbb{R}^{K^{l}}$, is convolved with a set
of $K^{l+1}$ filters, denoted
$\psi^{l}_{i}:\mathbb{Z}^{2}\rightarrow\mathbb{R}^{K^{l}}$, where $i$ ranges
from 1 to $K^{l+1}$. The result of the convolution can be written:

$f^{l+1}_{i}=[f^{l}*\psi_{i}^{l}](x)=\sum_{y\in\mathbb{Z}^{2}}\sum_{k=1}^{K^{l}}f^{l}_{k}(y)\psi^{l}_{i,k}(x-y),\;x\in\mathbb{Z}^{2}.$
(5)

The convolution of conventional CNNs is equivariant to the action of the group
$(\mathbb{Z}^{2},+)$, that is, the group formed by summing over the integers.
The more general group convolution can be written:

$[f^{l}*\psi_{i}^{l}](g)=\sum_{h\in
G}\sum_{k=1}^{K^{l}}f^{l}_{k}(h)\psi_{i,k}^{l}(h^{-1}g),$ (6)

and is equivariant to the action of the group $G$.

##  4 Method

$f(\varphi(g,x))$$\varphi(g,x)$$\phi(g,f(x))$$f(x)$ (a) Hue equivariance

(b) Lifting layer

Figure 1: Color equivariant network. (a) The equivariance of our hue
equivariant model is illustrated by the commutativity of the (hue) rotation
and neural network mapping. A hue rotation of 90∘ in the input image space
(top-left to bottom-left), results in a feature map rotation at each layer of
the network (top-right to bottom-right). (b) An input image (left) is “lifted”
to the hue-saturation group (right) by shifting its hue and saturation values.

In this section we present our color equivariant network. We begin by
presenting definitions for the hue and saturation groups and their group
actions, then define the lifting layers, and group convolution layers for our
color equivariant networks.

##### Hue group and group action.

In the HSL color space, hue is represented by angular position, and can
therefore be identified with the 2D rotation group. As in group equivariant
network [6], we consider a finite group representation. Specifically, we
identify elements of the discretized hue group, $H_{N}$, with those of the
cyclic group $C_{N}$.

We define the action of hue group on HSL images, $x\in X$ where
$x:\mathbb{Z}^{2}\rightarrow\mathbb{R}^{3}$, and functions on the discrete hue
group, $y\in Y$, where $y:\mathbb{Z}^{2}\times
H_{N}\rightarrow\mathbb{R}^{K}$. An element of the hue group acts on an HSL
image by the group action $\varphi_{h}:H_{N}\times X\rightarrow X$, which
shifts the hue channel of the image. Concretely, for an HSL image $x\in X$
defined as the concatenation of hue, saturation and luminance channels, i.e.,
$x=(x_{h},x_{s},x_{l})$, the action of an element $h_{i}$ of the hue group
$H_{N}$ is given by

$\varphi_{h}(h_{i},x)=((x_{h}+h_{i})(\text{mod}\;255),x_{s},x_{l}),$ (7)

where $(\cdot)\text{mod}(\cdot)$ denotes the modulus operation and is applied
to the pixel value rather than its corresponding degree.

An element of the hue group acts on a function on the discrete hue group by
the group action $\phi_{h}:H_{N}\times Y\rightarrow Y$, which “rotates" the
function on the group. Concretely, for a function $f$ on the discrete hue
group $H_{N}$ defined as the concatenation of functions
$f=(f_{1},\dots,f_{N})$, the action of an element $h_{i}$ in the hue group
$H_{N}$ is given by

$\phi_{h}(h_{i},f)=(f_{(1+i)(\text{mod}\;N)},\dots,f_{(N+i)(\text{mod}\;N)}).$
(8)

##### Saturation group and group action.

In the HSL color space, saturation can be represented by a real number in the
interval $[0,1]$. We introduce two approximations to give group structure to
the saturation space. First, we observe that there is a bijection between the
real numbers and the open interval $(0,1)$, so we use the structure of the
group $(\mathbb{R},+)$. Second, we consider a finite subset of the group, a
necessary and commonly used practice in both conventional CNNs (recall that
the translation group is infinite) and group-equivariant CNNs such as [20].

We define the action of the saturation group on HSL images, $x\in X$ where
$x:\mathbb{Z}^{2}\rightarrow\mathbb{R}^{3}$, and functions on the discrete
saturation group, $y\in Y$, where $y:\mathbb{Z}^{2}\times
S_{N}\rightarrow\mathbb{R}^{K}$. An element of the saturation group acts on an
HSL image by the group action $\varphi_{s}:S_{N}\times X\rightarrow X$, which
shifts the saturation channel of the image. Concretely, for an HSL image $x\in
X$ defined as the concatenation of hue, saturation and luminance channels,
i.e., $x=(x_{h},x_{s},x_{l})$, the action of an element $s_{i}$ of the
saturation group $S_{N}$ is given by

$\varphi_{s}(s_{i},x)=(x_{h},\text{min}(x_{s}+s_{i},255),x_{l}).$ (9)

An element of the saturation group acts on a function on the discrete
saturation group by the group action $\phi_{s}:S_{N}\times Y\rightarrow Y$,
which “translates" the function on the group. Concretely, for a function $f$
on the discrete saturation group $S_{N}$ defined as the concatenation of
functions $f=(f_{1},\dots,f_{N})$, the action of an element $s_{i}$ in the
saturation group $S_{N}$ is given by

$\phi_{s}(s_{i},f)=(f_{1+i},\dots,f_{N},\underbrace{\mathbf{0},\dots,\mathbf{0}}_{i}).$
(10)

##### Hue-Saturation group action.

We define the action of the hue-saturation group as a composition of the hue
and saturation group actions. An element of the hue-saturation group acts on
an HSL image by the group action $\varphi_{hs}:H_{N}\times S_{M}\times
X\rightarrow X$, which shifts both the hue and saturation channels of the
image. For an HSL image $x\in X$ defined as the concatenation of hue,
saturation and luminance channels, i.e., $x=(x_{h},x_{s},x_{l})$, the action
of an element $(h_{i},s_{j})$ of the hue-saturation group $H_{N}\times S_{M}$
is given by

$\varphi_{hs}((h_{i},s_{j}),x)=\varphi_{h}(h_{i},\varphi_{s}(s_{j},x)).$ (11)

An element of the hue-saturation group acts on a function on the discrete
saturation group by the group action $\phi_{hs}:H_{N}\times S_{M}\times
Y\rightarrow Y$, which “rotates" and “translates" the function on the group.
Concretely, for a function $f$ on the discrete hue-saturation group
$H_{N}\times S_{M}$ defined as the concatenation of functions
$f=(f_{11},\dots,f_{1M},f_{21},\dots,f_{NM})$, the action of an element
$(h_{i},s_{j})$ in the hue-saturation group $H_{N}\times S_{M}$ is given by

$\phi_{hs}((h_{i},s_{j}),f)=\phi_{h}(h_{i},\phi_{s}(s_{j},f)).$ (12)

##### Lifting layer.

The first layer of a group convolutional neural network “lifts" the input
image to the group (see Figure 1(b)). We can lift an input image to the
product space of the image grid $\mathbb{Z}^{2}$, and discretized hue shifts
$H_{N}=\\{h_{0},h_{1},\dots,h_{N}\\}$, by convolving with hue shifted filters,

$[f^{0}*\psi^{0}_{i}](g_{x,j})=\sum_{y\in\mathbb{Z}^{2}}\sum_{k=1}^{K^{0}}f^{0}_{k}(y)h_{j}\psi^{0}_{i,k}(x-y).$
(13)

Here we denote an element of the product space $g_{x,i}$, where the subscript
$x$ references an element of $\mathbb{Z}^{2}$, and $j$ references the element
$h_{j}$ in $H_{N}$. The input image $f^{0}$ and lifting filters $\psi_{m,n}$
are functions on $\mathbb{Z}^{2}$.

Lifting to the group can also be achieved by transforming the input image
rather than the filters,

$f^{1}(g_{x,j})=\varphi_{h}(h_{j},f^{0})(x).$ (14)

We use this alternative in our implementation as it is both simple and does
not result in pixels falling outside the RBG cube as in [29]. The lifting
layers for the saturation group, and hue-saturation group are constructed
analogously.

##### Equivariance of the lifting layer.

We can show that our lifting layer is equivariant to hue shifts. We let
$f^{1}(g_{x,j})=\varphi_{h}(h_{j},f^{0})(x)$ as in equation 14 and show that a
hue shift of $f^{0}$ results in a hue shift of $f^{1}$. In the first step we
use the fact that $\varphi_{h}$ is a group action (see Appendix C.1), and in
the second step we use commutativity of the hue group:

$\displaystyle\varphi_{h}(h_{j},\varphi_{h}(h_{m},f^{0}))(x)$
$\displaystyle=\varphi(h_{j}h_{m},f^{0})(x)$ (15)
$\displaystyle=\varphi_{h}(h_{m}h_{j},f^{0})(x)$ (16)
$\displaystyle=\varphi_{h}(h_{m},\varphi_{h}(h_{j},f^{0}))(x)$ (17)
$\displaystyle=\varphi_{h}(h_{m},f^{1}(g_{x,j})).$ (18)

Equivariance of the saturation, and hue-saturation lifting layers can be shown
analogously.

##### Group convolution layer.

For all layers after the first layer, the feature maps $f^{l}$ are on the
product space $\mathbb{Z}^{2}\times H_{N}$. Since hue shifts can only be
performed on three dimensional inputs, we leverage the identification of hue
shifts with the discrete 2D rotation group, i.e., $H_{N}\cong C_{N}$ and
interpret the feature maps $f^{l}$ as functions on the product space
$G:=\mathbb{Z}^{2}\times C_{N}$. With this interpretation we perform group
convolution on $G$ as follows,

$[f^{l}*\psi^{l}_{i}](g)=\sum_{h\in
H}\sum_{k=1}^{K^{0}}f^{l}_{k}(h)\psi^{l}_{i,k}(g^{-1}h),$ (19)

where $\psi^{l}$ is a function on $G$. Group convolution on the saturation,
and hue-saturation groups are performed analogously.

##  5 Experiments

In this section we demonstrate the strength of our method in the presence of
global and local hue variation, and global saturation variation on synthetic
and real world datasets. We also demonstrate the ability of our model to
perform color dependent tasks such as sorting.

###  5.1 Hue-shift MNIST: Global hue equivariance

Table 1: Classification error on the hue shift MNIST dataset. Our networks are marked with (*). The number of parameters for all networks is reported.  | Z2CNN | Hue-4* | Hue-3* | CEConv | CEConv-2
---|---|---|---|---|---
$A$/$A$ | 1.54 (0.10) | 1.97 (0.25) | 1.79 (0.25) | 1.79 (0.13) | 2.81 (0.44)
$A$/$B$ | 57.38 (22.06) | 2.08 (0.15) | 1.81 (0.28) | 1.83 (0.19) | 5.08 (0.18)
Params | 22,130 | 25,690 | 22,658 | 28,739 | 30,539

(a) Z2CNN

(b) Hue-CNN

Figure 2: Hue shift MNIST feature map visualization. We compare the feature
map trajectories of MNIST digits as their hue is varied from 1 to 360∘. The
color of the trajectory corresponds to the class label. (a) tSNE projection of
hue shifted feature map trajectories in the Z2CNN baseline model. As the hue
of the input changes, the location of the digit in the feature space changes
significantly. (b) tSNE projection of hue shifted feature map trajectories in
our hue-equivariant CNN. In contrast to the Z2CNN baseline, the location of
the digit in the feature space changes minimally.

We evaluate the performance of our models in the presence of global hue shifts
on a classification dataset derived from MNIST [31]. The dataset consists of
70k examples, 60k of which are used for training and 10k of which are used for
testing.

To test the performance of our models, we partition the testing dataset into
two subsets: an in-distribution testing set and an out-of-distribution testing
set. In the training set and in-distribution testing set, denoted subset $A$,
digits are colored with a randomly selected hue between $0^{\circ}$ and
$240^{\circ}$ (see Figure 5(a)). In the out-of-distribution testing set,
subset $B$, digits are colored with a randomly selected hue between
$240^{\circ}$ and $360^{\circ}$ (see Figure 5(b)).

Model performance is reported in Table 1. On the in-distribution test case
($A$/$A$), the performance of our hue-equivariant models (Hue-x) and the
conventional model (Z2CNN) are comparable. On the out-of-distribution test
case ($A$/$B$), the performance of our hue-equivariant models is preserved,
while the performance of the conventional model deteriorates significantly.

The performance gap on the out-of-distribution test case can be attributed to
the difference in internal representation structure. To understand this
better, we generate feature representation trajectories in the penultimate
layer of our network and the baseline architecture. Feature representation
trajectories are generated by continuously varying the hue of an input image,
and visualized using tSNE projection (see Figure 2). For the baseline
architecture, the trajectories of different digits overlap for some hues
(Figure 2(a)). For our model, the trajectories of different digits are
confined to separate clusters (Figure 2(b)).

We also compare our model to the hue-equivariant (CEConv), and hybrid network
(CEConv-2) proposed in [29]. With the same hue discretization (i.e., $N=3$),
our model performs comparably with the baseline hue-equivariant model, and out
performs the baseline hybrid model. We investigate differences in the internal
representations of our model, the conventional architecture, and the baseline
hybrid model in Figure 3. For each network, we visualize the feature maps
generated for a hue rotated input. For a hue discretization of $N=3$, the
network feature maps have three channels and can be visualized as an RGB
image. Only the feature maps of our network shift predictably in response to
hue shifts of the input.

(a) Z2CNN (b) Hue-CNN (c) Hybrid

Figure 3: Comparison of feature maps for equivariant and non-equvariant
architectures. In the conventional network (a), the relationship between
feature maps is unclear. In our model (b), a hue shift in the input results in
a predictable hue shift of the internal representations. In the hybrid network
(c), the feature maps appear approximately invariant to hue shift.

###  5.2 Hue-shift 3D Shapes: Local hue equivariance

We evaluate the performance of our models in the presence of local hue shifts
on the 3D Shapes classification dataset [32]. The dataset consists of RGB
images of 3D shapes; the color of the shape, the floor, and the walls vary
across examples, as do the scale and orientation of the shape.

To test the performance of our models, we partition the dataset into four
subsets: a training set of 48k examples, and a test set of 12k examples. In-
distribution and out-of-distribution test sets differ by color range assigned
to the shape, floor and walls. In the training set and in-distribution testing
set (subset $A$), the color of the wall, the floor and the shape are randomly
selected from the first half of the color space (colors 0-4 as defined in
[32]). The out-of-distribution testing sets are designed to measure robustness
to global hue shift, and local hue shift. In the out-of-distribution testing
set designed to measure robustness to global hue shift (subset $B$), the color
of the wall, the floor and the shape are randomly selected from the second
half of the color space (colors 5-9 as defined in [32]). In the out-of-
distribution testing set designed to measure robustness to local hue shift
(subset $C$), the color of the wall and the floor are randomly selected from
the first half of the color space, and the color of the shape is randomly
selected from the second half of the color space.

Model performance is reported in Table 2. Our hue-equivariant architectures
outperform all baseline models for all test cases. Noteably, our hue-
equivariant architecture with CNN backbone and $\sim$22k parameters
outperforms both the ResNet44 and ResNet18 architectures with $\sim$2M and
$\sim$11M parameters respectively.

Table 2: Classification error on the 3D Shapes dataset. Our networks are marked with (*). The number of parameters for all networks is reported.  Network | Group |  $A$/$A$ |  $A$/$B$ |  $A$/$C$ | Params
---|---|---|---|---|---
ResNet18 | $\mathbb{Z}^{2}$ | 0.00 (0.00) | 38.53 (3.19) | 17.73 (0.90) | 11.17M
Hue-4* | 0.00 | 0.00 | 0.00 | 11.17M
ResNet44 | $\mathbb{Z}^{2}$ | 0.00 (0.00) | 41.34 (1.73) | 22.29 (10.80) | 2.64M
Hue-4* | 0.00 (0.00) | 0.00 (0.00) | 0.00 (0.00) | 2.63M
CNN | $\mathbb{Z}^{2}$ | 0.00 (0.00) | 51.25 (9.59) | 26.66 (19.60) | 20,192
Hue-4* | 0.00 (0.00) | 0.00 (0.00) | 0.00 (0.00) | 21,832
Hue-3* | 0.00 (0.00) | 0.03 (0.04) | 0.04 (0.03) | 19,478
CEConv | 0.00 (0.00) | 0.02 (0.03) | 0.05 (0.04) | 26,441

###  5.3 Camelyon17: Saturation shift in the wild

We evaluate the performance of our model in the presence of natural saturation
shifts on the Camelyon17 classification dataset [33]. The dataset consist of
images of human tissue collected from five different hospitals. Variation in
tissue images results from variation in the data collection and processing
procedures. The dataset consists of 387,490 examples, 302,436 of which are
used for training and 85,054 of which are used for testing. Hospitals 1, 2,
and 3 are represented in the training set, hospital 5 is represented in the
testing set.

We present experimental results for four discretizations of the saturation
space. Our choice of discretizations, i.e.,
$d\in\\{\nicefrac{{1}}{{20}},\nicefrac{{1}}{{10}},\nicefrac{{3}}{{20}},\nicefrac{{1}}{{2}}\\}$,
are determined with consideration of the saturation range of the training set.
For a given choice of discretization $d$, an element $s_{i}\in S_{N}$ shifts
the saturation of an input image by $s_{i}=i*d*255$. We limit computational
expense of lifting and convolution by performing these operations over the
truncated set $\\{s_{-1},s_{0},s_{1}\\}$.

Table 3: Classification error on the Camelyon17 dataset. Our networks are
marked with (*). The number of parameters for all networks is reported.
| ResNet50 | Sat-$d_{1}$* | Sat-$d_{2}$* | Sat-$d_{3}$* | Sat-$d_{4}$*
---|---|---|---|---|---
Error | 28.91 (7.58) | 19.60 (2.08) | 16.08 (2.68) | 24.53 (7.13) | 23.57 (6.04)
Params | 23.5M | 23.3M | 23.3M | 23.3M | 23.3M

Model performance is reported in Table 3. The performance of our saturation-
equivariant models out perform all other models, and the performance of our
hue-equivariant model is on par with the baseline model.

###  5.4 CIFAR-10: Color shift in the wild

We evaluate the performance of our model in the presence of natural hue shifts
on the CIFAR-10 [34] classification dataset. The dataset consists of 60k
examples, 40k of which are used for training, 10k for validation, and 10k for
testing.

Table 4: Classification error on the CIFAR-10 dataset. Our networks are marked
with (*). The number of parameters for all networks is reported.
| ResNet44 | Hue-4* | Sat-$d_{1}$* | Sat-$d_{2}$* | Sat-$d_{3}$*
---|---|---|---|---|---
Error | 7.86 (1.14) | 8.83 (0.64) | 10.45 (1.76) | 10.26 (0.43) | 9.24 (0.27)
Params | 2.64M | 2.63M | 2.55 | 2.55 | 2.55M
| Hue-Sat-$d_{1}$* | Hue-Sat-$d_{2}$* | Hue-Sat-$d_{3}$* | Hue-3* | CEConv
Error | 10.68 (0.78) | 11.13 (0.84) | 11.16 (0.38) | 8.42 (0.39) | 8.86 (0.33)
Params | 2.50M | 2.50M | 2.50M | 2.55M | 1.15M
Figure 4: CIFAR-10 automobile class sorted by hue.

Model performance is reported in Table 4. The performance of our color-
equivariant models are comparable to the baseline model.

While our hue-equivariant model performs on par with the baseline model, the
structure of the representation can be leveraged for tasks that are not
possible with the baseline alone. To illustrate this, we use our model to sort
images in the automobile class by hue (see Figure 4). To produce an hue
ordering, we compute the hue shifted pairwise Euclidean distance between the
penultimate layer hue group feature maps of instances in the automobile class.
Images $x_{1}$ and $x_{2}$ are close in the hue space if

$d(\Phi(x_{1}),\Phi(x_{2}))=\min_{i}d(h_{i}\Phi(x_{1}),\Phi(x_{2})),$ (20)

where $\Phi(x_{i})$ denotes the penultimate layer hue group feature map for
image $i$.

##  6 Conclusion

In this paper we address the challenge of learning hue- and saturation-
equivariant representations. Leveraging the observation that these perceptual
transformations have geometric structure, we propose a group structure for
each, and a group convolutional neural network that is equivariant to the
transformations by design. By encoding perceptual variation in a finite group
structure, we can use learned representations for color based sorting.
Moreover, with an efficient implementation, our network achieves comparable
performance to conventional architectures with comparable parameters.

###  6.1 Limitations and future work

While our method is equivariant to hue and saturation by design, it is not
guaranteed to be equivariant to luminance. However, since luminance is
computed as a linear combination of the red, green and blue channel values of
an input image, if the preprocessing statistics (e.g., data mean) on the
training set are valid for the testing set, luminance is approximately
unchanced.

Additional limitations derive from our choice of equivariant modeling
structure. Group equivariant networks are more computationally expensive than
their conventional counterparts since they require computation of the filter
orbit at each layer. This is compounded with the requirement for a finite
filter orbit which can at best approximate a continuous group. Future work
will investigate a Fourier series representation of the group to resolve both
of the aforementioned challenges.

## References

  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Lenc and Vedaldi [2015] Karel Lenc and Andrea Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 991–999, 2015.
  * Bruna and Mallat [2013] Joan Bruna and Stéphane Mallat.  Invariant scattering convolution networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1872–1886, 2013.
  * Cohen and Welling [2016] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In _International conference on machine learning_ , pages 2990–2999. PMLR, 2016.
  * Gilmer et al. [2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In _International conference on machine learning_ , pages 1263–1272. PMLR, 2017.
  * Qi et al. [2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017.
  * Hinton et al. [2018] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst.  Matrix capsules with em routing.  In _International conference on learning representations_ , 2018.
  * Greydanus et al. [2019] Samuel Greydanus, Misko Dzamba, and Jason Yosinski.  Hamiltonian neural networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Veta et al. [2016] Mitko Veta, Paul J Van Diest, Mehdi Jiwa, Shaimaa Al-Janabi, and Josien PW Pluim.  Mitosis counting in breast cancer: Object-level interobserver agreement and comparison to an automatic method.  _PloS one_ , 11(8):e0161286, 2016.
  * Engilberge et al. [2017] Martin Engilberge, Edo Collins, and Sabine Süsstrunk.  Color representation in deep neural networks.  In _2017 IEEE International Conference on Image Processing (ICIP)_ , pages 2786–2790. IEEE, 2017.
  * De and Pedersen [2021] Kanjar De and Marius Pedersen.  Impact of colour on robustness of deep neural networks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 21–30, 2021.
  * Pakzad et al. [2022] Arezou Pakzad, Kumar Abhishek, and Ghassan Hamarneh.  Circle: Color invariant representation learning for unbiased classification of skin lesions.  In _European Conference on Computer Vision_ , pages 203–219. Springer, 2022.
  * Benton et al. [2020] Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew G Wilson.  Learning invariances in neural networks from training data.  _Advances in neural information processing systems_ , 33:17605–17616, 2020.
  * Kondor and Trivedi [2018] Risi Kondor and Shubhendu Trivedi.  On the generalization of equivariance and convolution in neural networks to the action of compact groups.  In _International Conference on Machine Learning_ , pages 2747–2755. PMLR, 2018.
  * Cohen et al. [2019a] Taco S Cohen, Mario Geiger, and Maurice Weiler.  A general theory of equivariant cnns on homogeneous spaces.  _Advances in neural information processing systems_ , 32, 2019a.
  * Worrall et al. [2017] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow.  Harmonic networks: Deep translation and rotation equivariance.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5028–5037, 2017.
  * Esteves et al. [2017] Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis.  Polar transformer networks.  _arXiv preprint arXiv:1709.01889_ , 2017.
  * Worrall and Welling [2019] Daniel Worrall and Max Welling.  Deep scale-spaces: Equivariance over scale.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Sun and Blu [2022] Zikai Sun and Thierry Blu.  Empowering networks with scale and rotation equivariance using a similarity convolution.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Thomas et al. [2018] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  _arXiv preprint arXiv:1802.08219_ , 2018.
  * Esteves et al. [2019] Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniilidis.  Equivariant multi-view networks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1568–1577, 2019.
  * Batzner et al. [2022] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky.  E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.  _Nature communications_ , 13(1):2453, 2022.
  * Finzi et al. [2020] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson.  Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data.  In _International Conference on Machine Learning_ , pages 3165–3176. PMLR, 2020.
  * Cohen et al. [2019b] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling.  Gauge equivariant convolutional networks and the icosahedral cnn.  In _International conference on Machine learning_ , pages 1321–1330. PMLR, 2019b.
  * Gupta et al. [2023] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka.  Structuring representation geometry with rotationally equivariant contrastive learning.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Chong et al. [2008] Hamilton Y Chong, Steven J Gortler, and Todd Zickler.  A perception-based color space for illumination-invariant image processing.  _ACM Transactions on Graphics (TOG)_ , 27(3):1–7, 2008.
  * Lengyel et al. [2024] Attila Lengyel, Ombretta Strafforello, Robert-Jan Bruintjes, Alexander Gielisse, and Jan van Gemert.  Color equivariant convolutional networks.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Gallier and Quaintance [2020] Jean Gallier and Jocelyn Quaintance.  _Differential geometry and lie groups_ , volume 12.  Springer, 2020.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Burgess and Kim [2018] Chris Burgess and Hyunjik Kim.  3d shapes dataset.  https://github.com/deepmind/3dshapes-dataset/, 2018.
  * Bandi et al. [2018] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al.  From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.  _IEEE Transactions on Medical Imaging_ , 2018.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Wiles et al. [2021] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Krishnamurthy Dvijotham, and Taylan Cemgil.  A fine-grained analysis on distribution shift, 2021.

## Appendix

###  A Datasets

####  A.1 Hue-shift MNIST

We introduce the Hue-shift MNIST dataset to evaluate the performance of our
equivariant architectures in the presence of global hue shift. We show select
examples from the in-distribution and out-of distribution datasets in Figures
5(a) and 5(b).

(a) Dataset $A$ examples

(b) Dataset $B$ examples

Figure 5: Hue-shift MNIST dataset. (a) Examples from the training dataset and
in-distribution testing dataset $A$ are colored with a randomly selected hue
between 0∘ and $240^{\circ}$. (b) Examples from out-of-distribution testing
dataset $B$ are colored with a randomly selected hue between $240^{\circ}$ and
$360^{\circ}$.

####  A.2 Hue-shift 3D Shapes

We introduce the Hue-shift 3D Shapes dataset to evaluate the performance of
our equivariant architectures in the presence of local hue shift. We show
select examples from the in-distribution and out-of distribution datasets in
Figures 6(a)\- 6(c).

(a) Dataset $A$ examples

(b) Dataset $B$ examples

(c) Dataset $C$ examples

Figure 6: Hue-shift 3D Shapes dataset. (a) Examples from the training dataset
and in-distribution testing dataset $A$. The color of the wall, the floor and
the shape are randomly selected from the first half of the color space (colors
0-4). (b) Examples from the out-of-distribution testing dataset $B$. The color
of the wall, the floor and the shape are randomly selected from the second
half of the color space (colors 5-9). (c) Examples from the out-of-
distribution testing dataset $C$. The color of the wall and the floor are
randomly selected from the first half of the color space, and the color of the
shape is randomly selected from the second half of the color space.

###  B Network architecture and training details

In this section we provide architectural details for our equivariant networks,
and training details for each experiment.

All experiments were performed over multiple random seeds to assess the
robustness of the model to initialization. Performance statistics on the Hue-
shift MNIST, Hue-shift 3D Shapes, and CIFAR-10 datasets were computed over
three random seeds (i.e., 1999, 2000, and 2001). Performance statistics the on
Camelyon17 dataset were computed over five random seeds (i.e., 1997, 1998,
1999, 2000, and 2001).

All models were trained on a shared research computing cluster. Each compute
node allocates an Nvidia L40 GPU, 24 core partitions of an Intel Xeon Gold
5320 CPU, and 24GBs of DDR4 3200MHz RDIMMs.

####  B.1 Hue-shift MNIST

For this task, we compare against the Z2CNN architecture proposed in Cohen and
Welling [2016]. Our hue-equivariant architecture has the same number of layers
as Z2CNN, but with a reduced filter count to maintain a similar number of
parameters at each layer. In the final layer of our hue-equivariant
architecture, we perform hue group pooling to yield an hue-invariant
representation.

We train our equivariant networks and the conventional architectures for 5
epochs with a batch size of 128. We optimize over a cross-entropy loss using
the Adam optimizer Kingma and Ba [2014] with $\beta_{1}=0.9$, and
$\beta_{2}=0.999$. We use an initial learning rate of $10^{-3}$ for the
$\mathbb{Z}^{2}$ network, and $10^{-4}$ for the hue-equivariant network. We
train CEConv architectures using the hyperparameters reported in Lengyel et
al. [2024].

####  B.2 Hue-shift 3D Shapes: Local hue equivariance

For this task we compare against the ResNet18 architecture proposed in He et
al. [2016], and the Z2CNN and ResNet44 architectures proposed in Cohen and
Welling [2016]. Our hue-equivariant architectures are designed with the same
network structure as the baselines, but with a reduced filter count to
maintain a similar number of parameters at each layer. In the final layer of
our hue-equivariant architectures, we perform hue group pooling to yield an
hue-invariant representation.

Following Wiles et al. [2021], we train the ResNet architectures for 100k
iterations with a batch size of 128. We optimize over a cross-entropy loss
using SGD with a learning rate of $10^{-2}$. Images are down-sampled by a
factor of 2 to train the CNN architectures. CNN architectures are trained as
described in Section B.1.

####  B.3 CIFAR-10: Color shift in the wild

For this task, we compare against the ResNet44 architecture proposed in Cohen
and Welling [2016]. Our hue-, saturation- and hue-saturation-equivariant
architectures have the same number of layers as ResNet44, but with a reduced
filter count to maintain a similar number of parameters at each layer. In the
final layer of our networks, we perform group pooling to yield group invariant
representation.

We train our equivariant networks and the conventional architectures for 300
epochs, and a batch size of 128. We optimize over a cross-entropy loss using
SGD with an initial learning rate of $10^{-1}$ and a cosine-annealing
scheduler. We train CEConv architectures using the hyperparameters reported in
Lengyel et al. [2024].

####  B.4 Camelyon17: Saturation shift in the wild

For this task, we compare against the ResNet50 architecture He et al. [2016].
Our hue- and saturation-equivariant architectures have the same number of
layers as ResNet50, but with a reduced filter count to maintain a similar
number of parameters at each layer. In the final layer of our hue- and
saturation-equivariant architectures, we perform a group pooling to yield
group invariant representation.

We train our equivariant networks and the conventional architectures for 10k
iterations with a batch size of 32. We optimize over a cross-entropy loss
using the Adam optimizer Kingma and Ba [2014] with an initial learning rate of
$10^{-2}$, $\beta_{1}=0.9$, and $\beta_{2}=0.999$. We train CEConv
architectures using the hyperparameters reported in Lengyel et al. [2024].

###  C Hue and saturation group action

In this section we prove the proposed hue group action and saturation group
action satisfy the axioms for a group action. By definition of a group action,
$\varphi_{h}$ is a group action if it satisfies the following properties:

  1. 1.

For all $g,h\in G$ and all $x\in X$

$\varphi(g,\varphi(h,x))=\varphi(gh,x)$ (21)

  2. 2.

For all $x\in X$

$\varphi(1,x)=x$ (22)

where $1\in G$ is the identity element of G.

In the remainder of this section we refer to equation 21 as ‘property one’ and
equation 22 as ‘property two’.

####  C.1 Hue group action

Here we show that the proposed hue group action is indeed a group action. The
proposed hue group action on the input space is defined:

$\varphi_{h}(h_{i},x)=((x_{h}+h_{i})(\text{mod}\;255),x_{s},x_{l}),$ (23)

in equation 7 of the main text.

First we show the proposed hue group action $\varphi_{h}$ satisfies property
one. For any hue shifts $h_{i}$ and $h_{j}$, we have

$\displaystyle\varphi_{h}(h_{i},\varphi_{h}(h_{j},x))$
$\displaystyle=\varphi_{h}(h_{i},((x_{h}+h_{j})(\text{mod}\;255),x_{s},x_{l}))$
(24)
$\displaystyle=(((x_{h}+h_{j})(\text{mod}\;255)+h_{i})(\text{mod}\;255),x_{s},x_{l}))$
(25) $\displaystyle=(((x_{h}+(h_{j}+h_{i}))(\text{mod}\;255),x_{s},x_{l}))$
(26) $\displaystyle=\varphi_{h}((h_{j}+h_{i}),x),$ (27)

which shows that $\varphi_{h}$ satisfies the property one. Now we show
$\varphi_{h}$ satisfies property two. We have the identity element hue
transformation $h_{0}=0$, and

$\displaystyle\varphi_{h}(h_{0},x)$
$\displaystyle=((x_{h}+h_{0})(\text{mod}\;255),x_{s},x_{l})$ (28)
$\displaystyle=(x_{h},x_{s},x_{l})=x,$ (29)

which shows $\varphi_{h}$ satisfies property two. Having shown that
$\varphi_{h}$ satisfies all properties of a group action, we have that
$\varphi_{h}$ is, in fact, a group action.

####  C.2 Saturation group action

Here we show the proposed saturation group action is a group action. The
proposed saturation group action is defined:

$\varphi_{s}(s_{i},x)=(x_{h},\text{min}(x_{s}+s_{i},255),x_{l}),$ (30)

in equation 9 of the main text. First we show the proposed saturation group
action $\varphi_{s}$ satisfies property one. For any saturation shifts $s_{i}$
and $s_{j}$, we have

$\displaystyle\varphi_{s}(s_{i},\varphi_{s}(s_{j},x))$
$\displaystyle=\varphi_{s}(s_{i},(x_{h},\text{min}(x_{s}+s_{j},255),x_{l}))$
(31)
$\displaystyle=(x_{h},\text{min}(\text{min}(x_{s}+s_{j},255)+s_{i},255),x_{l})$
(32) $\displaystyle=(x_{h},\text{min}(x_{s}+(s_{j}+s_{i}),255),x_{l})$ (33)
$\displaystyle=\varphi_{s}((s_{j}+s_{i}),x),$ (34)

which shows $\varphi_{s}$ satisfies property one. Now we show $\varphi_{s}$
satisfies property two. We have the identity element saturation transformation
$s_{0}=0$, and

$\displaystyle\varphi_{s}(s_{0},x)$
$\displaystyle=(x_{h},\text{min}(x_{s}+s_{0},255),x_{l})$ (35)
$\displaystyle=(x_{h},x_{s},x_{l})=x,$ (36)

which shows $\varphi_{s}$ satisfies property two. Having shown that
$\varphi_{s}$ satisfies all properties of a group action, we have that
$\varphi_{s}$ is, in fact, a group action.


# Web Content Extraction – a Meta-Analysis of its Past and Thoughts on its
Future

Tim Weninger<EMAIL_ADDRESS>Rodrigo Palacios<EMAIL_ADDRESS>Valter
Crescenzi<EMAIL_ADDRESS>Thomas Gottron<EMAIL_ADDRESS>Paolo Merialdo<EMAIL_ADDRESS>∗†Department of Computer Science and
Engineering, University of Notre Dame, Notre Dame, Indiana, USA
§Institute for Web Science and Technologies, University of Koblenz-Landau,
German
¶Università Roma Tre, Dipartimento di Ingegneria, Rome, Italy

###### Abstract

In this paper, we present a meta-analysis of several Web content extraction
algorithms, and make recommendations for the future of content extraction on
the Web. First, we find that nearly all Web content extractors do not consider
a very large, and growing, portion of modern Web pages. Second, it is well
understood that wrapper induction extractors tend to break as the Web changes;
heuristic/feature engineering extractors were thought to be immune to a Web
site’s evolution, but we find that this is not the case: heuristic content
extractor performance also tends to degrade over time due to the evolution of
Web site forms and practices. We conclude with recommendations for future work
that address these and other findings.

##  I Introduction

The field of content extraction, within the larger pervue of data mining and
information retrieval, is primarily concerned with the identification of the
main text of a document, such as a Web page or Web site. The principle
argument is that tools that make use of Web page data, _e.g._ , search
engines, mobile devices, various analytical tools, demonstrate poor
performance due to noise introduced by text not-related to the main content
Finn2001 ; J:FI:2012:MartinG .

In response the field of content extraction has developed methods that extract
the main content from a given Web page or set of Web pages, _i.e._ , a Web
site Liu2003 ; Zhai2005 . Frequently, these content extraction methods are
based on pattern mining and the construction of well-crafted rules. In other
cases, content extractors learn the general skeleton of a Web page by
examining multiple Web pages in a Web site Kushmerick1999 ; Bar-Yossef2002 ;
Crescenzi2001 ; Crescenzi2008 . These two classes of content extractors are
referred to as heuristic and wrapper induction respectively; and each class of
algorithms have their own merits and disadvantages. Generally speaking,
wrapper induction methods are more accurate than heuristic approaches, but
require some amount of training data in order to initially induce an
appropriate wrapper. Conversely, heuristic approaches are able to function
without an induction step, but are generally less accurate.

The main criticism of content extraction via wrapper induction is that the
learned rules are often brittle and are unable to cope with even minor changes
to a Web pages’ template Gibson2005 . When a Web site modifies its template,
as they often do, the learned wrappers need to be refreshed by re-computing
the expensive induction step. Certain improvements in wrapper induction
attempt to induce extraction rules that are more robust to minor changes
Davulcu2000 ; Dalvi2009 ; ParameswaranDGR11 , but the more robust rules only
delay the inevitable Chidlovskii2006 .

Heuristic approaches are often criticised for their lack of generality. That
is, heuristics that may work on a certain type of Web site, say a news agency,
are often ill suited for business Web sites or message boards, etc. Most
approaches also ignore the vast majority of the Web pages that dynamically
download or incorporate content via external reference calls during the
rendering process, _e.g._ , CSS, JavaScript, images.

The goal of this paper is not to survey the whole of content extraction, so we
resist the temptation to verbosely compare and contrast the numerous published
methods. Rather, in this paper we make a frank assessment on the state of the
field, provide an analysis of content extraction effectiveness over time, and
make recommendations for the future of content extraction.

In this paper we make three main contributions:

  1. 1.

We define the vectors of change in the function and presentation of content on
the Web,

  2. 2.

We examine the state of content extraction with respect to the ever changing
Web, and

  3. 3.

We perform a temporal evaluation on various content extractors

Finally, we call for a change in the direction of content extraction research
and development.

The evolution of Web practices is the central to the theme of this paper. A
scientific discipline ought to strive to have some invariance in the results
over time. Of course, as technology changes, our study of it must also change
as well. With this in mind, one way to determine the success of a model is to
measure its stability or durability as the input changes over time.

To that end, we present the results of a case study that compares content
extraction algorithms, both old and new, on an evolving dataset. The goal is
to identify which measures, if any, are invariant to the evolution of Web
practices.

Web site
---
news.bbc.co.uk
cnn.com
news.yahoo.com
thenation.com
latimes.com
entertainment.msn.com
foxnews.com
forbes.com
nymag.com
esquire.com
Table 1: Dataset used in case study. 25 Web pages crawled from each Web site
per lustrum (5-year period), over 4 lustra and 10 Web sites totals 1,000 Web
pages.

To that end, we collected a dataset of 1000 Web pages from 10 different
domains, listed in Table 1, where each domain has a set of pages from years
2000, 2005, 2010, and 2015. There are 25 HTML documents per lustrum (_i.e._.,
5-year period), for a total of 100 documents per Web site. The documents were
automatically and manually gathered from two types of sources:
archives111WayBack Machine - http://archive.org/web/ and the original websites
themselves for the 2015 lustrum.

We review the evolution that has occurred in Web content delivery and
extraction, referring explicitly to recent changes that undermine the
effectiveness of exiting content extractors. To show this explicitly we
perform a large case study wherein we compare the performance over time of
several content extraction algorithms. Based on our findings we call for a
change in content extraction research and make recommendations for future
work.

Figure 1: The Web page of http://www.kdd.org/kdd2015/ fully rendered in a modern Web browser (Left). Web page with JavaScript disabled (Middle). Downloaded Web page HTML, statically rendered without any external content (Right). Most extractors operate on the Web page on the right. | src | link | iframe | script | js | jquery | css | size
---|---|---|---|---|---|---|---|---
2000 | 37.092 | 1.152 | 0.388 | 7.600 | 6.588 | 0.908 | 1.828 | 39,121.90
2005 | 61.812 | 2.528 | 0.408 | 21.200 | 14.280 | 0.944 | 2.612 | 52,633.82
2010 | 57.976 | 10.104 | 1.408 | 44.044 | 24.096 | 1.000 | 10.468 | 81,033.89
2015 | 49.396 | 18.256 | 10.032 | 40.652 | 37.052 | 0.620 | 11.692 | 174,801.64
Table 2: The mean-average occurrences of certain HTML tags and attributes that
represent ancillary source files in our dataset of 1,000 news Web pages over 4
equal sized lustra (5-year periods). The use of external content and client-
side scripting has been growing quickly and steadily.

##  II Evolving Web Practices

We begin with the observation that the content delivery on the Web has changed
dramatically since it was first conceived. The case for content extraction is
centered around the philosophy that HTML is a markup language that describes
how a Web page ought to look, rather than what a Web page contains. Here, the
classic form versus function debate is manifest. Yet, in recent years the Web
has seen a simultaneous marriage and divorce of form and function with the
massive adoption of scripting languages like JavaScript and with the
finalization of HTML5.

In this section we argue that because Web technologies have changed, the way
we perform and evaluate content extraction must also change.

###  II.1 Evolution of Form and Function

JavaScript. Nearly all content extraction algorithms operate by downloading
the HTML of the Web page(s) under consideration, and only the HTML. In many
cases, Web pages refer directly or indirectly to dozens of client side
scripts, _i.e._ , JavaScript files, that may be executed at load-time. Most of
the time content extractors do not even bother to download referenced scripts
even though JavaScript functions can (and frequently do) completely modify the
DOM and content of the downloaded HTML. Indeed, most of the spam and
advertisements that content extraction technologies explicitly claim to catch
are loaded via JavaScript and are therefore not part of most content
extraction testbeds.

CSS. Style sheets pose a problem similar in nature to JavaScript in that
structural changes to the displayed content on a Web site are frequently
performed by instructions embedded in cascading style sheets. Although CSS
instructions are not as expressive as JavaScript functions – they were built
for different purposes – the omission of a style sheet often severely affects
the rendering of a Web page.

Furthermore, many of the content extractors described earlier rely on
formatting hints that live within HTML in order to perform effective
extraction. Unfortunately, the ubiquitous use of CSS removes many of the HTML
hints that extractors depend upon. Using CSS, it is certainly possible that a
complex Web site is made entirely of div-tags.

HTML5. The new markup standards introduced by HTML5 include many new tags,
including main, article, header, etc., meant to specify the semantic meaning
of content. Widespread adoption of HTML5 is in progress, so it is unclear
whether and how the new markup languages will be used or what the negative
side effects will be, if any.

The semantic tags in HTML5 are actually a severe departure form the original
intent of HTML. That is, HTML4 was originally meant to be a markup for the
structure of the Web page, not a description language. Indeed the general lack
of semantic tags is one of the main reasons why content extraction algorithms
were created in the first place.

Further addition of semantics into HTML markup is provided by the schema.org
project. Schema.org is a collaboration among the major Web search providers to
provide a unified description language that can be embedded into HTML4/5 tag
attributes. Web site developers can use these tags to encode what HTML data
represents, for example, a Person-itemtype, which may have a name-itemprop,
can then be used by search engines and other Web-services to built intelligent
analytics tools. Other efforts to encode semantic meaning in HTML can be found
in the Microformats.org project, the Resource Description Framework in
Attributes (RDFa) extension to HTML5, and others.

| itmscp | itmtp | itmprp | sctn | artcl
---|---|---|---|---|---
Mean | 162.2 | 157.8 | 899.0 | 261.0 | 403.4
Median | 65.5 | 54.5 | 374.5 | 25 | 166.5
Table 3: Mean and Median number of occurrences of semantic tags from
schema.org: itemscope, itemtype and itemprop tags, and from HTML5: article and
section found in 2015-subset of the dataset. Semantic tags are only found in
dataset from 2015.

Table 3 shows the mean and median number of Schema.org and HTML5 semantic tags
in our 2015 dataset. We find that 9 out of 10 Web sites we crawled had adopted
the Schema.org tagging system, and that 9 out of 10 Web sites had adopted the
section and article tags from HTML5 (8/10 adopted both Schema.org and HTML5).

The advent and widespread adoption of HTML5 and Schema.org decreases the need
for many extraction tools because the content or data is explicitly marked and
described in HTML.

AJAX. Often, modern Web pages are delivered to the client without the content
at all. Instead, the content is delivered in a separate JSON or XML message
via AJAX. These are not rare cases, as of April 2015, Web Technologies
research finds that AJAX is used within 67% of all Web
sites222http://w3techs.com/technologies/overview/javascript_library/all.
Accessed May 6, 2015.. Thus, it is conceivable that the vast majority of
content extractors over estimate their effectiveness in 67% of the cases,
because a large portion of the final, visually-rendered Web page is not
actually present in the HTML file.

In fact, in our experiments we find that the most frequent last-word found by
many content extractors on NY Times articles is “loading…”

Table 2 shows the mean-average number of occurrences of certain HTML tags and
attributes that represent ancillary source files in our dataset of 1,000 Web
pages. In this table, src refers to the occurrence of the common tag attribute
which can refer to a wide range of file types. link refers to the occurrence
of the <link> HTML tag which frequently (although not necessarily) references
external CSS files. iframe refers to the occurrence of the HTML tag which is
used to embed another HTML document into the current HTML document. script
refers to the occurrence of the HTML tag which is used to denote a client-side
script such as (but not necessarily) JavaScript. js refers to the occurrences
of externally referenced JavaScript files; css similarly refers to the
occurrences of externally referenced CSS files. The jquery column shows the
percentage of Web pages that employ AJAX via the jQuery library; alternative
AJAX libraries were found but their occurrence rates were very small.

In many ways the above observations show that the Web is trending towards a
further decoupling of form from content: JavaScript decouples the rendered DOM
from the downloaded HTML, CSS similarly separates the final presentation from
the downloaded HTML, and AJAX allows for the HTML and extractable content to
be separate files entirely. Yet, despite these trends, most content extraction
methodologies rely on extractions from statically downloaded HTML files.

An example of why this should be considered a bad practice is highlighted in
Figure 1 where the Web page http://kdd.org/kdd2015 is shown rendered in a
browser (at left), rendered without JavaScript (center), and rendered with
only the static HTML document (at right). The information conveyed to the end
user is presented in its complete form in the rendered version; thus, content
extractors should strive to operate within the fully rendered document (at
left), instead of the HTML-only extraction as is the current practice (at
right).

###  II.2 Keeping Pace with the Changing Web

Web presentation has evolved in remarkable ways in a very short time period.
Content Extraction algorithms have attempted to keep pace with evolving Web
practices, but many content extraction algorithms quickly become obsolete.

Counter-intuitively, it seems that as although the number of Web sites has
increased, the variety of presentation styles has actually decreased. For a
variety of reasons, most Web pages within the same Web site look strikingly
similar. Marketing and brand-management often dictate that a Web site
maintains style distinct from competitors, but are similar to other pages in
the same Web site.

Wrapper Induction. The self-similarity of pages in a Web site stem from the
fact that the vast majority of Web sites use scripts to generate Web page
content retrieved from backend databases. Because of the structural similarity
of Web pages within the same Web site, it is possible to reverse engineer the
page generation process to find and remove the Web site’s skeleton, leaving
only the content remaining Kushmerick1999 ; Bar-Yossef2002 ; Crescenzi2001 ;
Crescenzi2008 .

A wrapper is induced on one Web site at a time and typically needs only a
handful of labelled examples. Once trained the learned wrapper can extract
information at near-perfect levels of accuracy. Unfortunately, the wrapper
induction techniques assume that the Web site template does not change. Even
the smallest of tweaks to a Web site’s template or the database schema breaks
the induced wrapper and requires retraining. Attempts to learn robust
wrappers, which are immune to minor changes in the Web page template have been
somewhat successful, but even the most robust wrapper rules eventually break
Dalvi2009 ; Gibson2005 .

Heuristics and Feature Engineering. Rather than learning rigid rules for
content extraction, other works have focused on identifying certain heuristics
as a signal for content extraction. The variety of the different heuristics is
impressive, and the statistical models learned through a combination of
various features may, in many cases, perform comparable to extractors based on
wrapper induction.

Each methodology and algorithm was invented at a different time in the
evolution of the Web and looked at different aspects of the Web content. From
the myriad of options we selected 11 algorithms from different time periods.
They are listed in Table 4.

Algorithm |  | Year
---|---|---
Body Text Extractor (BTE) | Finn2001  | 2001
Largest Size Increase (LSI) | Han2001  | 2001
Document Slope Curve (DSC) | Pinto2002  | 2002
Link Quota Filter | Mantratzis2005  | 2005
K-Feature Extractor (KFE) | Debnath2005  | 2005
Advanced DSC (ADSC) | Gottron2008  | 2007
Content Code Blurring (CCB) | Gottron2008a  | 2008
RoadRunner∗ (RR) | Crescenzi2008  | 2008
Content Extraction via Tag Ratios (CETR) | Weninger2010a  | 2010
BoilerPipe | Kohlschutter2010  | 2010
Eatiht | Palacios2015  | 2015
Table 4: Content extraction algorithms, with their citation and publication
date. ∗RoadRunner is a wrapper induction algorithm; all others are heuristic
methods.

Each algorithm, heuristic, model or methodology is predicated on the form and
function of the Web at the time of its development. Each was evaluated
similarly on the state of the Web that existed at the time, presumably, just
before publication. Furthermore, each algorithm does not consider JavaScript,
CSS, or AJAX changes to the Web page, therefore the majority of the Web page
may not actually be present for extraction, as is the case in Figure 1.

(a) _ca_. 2000

(b) _ca_. 2005

(c) _ca_. 2010

(d) _ca_. 2015

Figure 2: F1-measure for various extractor cohorts by lustrum (5-year period).

##  III Case Study

We present the results of a case study that compares content extraction
algorithms, both old and new, on an evolving dataset. The goal is to test the
performance variability of content extractors over time as Web sites evolve.
So, for each Web page of each lustrum of each Web site, a gold-standard
dataset was created manually by the second author. Each Web content extractor
attempted to extract the main content from the Web page.

For the first seven content extractors in Table 4, we used the implementation
from the CombineE System Gottron2008 . The Eatiht, BoilerPipe and CETR
implementations are all available online. BoilerPipe provides a standard
implementation as well as an article extractor (AE), Sentence extractor (Sen),
an extractor trained on data from KrdWrd-Canola
corpus333https://krdwrd.org/trac/raw-
attachment/wiki/Corpora/Canola/CANOLA.pdf, and two “number of words”
extractors: a decision tree induced extractor (W) and a decision tree induced
extractor manually tuned to at least 15 words per content area (15W). CETR has
a default algorithm as well as a threshold option based on the standard
deviation of the tag ratios (Th), and a 1 dimension clustering option (1D).
See the respective papers for details.

An attempt was made to induce wrappers using the Roadrunner wrapper induction
system Crescenzi2008 , which was successful on each set of 25 Web pages, but
performed very poorly on the proceeding lustrum. Wrapper-breakage is a well
known problem for wrapper induction techniques Dalvi2009 ; Gibson2005 . A
five-year window is too long for any wrapper to continue to be effective. Thus
Roadrunner had to be trained and evaluated slightly differently. In this case
we manually identified Web pages that have very similar HTML structure and
learned a wrapper on those few pages. In most cases 90-95% of the Web pages in
a single domain could be used to generate a wrapper, but in 2 Web sites only
about half of the Web pages were found to have the same style and were useful
for training. We used the induced wrapper to extract the content from the Web
pages on which it was trained.

We emphasize that our methodology follows that of most content extraction
methodologies. Namely, we download the raw HTML of the Web page and perform
content extraction on _only_ that static HTML. We further emphasize that this
ignores a very large portion of the overall rendered Web page – renderings
that are increasingly reliant on external sources for content and form via
AJAX, stylesheets, iframes, etc. The disadvantages of this methodology are
clear, but we are beholden to them because the existing extractors require
only static HTML.

####  III.0.1 Evaluation

We employ standard content extraction metrics to compare the performance of
different methods. Precision, recall and F1-scores are calculated by comparing
the results/output of each methods to a hand-labeled gold standard. The
F1-scores are computed as usual and all results are calculated by averaging
each of the metrics over all examples.

The main criticism of these metrics is that they are likely to be inflated.
This is because every word in a document is considered to be distinct even if
two words are lexically the same. This makes it impossible to align words with
the original page and therefore forces us to treat the hand labeled content
and automatically extracted content as a bag of words, _e.g._ i.e., where two
words are considered the same if they are lexically the same. The bag of words
measurement is more lenient and as a result scores may be inflated.

The CleanEval competition has a hand-labeled gold standard as well from a
shared list of 684 English Web pages and 653 Chinese Web pages downloaded in
2006 by “[collecting] URLs returned by making queries to Google, which
consisted of four words frequent in an individual language”Baroni2008 .
CleanEval uses a different approach when computing extraction performance.
Their scoring method is based on a word-at-a-time version of the Levenshtein
distance between the extraction algorithm and the gold standard divided by the
alignment length.

|  |  |  | Lustrum
---|---|---|---|---
|  |  |  | 2000 | 2005 | 2010 | 2015
|  | Extractor | Year | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | Acc
|  | All Text | – | 45.65 | 100 | 45.65 | 38.33 | 100 | 38.33 | 25.78 | 100 | 25.78 | 20.14 | 100 | 20.14
Cohort |  2000 | BTE | 2001 | 76.36 | 92.74 | 82.74 | 58.15 | 89.37 | 72.22 | 34.32 | 88.67 | 53.92 | 20.67 | 85.47 | 44.05
LSI | 2001 | 83.37 | 89.79 | 87.87 | 64.19 | 88.71 | 77.32 | 43.35 | 80.47 | 65.7 | 23.66 | 77.51 | 48.56
DSC | 2002 | 85.42 | 83.25 | 86.2 | 66.88 | 82.98 | 77.55 | 46.37 | 75.74 | 71.09 | 23.89 | 72.97 | 50.15
2005 | KFE | 2005 | 74.21 | 75.88 | 83.34 | 50 | 69.95 | 70.71 | 35.28 | 63.45 | 65.4 | 19.78 | 64.92 | 47.72
LQF | 2005 | 71.11 | 93.81 | 81.49 | 56.57 | 92.39 | 72.23 | 38.68 | 85.44 | 61.94 | 20.91 | 84.47 | 45.24
ADSC | 2007 | 74.39 | 92.91 | 83.68 | 57.68 | 91.36 | 73.51 | 36.95 | 86.74 | 59.28 | 20.27 | 85.59 | 44.25
CCB | 2008 | 85.28 | 86.95 | 88.26 | 65.79 | 84.91 | 77.78 | 44.45 | 77.06 | 68.95 | 22.92 | 74.43 | 48.72
RR | 2008 | 81.97 | 92.11 | 88.96 | 70.73 | 89.75 | 86.54 | 61.32 | 76.57 | 70.82 | 47.75 | 86.25 | 79.70
2010 | CETR | 2010 | 86.74 | 85.18 | 88.98 | 76.05 | 82.08 | 85.13 | 59.01 | 81.32 | 80.79 | 54.66 | 67.86 | 88.03
CETR-1D | 2010 | 85.3 | 85.62 | 88.55 | 76.23 | 82.64 | 85.41 | 59.31 | 80.35 | 80.89 | 56.57 | 67.13 | 87.78
CETR-Th | 2010 | 89.92 | 81.95 | 89.16 | 82.1 | 77.52 | 86.74 | 65.63 | 78.31 | 84.21 | 57.42 | 72.76 | 89.6
BP | 2010 | 93.51 | 85.92 | 92.26 | 91.84 | 82.64 | 92.45 | 79.12 | 75.72 | 88.86 | 83.17 | 68.84 | 93.74
BP-AE | 2010 | 94.76 | 87.11 | 92.86 | 92.97 | 84.25 | 92.54 | 94.99 | 69.39 | 91.35 | 85.79 | 63.21 | 92.96
BP-Sen | 2010 | 97.37 | 84.43 | 92.78 | 97.47 | 81.71 | 93.53 | 97.19 | 66.84 | 91.26 | 89.06 | 61.33 | 93.09
BP-Canola | 2010 | 93.43 | 87.36 | 92.58 | 88.09 | 84.56 | 90.97 | 77.33 | 77.47 | 88.71 | 68.74 | 71.47 | 92.02
BP-15W | 2010 | 94.5 | 83.7 | 91.55 | 89.09 | 80.62 | 90.22 | 82.1 | 74.04 | 89.37 | 73.89 | 68.51 | 92.4
BP-W | 2010 | 91.45 | 88.83 | 92.51 | 88.31 | 86.12 | 91.76 | 81.79 | 78.54 | 89.58 | 83.31 | 70.84 | 93.97
2015 | eatiht | 2015 | 81.89 | 76.3 | 80.17 | 82.04 | 80.04 | 83.76 | 93.75 | 69.18 | 91.13 | 88.48 | 62.93 | 94.39
Table 5: Precision, recall and accuracy breakdown by lustrum (_i.e._ , the
5-year period in which data was collected) and cohort (_i.e._ , the set of
extractors that were developed in the same time period)

###  III.1 Results

First, we begin with a straightforward analysis of the results of each
algorithm on the dataset. Figure 2(a)–2(d) shows the F1-measure for each
lustrum, _i.e._ , each 5-year time period, organized by extractor cohort. For
example, the BTE-extractor was published in 2001, and is therefore part of the
_ca_. 2000 cohort of extractors; it’s performance is illustrated in Figure
2(a). The eatiht-extractor was published in 2015 and is therefore part of the
_ca_. 2015 cohort of extractors, and is illustrated in Figure 2(d).

The shape of the performance curves in Figure 2(a)–2(d) over time exactly
demonstrate the primary thesis of this paper: extractors quickly become
obsolete.

Indeed, Figure 3 averages the F1-measure of each cohort and plots their
aggregate performance together. We can clearly see that all of the extractor
cohorts begin at approximately the same performance on Web page data from the
year 2000, but the performance quickly falls as the form and function of the
Web pages change. As a naive baseline, we also measure the results if all non-
HTML text was extracted and treated as content; in this case, the F1-measure
is buoyed by the perfect recall score, but the precision and accuracies are
bad as expected.

2015-extractors are most invariant to changes in the Web because the
developers likely created the extractor knowing the state of the Web in 2015
and with an understanding of the history of the Web. 2010-extractors perform
well on data from 2010 and prior, but were unable to adapt to unforeseen
changes that appeared in 2015. Similarly extractors from 2005 performed well
on data from 2005 and prior, but did not predict Web changes and quickly
became obsolete.

The F1-measure is arguably the best single performance metric to analyze this
type of data, however, individual precision, recall and accuracy
considerations may be important to various applications. The raw scores are
listed in Table 5.

Figure 3: Mean average F1 measure per cohort over each lustrum.

We find that extractors from 2000 and 2005 have a steep downward trend and
extractors from 2010 also has a downward trend, although not as steep. Only
the 2015 extractor performs steadily. These results indicate that changes Web
design and implementation has adversely affected content extraction tools.

The semantic tags found in new Web standards like HTML5 may be one solution to
the falling extractor performance. Table 6 demonstrates surprisingly good
extraction performance by extracting only (and all of) the text inside the
article tag from the 2015 lustrum as the articles content. Compared to the
results from the complex algorithms shown in Table 5 the simple HTML5
extraction rule shows reasonable results with very little effort.

| Precision | Recall | Accuracy
---|---|---|---
Mean | 57.3 | 67.3 | 82.4
Median | 60.7 | 72.3 | 83.4
Table 6: Extraction results using only HTML5 article tags.

This further demonstrates that the nature of the Web is changing, and as a
result, our thinking about content extraction must change too.

###  III.2 Discussion

The main critique of wrapper induction methods is that they frequently require
re-training. In response many heuristic/feature engineering approaches have
been developed that are said to not require training and simply work out of
the box.

These results underscore a robustness problem in Web content extraction.
Ideally, Web science research should be at least partially invariant to
change. If published content extractors are to be adopted and widely used they
ought to be able to withstand changing Web standards. Wrapper induction
techniques admit this problem; however, we find that heuristic content
extractors are prone to obsolescence as well.

##  IV Conclusions

We conclude by recapping our main findings.

First, we put into concrete terms the changes to the form and function of the
Web. We argue that most content extraction methodologies, by their reliance on
unrendered, downloaded HTML markup, do not count very large portion of the
final rendered Web page. This is due to the Web’s increasing reliance on
external sources for content and data via JavaScript, iframes, and so on.

Second, we find that although wrapper induction techniques are prone to
breakage and require frequent retraining, the heuristic/feature engineering
extractors studied in this paper, which argued to not require training at all,
are also quickly obsolete.

###  IV.1 Recommendations for future work

We argue that the two findings presented in this paper be immediately
addressed by the content extraction community, and we make the following
recommendations.

  1. 1.

Future content extraction methodologies should be performed on completely
rendered Web pages, and should therefore be created as Web browser extensions
or with a similar rendered-in-browser setup using a headless browser like
PhantonJS, etc. This methodology will allow for all of the content to be
loaded so that it may be fully extracted. A browser-based content extractor
might operate similar to the popular AdBlock software, but only render content
rather than simply removing blacklisted advertisers. Aside from executing
JavaScript and gathering all of the external resources, a browser based
content extractor would also allow for a visual-DOM representation that may
improve extraction effectiveness.

  2. 2.

Future content extraction studies should examine Web pages and Web sites from
different time periods to measure the overall robustness of the dataset. This
is a difficult task and is perhaps contrary to the first recommendation
because the external data from old Web pages may not be be easily rendered
because the external may sources cease to exist. Nevertheless, it is possible
to denote Web pages which have not changed via through Change Detection and
Notification (CDN) systems Chakravarthy2006  or through Last-Modified or ETag
HTTP headers.

  3. 3.

With the adoption semantic tags in HTML5, such as section, header, main, etc.,
as well as the creation of semantic attributes within the schema.org
framework, it is important to ask whether content extraction algorithms are
still needed at all. Many Web sites have mobile versions that streamline
content delivery and a large number of content provides have content
syndication systems or APIs that deliver pure content. It may be more
important in the near future to focus attention on structured data extraction
from lists and tables Pimplikar2012 ; Gupta2009 ; Limaye2010 ; Cafarella2008 ;
Cafarella2011  and integrating that data for meaningful analysis.

Content extraction research has been an important part of the history and
development of the Web, but this area of study would greatly benefit by
considering these recommendations as they would lead to new approaches that
are more robust and reliable.

## References

  * (1) Z. Bar-Yossef and S. Rajagopalan.  Template detection via data mining and its applications.  In WWW, page 580, New York, New York, USA, May 2002. ACM Press.
  * (2) M. J. Cafarella, A. Halevy, and J. Madhavan.  Structured data on the web.  Communications of the ACM, 54(2):72–79, 2011.
  * (3) M. J. Cafarella, A. Halevy, D. Z. Wang, E. Wu, and Y. Zhang.  Webtables: Exploring the power of tables on the web.  Proc. VLDB Endow., 1(1):538–549, Aug. 2008.
  * (4) S. Chakravarthy and S. C. H. Hara.  Automating change detection and notification of web pages.  In 17th International Conference on Database and Expert Systems Applications, pages 465–469. IEEE, 2006.
  * (5) B. Chidlovskii, B. Roustant, and M. Brette.  Documentum eci self-repairing wrappers: Performance analysis.  In Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data, SIGMOD ’06, pages 708–717, New York, NY, USA, 2006. ACM.
  * (6) V. Crescenzi, G. Mecca, and P. Merialdo.  RoadRunner: Towards Automatic Data Extraction from Large Web Sites.  VLDB, pages 109–118, Sept. 2001.
  * (7) V. Crescenzi and P. Merialdo.  Wrapper inference for ambiguous web pages.  Applied Artificial Intelligence, 22(1&2):21–52, 2008.
  * (8) N. Dalvi, P. Bohannon, and F. Sha.  Robust web extraction.  In SIGMOD, page 335, New York, New York, USA, June 2009. ACM Press.
  * (9) H. Davulcu, G. Yang, M. Kifer, and I. V. Ramakrishnan.  Computational aspects of resilient data extraction from semistructured sources (extended abstract).  In Proceedings of the Nineteenth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS ’00, pages 136–144, New York, NY, USA, 2000. ACM.
  * (10) S. Debnath, P. Mitra, and C. L. Giles.  Automatic extraction of informative blocks from webpages.  In SAC, page 1722, New York, New York, USA, Mar. 2005. ACM Press.
  * (11) A. Finn, N. Kushmerick, and B. Smyth.  Fact or fiction: Content classification for digital libraries.  In DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries, 2001.
  * (12) D. Gibson, K. Punera, and A. Tomkins.  The volume and evolution of web page templates.  In Special interest tracks and posters of the 14th international conference on World Wide Web - WWW ’05, page 830, New York, New York, USA, May 2005. ACM Press.
  * (13) T. Gottron.  Combining content extraction heuristics.  In iiWAS, page 591, New York, New York, USA, Nov. 2008. ACM Press.
  * (14) T. Gottron.  Content Code Blurring: A New Approach to Content Extraction.  In DEXA TIR Workshop, pages 29–33. IEEE, Sept. 2008.
  * (15) R. Gupta and S. Sarawagi.  Answering table augmentation queries from unstructured lists on the web.  Proc. VLDB Endow., 2(1):289–300, Aug. 2009.
  * (16) W. Han, D. Buttler, and C. Pu.  Wrapping web data into XML.  ACM SIGMOD Record, 30(3):33, Sept. 2001.
  * (17) C. Kohlschütter, P. Fankhauser, and W. Nejdl.  Boilerplate detection using shallow text features.  In Proceedings of the third ACM international conference on Web search and data mining - WSDM ’10, page 441, New York, New York, USA, Feb. 2010\. ACM Press.
  * (18) N. Kushmerick.  Learning to remove internet advertisements.  In Proceedings of the third annual conference on Autonomous Agents, pages 175–181. ACM, 1999.
  * (19) G. Limaye, S. Sarawagi, and S. Chakrabarti.  Annotating and searching web tables using entities, types and relationships.  Proc. VLDB Endow., 3(1-2):1338–1347, Sept. 2010.
  * (20) B. Liu, R. Grossman, and Y. Zhai.  Mining data records in Web pages.  In SIGKDD, page 601, New York, New York, USA, Aug. 2003. ACM Press.
  * (21) C. Mantratzis, M. Orgun, and S. Cassidy.  Separating XHTML content from navigation clutter using DOM-structure block analysis.  In HT, page 145, New York, New York, USA, Sept. 2005. ACM Press.
  * (22) Marco Baroni, F. Chantree, A. Kilgarriff, and S. Sharoff.  CleanEval: a competition for cleaning webpages.  In International Language Resources and Evaluation, 2008.
  * (23) L. Martin and T. Gottron.  Readability and the Web.  Future Internet, 4:238–252, 2012.  Special Issue Selected Papers from ITA 11.
  * (24) WayBack Machine - http://archive.org/web/.
  * (25) http://w3techs.com/technologies/overview/javascript_library/all. Accessed May 6, 2015.
  * (26) https://krdwrd.org/trac/raw-attachment/wiki/Corpora/Canola/CANOLA.pdf.
  * (27) R. Palacios.  Eatiht.  http://rodricios.github.io/eatiht/, 2015.
  * (28) A. G. Parameswaran, N. N. Dalvi, H. Garcia-Molina, and R. Rastogi.  Optimal schemes for robust web extraction.  PVLDB, 4(11):980–991, 2011.
  * (29) R. Pimplikar and S. Sarawagi.  Answering table queries on the web using column keywords.  Proceedings of the VLDB Endowment, 5(10):908–919, 2012.
  * (30) D. Pinto, M. Branstein, R. Coleman, W. B. Croft, M. King, W. Li, and X. Wei.  QuASM: a system for question answering using semi-structured data.  In JCDL ’02: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, pages 46–55, New York, NY, USA, 2002. ACM Press.
  * (31) T. Weninger, W. H. Hsu, and J. Han.  CETR – Content Extraction via Tag Ratios.  In WWW, page 971, New York, New York, USA, Apr. 2010. ACM Press.
  * (32) Y. Zhai and B. Liu.  Web data extraction based on partial tree alignment.  In WWW, page 76, New York, New York, USA, May 2005. ACM Press.