# Compact Binary Fingerprint for Image Copy Re-Ranking

Nazar Mohammad  Junaid Baber  Maheen Bakhtyar  Bilal Ahmed Chandio  Anwar Ali
Sanjrani
Department of CS and IT, University of Balochistan

###### Abstract

Image copy detection is challenging and appealing topic in computer vision and
signal processing. Recent advancements in multimedia have made distribution of
image across the global easy and fast: that leads to many other issues such as
forgery and image copy retrieval.

Local keypoint descriptors such as SIFT are used to represent the images, and
based on those descriptors matching, images are matched and retrieved.
Features are quantized so that searching/matching may be made feasible for
large databases at the cost of accuracy loss. In this paper, we propose binary
feature that is obtained by quantizing the SIFT into binary, and rank list is
re-examined to remove the false positives. Experiments on challenging dataset
shows the gain in accuracy and time.

_Keywords_ SIFT Ranking $\cdot$ Binary Feature Space $\cdot$ Image Copy
Retrieval

##  1 Introduction

Internet is the most powerful tool used in these days. Visual graphics like
videos and images are the most attention seekers rather than text paragraphs,
so that image manipulation and editing softwares are easily available online
or offline. That’s why thousands of images are being edited and shared among
different servers of social media and other sites such as
Facebook111https://www.facebook.com/, Flicker222http://www.flickr.com/, and
ImageShake333http://imageshack.us/. Image databases are expanding rapidly in
scope and immensity which is being a major cause of in-efficient image
recuperation such as image infringement and extravagant. Many researchers
proclaimed different elucidations to fix this problem; image copy retrieval
and partial-duplicate image retrieval overcome the problem in some such way.
Image copy retrieval and partial-duplicate image retrieval somehow solve these
problems. Image copy is defined as a segment of image derived from another
image usually by means of various challenging transformations such as pattern
addition, content deletion, modification of image contents (such as aspect
ratio, color, contrast, encoding), cam-cording, and morphing.

Local attributes are extensively used for image and video applications.
Initially, key-points are detected from the images and represented by some
robust descriptors such as SIFT Lowe (2004), GLOH Mikolajczyk and Schmid
(2005a), and CSLBP Heikkila et al. (2009).

The affine regions are decisive in the calculation of the descriptors over the
key-points; the descriptors performance around the key-points is very delicate
while assessment of the affine regions Mikolajczyk and Schmid (2005a).
Therefore, the descriptors computation only prosperous and immense
recapitulation affine regions are preferred Mikolajczyk and Schmid (2004,
2005a). The descriptors centered from the affine regions computation to key-
points are recapitulation under image transformation, but that is not an
effective solution because two totally different patches can have same
consistency. The previous research presented about the descriptors have lack
in robustness and distinctiveness Baber et al. (2015). The robustness of the
descriptors is generally perceived as the limit of descriptors to make due in
picture clamor, and unmistakable characterizes the ability of the descriptors
to see the difference between two picture fixes or surface. The descriptors
processed from various scenes or settings might be viewed as comparative
(absence of uniqueness), or comparative descriptors might be considered as
different because of image clamor (absence of robustness); the comparability
between descriptors is generally registered by the separations between them in
include space.

Similarity between pair of images is computed by the distance between the
local keypoint descriptors. Since, one image may have several thousand
keypoints, so finding the nearest neighbor of single keypoint in other image,
and then repeating the steps for all other keypoints, makes it computationally
very expensive. To make searching feasible on large databases, features are
quantized. The most common feature quantization is bag of visual word model
(BoVW) Philbin et al. (2007a); Wu et al. (2009); Zhou et al. (2010); Baber. et
al. (2014). In BoVW, image is represented by histogram of visual words present
in that image, visual words are basically the centroids that are obtained by
SIFT features clustering. The BoVW model is computationally efficient compared
to raw SIFT based retrieval, whereas, SIFT has better accuracy compared to
BoVW. Geometrical connection between key-points and their descriptors are used
in the field of image copy detection Fergus et al. (2003); Liu et al. (2008);
Morioka and Satoh (2010, 2011). It has been argued that the performance of
local features is significantly improved from bag of visual words to bag of
pair of visual words Liu et al. (2008). Geometrical connection between key-
points and their descriptors are used in the field of image copy detection
Philbin et al. (2007a); Wu et al. (2009); Zhou et al. (2010). Many researchers
trying to improve their methods or algorithms to enhance the performance of
content based copy detection (CBCD) performance in the manner of robustness
and distinctiveness before the quantization of Bag-of-Visual words Xu et al.
(2011a); Ke and Sukthankar (2004a); Ke et al. (2004), or sometimes after BoW
quantization Philbin et al. (2007a); Wu et al. (2009); Zhou et al. (2010).

In this paper, we propose re-ranking of BoVW by compact binary features.
Initially, images are presented by SIFT features, and later quantized into
BoVW model. Initial ranking is obtained by computing the distances between
query image and databases.

Furthermore, attempt to improve the heartiness and uniqueness of local key-
point descriptors for image duplicate identification sorts of uses. The more
robust and distinctive feature descriptors are having a spatial data inside
the local patches. Expanding or decreasing the size of the key-points can be
helpful to accomplish the goal and makes the descriptor vector more efficient.
We tentatively described and shows the presentation of improved CBCD, and the
descriptors which are impacted from their region are utilized.

##  2 Related work

This area is divided into two sections. In the first section, we described the
different portions of image copy detection and its methods, then the Second
section will clarify about two well-known feature descriptors which are used
in our experiments.

###  2.1 Image copy detection

Watermarking and CBCD techniques are mostly used to prevent copyright issues.
Moreover, CBCD is the correspondent approach for watermarking. Initially,
global features were used for CBCD which will later switched into local
features.

Chang et al. Chang et al. (1998a) propose RIME (Replicated IMage dEtector)
technique through which we can identifies the pilfered duplicate images on the
world wide web by utilizing color space and wavelets Chang et al. (1998b)..
The fundamental sort of changes can be made with a good accuracy by the
system. Kim Kim (2003) utilizes Discrete Cosine Transform (DCT) for CBCD, as
DCT is increasingly vigorous to numerous mutilations and image changes. Images
are changed over into YUV format and _Y_ segment is utilized, as it is
contended that hues don’t have essential influence in image recovery. RIME
effectively identifies the duplicates of the test image with and without
alterations, anyway they neglect to identify the duplicates with $90^{\circ}$
or $270^{\circ}$ turn. The worldwide highlights are effective for basic sorts
of changes, be that as it may, if there should arise an occurrence of serious
changes, the exhibition of worldwide highlights is poor; for instance, if
there should arise an occurrence of trimming, impediment, and angle proportion
change. There are many scheme proposed for the image copy detection and
forgery detection. SIFT and SURF are the mostly used methods for the feature
extraction. Cong lin et al. Lin et al. (2019) proposed a method that combines
SIFT and Local Intensity Order Pattern (LIOP) to adopt good feature results.
As SIFT is invariant feature extractor with image scale, image rotation and
noise removal etc. Meanwhile LIOP descriptor is used for image scale, image
rotation, viewpoint change, image blur and JPEG compression. Local features
have proven to be more resistant and robust for severe image transformations
as compared to the global features. Local features have demonstrated to be
progressively safe and powerful for extreme image changes when contrasted with
the worldwide highlights. Numerous CBCD and image recovery frameworks have
been proposed dependent on SIFT and other local features Chum et al. (2011);
Nistér and Stewénius (2006); Philbin et al. (2007a, 2008); Wu et al. (2009);
Xu et al. (2011a); Zhou et al. (2011, 2010). Xu et al. Xu et al. (2011a) Local
features have demonstrated to be progressively safe and powerful for extreme
image changes when contrasted with the worldwide highlights. Numerous CBCD and
image recovery frameworks have been proposed dependent on SIFT and other local
features. Zhou et al. Zhou et al. (2010) propose a system for fractional copy
image location for huge scope applications by utilizing sack of-visual-words
model. They quantized the SIFT in descriptor space and direction space. They
encode the spatial design of keypoints by XMAP and YMAP technique, which
assists with expelling the exceptions. Be that as it may, their structure is
touchy to computerized blunders, for example, floating or moving of keypoints
because of change which causes to miss the genuine matches. There are
different methods are being used for the detection of keypoints and
descriptors for instance FAST and its variants Rosten and Drummond (2006);
Rosten et al. (2008) are used to find keypoints in real-time in which system
visual features are matched, like parallel tracking and mapping Klein and
Murray (2007). It is effective and finds sensible corner keypoints, in the
spite of the fact that it must be enlarged pyramid business models for scale
Klein and Murray (2008), and in this case, a Harris corner filter Harris et
al. (1988) is used to dismiss edges and give a sensible score. And for the
finding of descriptors, BRIEF Calonder et al. (2010) is a latest feature
descriptor that utilized basic binary tests between pixels in a smoothed image
patch. Furthermore, BRIEF is similar with SIFT in many aspects like its
robustness, blurriness and perspective distortion. Be that as it may, it is
delicate to in-plane rotation. BRIEF as a research method uses binary tests
that train a set of classification trees Calonder et al. (2008). When prepared
on a lot of 500 or so run of the typical keypoints, the trees can be used to
restore a mark for any self-assertive keypoint Calonder et al. (2009).

Joly et al. Joly et al. (2007, 2003) proposed a method which represents 20D
spatio-temporal descriptors to each keyframe that computes about Harris
interest points although it requires a large amount of data to store per
keyframe, possibly hundreds of Harris key points and their descriptors.

Recently, image retrieval based on hashing technique has attracted a focus of
researchers. Torralba et al. Torralba et al. (2008) represents a method that
can learn short descriptors to retrieve same images from a large database. The
technique depends on on a dense 128D global image descriptor, which confines
the way to deal with no geometric/perspective invariance. Jain et al. Jain et
al. (2008) presented a strategy for proficient augmentation of Locally
Sensitive Hashing scheme Indyk (2000) for Mahalanobis distance. Both above
mentioned techniques uses a bit string as a unique symbol of the image. In
such a portrayal, direct impact of comparative images in a solitary container
of the hashing table is impossible and a search over different receptacles
must be performed. This is achievable (or indeed, even favorable) for rough
closest neighbor or range search when the question model is given. Be that as
it may, for grouping undertakings, (for example, discovering all gatherings of
close copied pictures in the database) the bit string portrayal is less
appropriate.

Over the other low dimensional descriptors, SIFT detector and descriptor gives
sufficient performance Mikolajczyk and Schmid (2005b). Therefore, it has been
used worldwide in the field of copy image retrieval Chum et al. (2008);
Auclair et al. (2009), image classification Nister and Stewenius (2006) and
processing medical images Jiang et al. (2014) for instance, tracking the
growth of cancerous existence. Several methods have been proposed to
accelerate the feature indexing process and reduced the Original SIFT
detector’s length. And this can be accomplished by ignoring some patches of
the original descriptor Khan et al. (2011) to get 96D, 64D and 32D descriptors
or by employing principle component analysis to obtain 64D SIFT descriptors Ke
and Sukthankar (2004b).

Another method for managing worldwide feature extraction is identified with
the investigation of data encoded in the image texture. Instances of the
calculations that follow this methodology are Steerable Pyramid Simoncelli and
Freeman (1995), Gabor Wavelet Transform He et al. (2002), Contour-let
Transform Do and Vetterli (2005) or Complex Directional Filter Bank Vo et al.
(2006). Worldwide highlights calculations are commonly viewed as
straightforward and quick, which regularly brings about the absence of in-
variance to change of point of view or light. To beat these issues local
features techniques were presented Górecki et al. (2012). For example, Schmid
Schmid and Mohr (1997) use Harris corner detector to distinguish intrigue
focuses which is insensitive toward change of image direction.

Recki et al. Górecki et al. (2012) proposed a strategy for The primary
commitment of this work is the Ranking by K-means Voting algorithm, whose
reason for existing is to make a positioning of comparable pictures. The
outcomes got in the trial meeting show the benefit of the technique proposed
right now the standard similitude measures, for our situation over the
Euclidean distance. Afra et al. Nürnberger et al. (2016) assessed the
exhibition of the RC-SIFT 64D descriptor to tackle the near duplicate recovery
task in two cases: Firstly, for benchmarks of various size. Besides, utilizing
a similar benchmark yet for various quantities of removed highlights.

Zhihua xu et al. Xu et al. (2011b) presented a powerful content-based image
copy detetion conspire. The fundamental commitment of this paper is in three
parts. (1) By looking into the distributed writing about the geometric
mutilation strong plans, it is obvious that the nearby invariant areas and
worldwide descriptor is an ideal technique to adapt to geometric contortion,
especially, editing and revolution; (2) Construct a progression of powerful,
homogeneous, and bigger size roundabout patches utilizing the SIFT finder; (3)
Adopt the MHD to produce include vector for every round fix.

Wu et al. Wu et al. (2009) propose a structure by taking multiple gathering of
keypoints rather than single keypoint. They utilize a SIFT keypoints alongside
maximally stable extreme areas (MSER) keypoints. MSER keypoints are relative
covariant key-points with higher repeatability contrasted with the SIFT key-
points, these focuses are likewise bigger in scale and generally littler in
number in image. On the off chance that SIFT key-points pack with MSER areas,
at the point, they have better robustness and performance with more
discrimination power of 45% accurately precision in Bag-of-Visual Word model
as compared with the simple SIFT BoW model. Be that as it may, the
computational time for search engine is higher when contrasted with cutting
edge. Wu et al. also presented a way to deal with locate the close duplicate
pictures. The SIFT and MSER feature descriptor were used to locate image
duplication. MSER is an area based methodology, while SIFT is a point feature
location descriptor. Each of them gathering to turns out the feature
extraction to be more distinctive than a single element. The MSER feature
extractor is also widely used for the image retrieval system just like SIFT
invariant feature extractor. Dissimilar to the SIFT feature detector, MSER
recognizes relative covariant stable locales. Each recognized curved locale is
standardized into a round district from which a SIFT descriptor is figured
out.

Ahmed Alzu’bi et al. Alzu’bi et al. (2016) presents a paper about examination
towards discovering smaller and discriminate image portrayals utilizing
worldwide and nearby multi-highlight plot. The led tests give bits of
knowledge into the connection between image features and other recovery
factors, including distance measures, quantization and visual code-books,
recovery speed, and memory necessities. A bank of image features is removed
and afterward defined into minimized image portrayals. The entirety of the
extricated highlights are assessed against eight diverse separation measures
for similarity measurements. The exploratory outcomes show that distinctive
image features and blends give diverse execution. At the last assessment
stage, Euclidean, cosine, and relationship estimates show nearly a similar
effect on both retrieval accuracy and efficiency. The Spear-man separation
measure has indicated the most noteworthy recovery precision for single
neighborhood descriptors contrasted with the joined worldwide or nearby ones.
Be that as it may, it takes more coordinating time than other distance
measures.

###  2.2 Patch based descriptors

There are three famous approaches for keypoint descriptors calculation. In the
first approach raw pixel values based calculations are used, such as CSLBP and
LBP Heikkila et al. (2009); Heikkilä and Pietikäinen (2006). In the second
approach, gradients are computed such as SIFT Lowe (2004); and in the third
approach binary features are used. These binary features can be computed by
quantizating the gradient histograms such as BIG-OH and CARD Baber. et al.
(2014); Ambai and Yoshida (2011), or sometimes comparing the pixel values,
such as BRIEF C. et al. (2010). The SIFT is widely used descriptor, and to
make this section self-contained, SIFT descriptors is explained below.
Scale-Invariant Feature Transform (SIFT) descriptor is the representation of
gradient orientation histograms. SIFT was introduced by David Lowe, after that
it was detailed analyzed and deeply developed in 2004 Lowe (2004) to
accomplish huge enhancements in stability and feature invariance. The standard
SIFT descriptor is built by testing the extents and directions of the image
slope in the region around the intrigue focuses, and portraying the
significant data of the local region by building smoothed direction
histogramsLiu et al. (2019). To compute the SIFT descriptor,the given patch
$P$ is divided into grid of $G_{x}\times G_{y}$. In each cell, the gradient
magnitude, $g(x,y)$, and orientation, $\theta(x,y)$, are computed for each
pixel. The gradient orientations are divided in to 8 different directions and
then histogram of each directions are calculated individually. Each gradient
sample added to its histogram by their gradient magnitude and Gaussian weight.
For Gaussian weight, circular window with a $\sigma$ that is 1.5 times that of
the scale of keypoint is taken Lowe (2004).

##  3 Proposed Re-Ranking Framework

To improve the accuracy efficiently of BoVW, the obtained rank list is re-
evaluated by image to image matching based on binary features. The binary
features transform the Euclidean space to binary space that makes the distance
computation very fast without significant compromise on accuracy. BoVW model
and image to image matching are explained in later in this section.

###  3.1 Image Representation

Mostly, images are in RGB format which are then presented by gray-scale
format, the given image $I\in\mathbb{R}^{m\times n\times c}$, where $m$ and
$n$ denote the dimensions (width and height), and $c$ denotes the channels, in
case of gray-scale format $c=1$ and image then can be defined as
$I\in\mathbb{R}^{m\times n}$. Color is most important and huge element of an
image if it is maintained in most significant way and in a perceptually
situated manner. Content Based Image Retrieval (CBIR) system proposed color as
a most significant element in image retrieval. The color histogram was
acquainted to distinguish the relationship between color circulation of pixels
and spatial relationship of colors Galshetwar et al. (2019). Image matching is
the method to find out geometric correspondence between the at least two
pictures of a similar scene. These images could be of different frames,
different time taken and different viewpoints. There are many detectors used
for the feature extraction and image matching. We have used Harris-Affine
detector for the feature key-point detection. Harris-Affine detector is based
on the Harris-Laplace detector, which is a corner detector mostly used in
image matching. Phase correlation method is applied to overcome the
mismatching issues by estimating the translation parameters of the image and
the image which is used to be matched for the feature detectionZheng and Zheng
(2019). Despite the fact that there is some unobtrusive difference between the
corner detection and feature extraction but both are mean to be same in this
paper. We use Harris Affine (HA) keypoint detector Mikolajczyk and Schmid
(2004). The HA keypoints have corner like structure with less localization
error and higher repeatability as compared to Difference of Gaussian (DOG)
keypoints which is used by SIFT algorithm Mikolajczyk and Schmid (2004). Each
key-point is spoken to by a circular area, which is controlled by scale, slope
edge and second grid Mikolajczyk and Schmid (2004). The detected elliptical
region is mapped to circular region and then normalized into $41\times 41$
pixels in Cartesian grid Heikkila et al. (2009); Mikolajczyk and Schmid
(2005a).

###  3.2 Image-to-Image Matching

Two images can be matched by computing the distance between their feature
vectors, and if the distance is closed enough, then those two images are said
to be similar. Rank list is obtained by the distance score received in
ascending order. However, image to image matching can be easily modeled if
there exists only one feature vector for given image such as gray scale
histogram or RGB histogram. Whereas, this can not be easily modeled for local
keypoints descriptors where one image is represented by set of features. For
example, in case of SIFT, image is represented by feature matrix
$M\in\mathbb{R}^{a\times b}$, where $a$ indicates the dimensions of one
descriptor and $b$ indicates the number of key point. The $b$ varies from
image to image, it depended on the contents of the image. On average, there
are three thousand to four thousand keypoints per image if difference of
Gaussian keypoint detector is used. Two images are comparable on the off
chance that they have numerous basic coordinating points, or the positioned
list is gotten dependent on image-to-image keypoint coordinating. Let $Q$ and
$R$ be the two images with their local keypoints descriptors $Q_{p}$ and
$R_{p}$, respectively. The point pair, $d_{q}\in Q_{p}$ and $d_{r}\in R_{p}$,
, is considered a match if following two conditions meet.

  * •

Nearest neighbor condition

$\mathcal{D}(d_{q},d_{r})=\min_{d_{k}\in R_{p}}\mathcal{D}(d_{q},d_{k})$ (1)

  * •

Reliable match

$\mathcal{D}(d_{q},d_{r})<\min_{d_{s}\in R_{p},s\neq
k}\mathcal{D}(d_{q},d_{s})\times\mathcal{S}$ (2)

Where $\mathcal{D}$ is some distance estimated and $\mathcal{S}$ is the
threshold, which is being utilized for the stable coordinating under commotion
conditions, as recommended by David Lowe (2004).

The feature distance used for the matching algorithms is an arbitrary, which
can be measured by any distance matching models. There is one distance
calculating model used widely to compute the distance matching in the field of
computer vision is Euclidean distance, which is presented mathematically as:

$\mathcal{D}(d_{q},d_{r})=\sqrt{\sum_{i=1}^{a}(d_{q}(i)-d_{r}(i))^{2}}$ (3)

Where $a$ represents the dimensions of the vector, i.e., 128 in case of SIFT.

###  3.3 Bag of Visual Word Model

The computational cost of image-to-image matching is very high as SIFT is high
dimensional vector. For image representation Bag-of-Visual-Words (BoVW) model
gave an extensive rise to capturing the local pattern variations. Basically
the BoVW model is famous in text retrieval from different domains and have
great efficiency of providing accurate results. It is generally motivated by
Bag-of-Words (BoW) model. Typically a content documents, reports and so on
have part of significant words a that can be spoken to by an feature vector
for different word checks, Similarly, a BoVW model speak to the image
retrieval approach is utilized to portrayed an image and its number of
occurrences in different visual wordsKulkarni et al. (2019). The computational
cost for each key-point from the first image to its nearest neighbor in the
other image is very expensive. Moderating the databases can be used by image-
to-image matching models. Bag of Visual Word (BoVW) model is used widely for
the accurate and feasible searches results in the large databases.

The BoVW model, $\mathbb{B}$, is basically the quantizer which quantizes the
descriptor $d\in\mathbb{R}^{b}$

$\begin{split}\mathbb{B}&:\mathbb{R}^{b}\rightarrow[1,K]\\\
&d\rightarrow\mathbb{B}(d)\end{split}$ (4)

The $\mathbb{B}$ quantizes descriptor $d\in\mathbb{R}^{b}$ to any of the $K$
integer cluster index, known as visual word. Visual words are quantized of
different local key-points where the histogram of different visual words are
calculated from the large image datasets. In this situation, they suggested to
keep the value of $K$ greater. In case its keeps the smaller value it will
give robust result but not in particular way and in the opposite case where
$K$ is even larger than it makes the model more particular but less robust. In
the current days, researchers have used number of values lies between 1
million to 1.5 million. Amato et al. (2016); Philbin et al. (2007b). In the
BoVW model the dataset of 2 to 3 thousand images contains approximately 1.0 to
1.5 million vocabulary keypoints makes the model very inadequate Amato et al.
(2016); Philbin et al. (2007b).

Flat K-mean Sivic and Zisserman (2003) or hierarchical K-mean Nister and
Stewenius (2006) clustering are commonly used to learn the vocabulary of
visual words.

---
(a)
(b)
Figure 1: CBCD and Oxford-5K dataset samples. (a) shows the CBCD dataset
sample of 10 different scenes with their 10 copies, and (b) shows the
Oxford-5K dataset sample of 4 landmark with their copies.

###  3.4 Compact Binary Fingerprint: BiSIFT

Finding the similar images using SIFT is very expensive task, as the the
descriptors contains floating values. SIFT is not only computationally
expensive for finding the similar images but also storage intensive. We
quantize the SIFT into binary vector, Binary SIFT (BiSIFT), similar to BIG-OH,
and then compute the image similarity. The BIG-OH binarizes each gradient
histogram on the spatial grid of $4\times 4$ and then concatenated at the end,
whereas, BiSIFT binarizes the whole vector at once. Experiments show that
there is negligible lose in accuracy but significant gain in computation. For
given SIFT descriptor $d\in\mathbb{R}^{b}$, that can be described as
$d=\\{f_{1},f_{2},\ldots,f_{b}\\}$, where $b=128$, the
$d_{B}\in\mathbb{B}^{b-1}$ is obtained which is
$d_{B}=\\{f^{B}_{1},f^{B}_{2},\ldots,f^{B}_{b-1}\\}$ by

$\begin{array}[]{l}f_{i}^{B}=\left\\{\begin{array}[]{l
l}1&\quad\text{if}\;\;f_{i}^{B}\geq f_{i+1}^{B}\\\ 0&\quad\text{otherwise}.\\\
\end{array}\right.\end{array}$ (5)

where $i\in\\{1,2,\ldots,b-1\\}$. The $d_{B}$ is 128-Bits binary vector, and
it is equivalently efficient as BIG-OH but comparatively simple.

##  4 Experimental Analysis

This section presents the dataset used for the experiments and evaluation
metrics followed by the experimental results.

###  4.1 Datasets

Three different datasets are used for the evaluation. The first dataset is
Synthetic dataset which is used to show the efficiency of the descriptors for
similarity computation and storage. The second and third dataset are used to
the show effectiveness of proposed binary descriptor.

####  4.1.1 Synthetic Dataset

For the computational speedup tests, we utilize engineered information
comprising of uniform irregular bytes for all descriptors, since genuine
information isn’t expected to assess crude match score figuring execution.
This dataset is used only to assure the distance calculation of the raw
descriptor about the total required time for the execution. Two different
Synthetic descriptors are generated, the first Synthetic descriptors are in
floating points, and second Synthetic descriptors are binary descriptors.

|
---|---
(a) | (b)
Figure 2: Number of descriptors per image on each dataset. There are total
745563 number key points together and 682.26 on average on CBCD dataset and
13800780 key points together and 2726 on average on the Oxford-5K dataset.

####  4.1.2 Copy Detection Dataset

The second dataset, CBCD, is provided by Zhou et al. Zhou et al. (2010). This
dataset is taken about 36 different camera scenes with total 1088 number of
images. The pictures are seriously misshaped by testing changes normally
experienced in duplicate location situation. The changes were made using the
camcorder by playing a video of one pictures on the screen but it which was an
image inside another image. The more transformation of an image like adding
different designs, JPEG compressions, change of brightness, editing,
obscuring, picture flipping, content inclusion, zoom in or out, change of
viewpoint and decreasing of an image quality. All of these changes occurring
alongside of the moving picture, differentiate changing and picture twisting.
Figure 1(a) gives one picture haphazardly chose from every scene with their
duplicates.

The third dataset is the Oxford-5K dataset Philbin et al. (2007a). This is the
mostly used dataset for the image recovery assessment test. It includes 55
questions pictures of 11 tourists spots including an aggregate of 5063 images
gathered from Flickr444http://www.flickr.com/, Figure 1(b) shows the 4
landmark with their copies.

We have also used an unlabeled dataset of the Paris 100k dataset. This dataset
is used for BoVW model learning. Philbin et at. Philbin et al. (2007a)
explained in detail about these datasets and where it can be found with the
original work details of Oxford 5K and Paris 100K.

The visualization of CBCD and Oxford-5K along with description of number of
images can be found in Figure 2.

|
---|---
(a) | (b)
Figure 3: Computational gain to compute the nearest neighbor for one
descriptor in varying database size. (a) shows the different possibilities to
compute the distance of integer SIFT and BiSIFT, and (b) shows the actual gain
of computation between integer SIFT and BiSIFT.

###  4.2 Experiment on Synthetic Dataset

Synthetic dataset is used to illustrate the computational gain for finding the
nearest neighbor (NN) of one keypoint descriptor in large dataset. The
original SIFT is floating point value that requires 512 bytes/ descriptor Lowe
(2004). There are number of different implementation of SIFT, VLFEAT
555https://www.vlfeat.org/ is one of them. Using VLFEAT API, SIFT requires 128
Bytes/ descriptor as each value is 8-Bit unsigned integer. The 128-Bytes
implementation of SIFT is very fast compared to original SIFT implementation
at the cost of little accuracy loss. The original SIFT takes 252 Seconds to
find the NN of given keypoints from the descriptor size of 500K points,
whereas, integer implementation takes only 0.04 seconds. The hamming space is
further far faster than the integer Euclidean space. That is why original SIFT
is quantized into binary, known as BiSIFT, to make NN searching faster. Figure
3 shows the computational gain of BiSIFT. In Figure 3, the first two curves
are the variants of equation 3, the third curve is the sum of the differences
of binary vectors, and the last curve shows the lookup based implementation of
finding the distances, the idea of lookup based search is inspired by BIG-OH
Baber. et al. (2014).

###  4.3 Experiment on Information Retrieval

This Experiment evaluate the effectiveness of the BiSIFT. As stated earlier,
the BiSIFT achieved competitive performance with significant computational
gain, the computational gain is shown in the Figure 3. Two datasets, CBCD and
Oxford-5K, are taken for the experiments. Precision, recall, and mean average
Precision (mAP) are used as evaluation matrics for this experiment.

Precision ($P$): Precision computes the ratio between correctly retrieved and
total retrieved images. The correctly retrieved are basically the true
positives (TP), and total retrieved is the sum of true positives and
incorrectly retrieved. The incorrectly retrieved are basically the false
positives (FP). It can be computed as follow:

$P=\dfrac{\text{TP}}{\text{TP}+\text{FP}}$ (6)

Recall ($R$): Recall computes the ratio between correctly retrieved and total
correct. The total correct refers to the total possible true positives of
given query image should have retrieved, that can be model as TP + FN, where
FN indicates the false negative (images could not be retrieved). It can be
computed as follow:

$R=\dfrac{\text{TP}}{\text{TP}+\text{FN}}$ (7)

mean Average Precision (mAP): An average precision is computed for each theme,
each theme has more than one queries, and finally average of the average is
calculated, denoted as mAP Philbin et al. (2007a).

Table 1 show the mAP of BiSIFT along with SIFT, BIG-OH, and BoVW models. In
BoVW mode, vocabulary of size 100K is learned on Paris 100K dataset dataset
using k-mean clustering. It can be seen that original SIFT is better than
BoVW, but BoVW is far efficient compared to the SIFT. On Oxford-5K dataset,
average time to search one query image takes 4 minutes on Dell server with 32
GB of RAM. Figure 4 shows the time of each query taken on Oxford-5K dataset to
be searched in 5K images. Since, every image have different number of
keypoints, therefore, the time for each query is different. Whereas, in case
of BoVW model, each image is represented by single feature: normalized
histogram of visual words present in the image. The BoVW model takes only 0.9
seconds for one query image to search. The feature size of BoVW is
$100000\times 5063$, where rows indicates the vocabulary size and column
indicates the number of images.

It can be seen that in Table 1 that the accuracy using BoVW is decreased. To
improve the accuracy of BoVW, spatial verification is applied on top $X$
images in the rank-list either by using RANSAC or other techniques Philbin et
al. (2007a); Zhou et al. (2010). Instead of applying geometric or spatial
verification, image to image matching is applied on $X$ images. Since, image
to image is more effective but computationally expensive, therefore, it should
not be applied on whole dataset but only on $X$ top images on the rank-list,
where $X=30$ in the experiments. Image to image based varification improves
the accuracy of BoVW with little more time of overhead. It takes only 0.1
seconds using BiSIFT and 1.27 seconds using SIFT, it can be seen that the
accuracy is improved when BoVW is piped with SIFT or BiSIFT.

Table 1: mAP of BiSIFT and SIFT | BoVW |  | BoVW +
---
SIFT
| BoVW +
---
BiSIFT
| BoVW +
---
BIG-OH
SIFT | BiSIFT | BIG-OH
CBCD | 0.47 | 0.57 | 0.58 | 0.60 | 0.61 | 0.59 | 0.60
Oxford-5K | 0.51 | 0.53 | 0.52 | 0.53 | 0.55 | 0.52 | 0.53
Figure 4: Time to search each query in Oxford-5K dataset. There are 55 queries
in Oxford-5K dataset.

##  5 Conclusion

Image retrieval is challenging task in large databases as images are easily
distorted, modified, and forged. To retrieve the copies or similar images,
images are represented by some robust and distinctive local descriptors such
as SIFT. However, local descriptor based image matching and then retrieving is
very expensive task. As shown in the experiments, it takes on average 4
minutes to search one query image on the database of size as long as 5K, where
feature extraction time is not include: only feature matching and sorting time
are included. To make search feasible in real-time, features are quantizes
into smaller features space at the cost of accuracy. BoVW is widely used in
image retrieval for feature quantization. It takes only 0.9 seconds to a query
image on the database of size 5K using BoVW model with vocabulary size of
100K. In this paper, we quantize the SIFT into binary vector as binary space
is faster compared to Euclidean space (Integer values) for searching, as shown
in Figure 3, without compromising on the accuracy, as shown in Table 1.

## Conflict of interest

The authors declare that they have no conflict of interest.

## References

  * Lowe [2004] D. G. Lowe.  Distinctive image features from scale-invariant keypoints.  _IJCV_ , 2004.
  * Mikolajczyk and Schmid [2005a] K. Mikolajczyk and C. Schmid.  A performance evaluation of local descriptors.  _IEEE Trans. on PAMI_ , 2005a.
  * Heikkila et al. [2009] M. Heikkila, M. Pietikainen, and C. Schmid.  Description of interest regions with local binary patterns.  _Pattern Recognition_ , 2009.
  * Mikolajczyk and Schmid [2004] K. Mikolajczyk and C. Schmid.  Scale and affine invariant interest point detectors.  _IJCV_ , 2004.
  * Baber et al. [2015] Junaid Baber, Erum Fida, Maheen Bakhtyar, and Humaira Ashraf.  Making patch based descriptors more distinguishable and robust for image copy retrieval.  In _2015 International Conference on Digital Image Computing: Techniques and Applications (DICTA)_ , pages 1–8. IEEE, 2015.
  * Philbin et al. [2007a] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In _CVPR_ , 2007a.
  * Wu et al. [2009] Zhong Wu, Qifa Ke, Michael Isard, and Jian Sun.  Bundling features for large scale partial-duplicate web image search.  In _CVPR_ , 2009.
  * Zhou et al. [2010] Wengang Zhou, Yijuan Lu, Houqiang Li, Yibing Song, and Qi Tian.  Spatial coding for large scale partial-duplicate web image search.  In _ICMR_ , 2010.
  * Baber. et al. [2014] J. Baber., M. Dailey, S. Satoh, and M. Bakhtyar.  BIG-OH: BInarization of gradient orientation histograms.  _Image and Vision Computing_ , 2014.
  * Fergus et al. [2003] R. Fergus, P. Perona, and A. Zisserman.  Object class recognition by unsupervised scale-invariant learning.  In _CVPR_ , 2003.
  * Liu et al. [2008] D. Liu, Gang Hua, P. Viola, and Tsuhan Chen.  Integrated feature selection and higher-order spatial feature extraction for object categorization.  In _CVPR_ , 2008.
  * Morioka and Satoh [2010] N. Morioka and S. Satoh.  Building compact local pairwise codebook with joint feature space clustering.  In _ECCV_ , 2010.
  * Morioka and Satoh [2011] N. Morioka and S. Satoh.  Compact correlation coding for visual object categorization.  In _ICCV_ , 2011.
  * Xu et al. [2011a] Zhihua Xu, Hefei Ling, Fuhao Zou, Zhengding Lu, and Ping Li.  A novel image copy detection scheme based on the local multi-resolution histogram descriptor.  _Multimedia Tools and Applications_ , 2011a.
  * Ke and Sukthankar [2004a] Y. Ke and R. Sukthankar.  Pca-sift: A more distinctive representation for local image descriptors.  In _Proc. CVPR_ , pages 511–517, 2004a.
  * Ke et al. [2004] Yan Ke, Rahul Sukthankar, Larry Huston, Yan Ke, and Rahul Sukthankar.  Efficient near-duplicate detection and sub-image retrieval.  In _In ACM Multimedia_ , 2004.
  * Chang et al. [1998a] Edward Y. Chang, James Ze Wang, Chen Li, and Gio Wiederhold.  RIME: A Replicated Image Detector for the World-Wide Web.  In _Storage and Retrieval for Image and Video Databases_ , 1998a.
  * Chang et al. [1998b] Edward Y Chang, James Ze Wang, Chen Li, and Gio Wiederhold.  Rime: A replicated image detector for the world wide web.  In _Multimedia Storage and Archiving Systems III_ , volume 3527, pages 58–67. International Society for Optics and Photonics, 1998b.
  * Kim [2003] C. Kim.  Content-based image copy detection.  _Signal Processing: Image Communication_ , 2003.
  * Lin et al. [2019] Cong Lin, Wei Lu, Xinchao Huang, Ke Liu, Wei Sun, Hanhui Lin, and Zhiyuan Tan.  Copy-move forgery detection using combined features and transitive matching.  _Multimedia Tools and Applications_ , 78(21):30081–30096, 2019.
  * Chum et al. [2011] O. Chum, J. Philbin, and A. Zisserman.  Near duplicate image detection: Min-hash and tf-idf weighting.  _Proc. BMVC, 2008_ , 2011.
  * Nistér and Stewénius [2006] D. Nistér and H. Stewénius.  Scalable recognition with a vocabulary tree.  In _CVPR_ , 2006.
  * Philbin et al. [2008] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman.  Lost in quantization: Improving particular object retrieval in large scale image databases.  _CVPR_ , 2008.
  * Zhou et al. [2011] W. Zhou, H. Li, Y. Lu, and Q. Tian.  Large scale partial-duplicate image retrieval with bi-space quantization and geometric consistency.  _Proc. ICASSP, 2010_ , 2011.
  * Rosten and Drummond [2006] Edward Rosten and Tom Drummond.  Machine learning for high-speed corner detection.  In _European conference on computer vision_ , pages 430–443. Springer, 2006.
  * Rosten et al. [2008] Edward Rosten, Reid Porter, and Tom Drummond.  Faster and better: A machine learning approach to corner detection.  _IEEE transactions on pattern analysis and machine intelligence_ , 32(1):105–119, 2008.
  * Klein and Murray [2007] Georg Klein and David Murray.  Parallel tracking and mapping for small ar workspaces.  In _2007 6th IEEE and ACM international symposium on mixed and augmented reality_ , pages 225–234. IEEE, 2007.
  * Klein and Murray [2008] Georg Klein and David Murray.  Improving the agility of keyframe-based slam.  In _European conference on computer vision_ , pages 802–815. Springer, 2008.
  * Harris et al. [1988] Christopher G Harris, Mike Stephens, et al.  A combined corner and edge detector.  In _Alvey vision conference_ , volume 15, pages 10–5244. Citeseer, 1988.
  * Calonder et al. [2010] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua.  Brief: Binary robust independent elementary features.  In _European conference on computer vision_ , pages 778–792. Springer, 2010.
  * Calonder et al. [2008] Michael Calonder, Vincent Lepetit, and Pascal Fua.  Keypoint signatures for fast learning and recognition.  In _European conference on computer vision_ , pages 58–71. Springer, 2008.
  * Calonder et al. [2009] M Calonder, V Lepetit, K Konolige, P Mihelich, and P Fua.  High-speed keypoint description and matching using dense signatures.  _Under review_ , 2, 2009.
  * Joly et al. [2007] Alexis Joly, Olivier Buisson, and Carl Frélicot.  Content-based copy retrieval using distortion-based probabilistic similarity search.  _ieee Transactions on Multimedia_ , 9(2):293–306, 2007.
  * Joly et al. [2003] Alexis Joly, Carl Frélicot, and Olivier Buisson.  Robust content-based video copy identification in a large reference database.  In _International Conference on Image and Video Retrieval_ , pages 414–424. Springer, 2003.
  * Torralba et al. [2008] Antonio Torralba, Rob Fergus, and Yair Weiss.  Small codes and large image databases for recognition.  In _2008 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8. IEEE, 2008.
  * Jain et al. [2008] Prateek Jain, Brian Kulis, and Kristen Grauman.  Fast image search for learned metrics.  In _2008 IEEE Conference on computer vision and pattern recognition_ , pages 1–8. IEEE, 2008.
  * Indyk [2000] Piotr Indyk.  Stable distributions, pseudorandom generators, embeddings and data stream computation.  In _Proceedings 41st Annual Symposium on Foundations of Computer Science_ , pages 189–197. IEEE, 2000.
  * Mikolajczyk and Schmid [2005b] Krystian Mikolajczyk and Cordelia Schmid.  A performance evaluation of local descriptors.  _IEEE transactions on pattern analysis and machine intelligence_ , 27(10):1615–1630, 2005b.
  * Chum et al. [2008] Ondrej Chum, James Philbin, Andrew Zisserman, et al.  Near duplicate image detection: min-hash and tf-idf weighting.  In _BMVC_ , volume 810, pages 812–815, 2008.
  * Auclair et al. [2009] Adrien Auclair, Nicole Vincent, and Laurent D Cohen.  Hash functions for near duplicate image retrieval.  In _2009 Workshop on Applications of Computer Vision (WACV)_ , pages 1–6. IEEE, 2009.
  * Nister and Stewenius [2006] David Nister and Henrik Stewenius.  Scalable recognition with a vocabulary tree.  In _Computer vision and pattern recognition, 2006 IEEE computer society conference on_ , volume 2, pages 2161–2168. Ieee, 2006.
  * Jiang et al. [2014] Menglin Jiang, Shaoting Zhang, Hongsheng Li, and Dimitris N Metaxas.  Computer-aided diagnosis of mammographic masses using scalable image retrieval.  _IEEE Transactions on Biomedical Engineering_ , 62(2):783–792, 2014.
  * Khan et al. [2011] Nabeel Younus Khan, Brendan McCane, and Geoff Wyvill.  Sift and surf performance evaluation against various image deformations on benchmark dataset.  In _2011 International Conference on Digital Image Computing: Techniques and Applications_ , pages 501–506. IEEE, 2011.
  * Ke and Sukthankar [2004b] Yan Ke and Rahul Sukthankar.  Pca-sift: A more distinctive representation for local image descriptors.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 2, pages II–II. IEEE, 2004b.
  * Simoncelli and Freeman [1995] Eero P Simoncelli and William T Freeman.  The steerable pyramid: A flexible architecture for multi-scale derivative computation.  In _Proceedings., International Conference on Image Processing_ , volume 3, pages 444–447. IEEE, 1995.
  * He et al. [2002] Chao He, Yuan F Zheng, and Stanley C Ahalt.  Object tracking using the gabor wavelet transform and the golden section algorithm.  _IEEE transactions on multimedia_ , 4(4):528–538, 2002.
  * Do and Vetterli [2005] Minh N Do and Martin Vetterli.  The contourlet transform: an efficient directional multiresolution image representation.  _IEEE Transactions on image processing_ , 14(12):2091–2106, 2005.
  * Vo et al. [2006] An Phuoc Nhu Vo, Truong T Nguyen, and Soontorn Oraintara.  Texture image retrieval using complex directional filter bank.  In _2006 IEEE International Symposium on Circuits and Systems_ , pages 4–pp. IEEE, 2006.
  * Górecki et al. [2012] Przemysław Górecki, Krzysztof Sopyła, and Paweł Drozda.  Ranking by k-means voting algorithm for similar image retrieval.  In _International Conference on Artificial Intelligence and Soft Computing_ , pages 509–517. Springer, 2012.
  * Schmid and Mohr [1997] Cordelia Schmid and Roger Mohr.  Local grayvalue invariants for image retrieval.  _IEEE transactions on pattern analysis and machine intelligence_ , 19(5):530–535, 1997.
  * Nürnberger et al. [2016] Andreas Nürnberger et al.  Near-duplicate retrieval: a benchmark study of modified sift descriptors.  In _International Conference on Pattern Recognition Applications and Methods_ , pages 121–138. Springer, 2016.
  * Xu et al. [2011b] Zhihua Xu, Hefei Ling, Fuhao Zou, Zhengding Lu, and Ping Li.  A novel image copy detection scheme based on the local multi-resolution histogram descriptor.  _Multimedia Tools and Applications_ , 52(2-3):445–463, 2011b.
  * Alzu’bi et al. [2016] Ahmad Alzu’bi, Abbes Amira, Naeem Ramzan, and Tareq Jaber.  Improving content-based image retrieval with compact global and local multi-features.  _International Journal of Multimedia Information Retrieval_ , 5(4):237–253, 2016.
  * Heikkilä and Pietikäinen [2006] Marko Heikkilä and Matti Pietikäinen.  A texture-based method for modeling the background and detecting moving objects.  _IEEE Trans. on PAMI_ , 2006.
  * Ambai and Yoshida [2011] Mitsuru Ambai and Yuichi Yoshida.  CARD: Compact And Real-time Descriptors.  In _ICCV_ , 2011.
  * C. et al. [2010] Michael C., Vincent L., Christoph S., and Pascal F.  BRIEF: binary robust independent elementary features.  In _ECCV_ , 2010.
  * Liu et al. [2019] Yujie Liu, Deng Yu, Xiaoming Chen, Zongmin Li, and Jianping Fan.  Top-sift: the selected sift descriptor based on dictionary learning.  _The Visual Computer_ , 35(5):667–677, 2019.
  * Galshetwar et al. [2019] GM Galshetwar, LM Waghmare, AB Gonde, and S Murala.  Local energy oriented pattern for image indexing and retrieval.  _Journal of Visual Communication and Image Representation_ , 64:102615, 2019.
  * Zheng and Zheng [2019] Yi Zheng and Ping Zheng.  Image matching based on harris-affine detectors and translation parameter estimation by phase correlation.  In _2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP)_ , pages 106–111. IEEE, 2019.
  * Kulkarni et al. [2019] Pradnya Kulkarni, Andrew Stranieri, and Herbert Jelinek.  Comparing pixel n-grams and bag of visual word features for the classification of diabetic retinopathy.  In _Proceedings of the Australasian Computer Science Week Multiconference_ , pages 1–7, 2019.
  * Amato et al. [2016] Giuseppe Amato, Fabrizio Falchi, and Claudio Gennaro.  On reducing the number of visual words in the bag-of-features representation.  _arXiv preprint arXiv:1604.04142_ , 2016.
  * Philbin et al. [2007b] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In _Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on_ , pages 1–8. IEEE, 2007b.
  * Sivic and Zisserman [2003] Josef Sivic and Andrew Zisserman.  Video google: A text retrieval approach to object matching in videos.  In _null_ , page 1470. IEEE, 2003.


# Challenges and Applications of Automated Extraction of Socio-political
Events from Text (CASE 2023): Workshop and Shared Task Report

Ali Hürriyetoğlu
KNAW Humanities Cluster DHLab
<EMAIL_ADDRESS>
Hristo Tanev
European Commission
<EMAIL_ADDRESS>
Osman Mutlu
Koc University
<EMAIL_ADDRESS>
Surendrabikram Thapa
Virginia Tech
<EMAIL_ADDRESS>
Fiona Anting Tan
National University of Singapore
<EMAIL_ADDRESS>
Erdem Yörük
Koc University
<EMAIL_ADDRESS>
###### Abstract

We provide a summary of the sixth edition of the CASE workshop that is held in
the scope of RANLP 2023. The workshop consists of regular papers, three
keynotes, working papers of shared task participants, and shared task overview
papers. This workshop series has been bringing together all aspects of event
information collection across technical and social science fields. In addition
to contributing to the progress in text based event extraction, the workshop
provides a space for the organization of a multimodal event information
collection task.

##  1 Introduction

Nowadays, the unprecedented quantity of easily accessible data on social,
political, and economic processes offers ground-breaking potential in guiding
data-driven analysis in social and human sciences and in driving informed
policy-making processes. Governments, multilateral organizations, and local
and global NGOs present an increasing demand for high-quality information
about a wide variety of events ranging from political violence, environmental
catastrophes, and conflict, to international economic and health crises
Coleman et al. (2014); Della Porta and Diani (2015) to prevent or resolve
conflicts, provide relief for those that are afflicted, or improve the lives
of and protect citizens in a variety of ways. The citizen actions against the
COVID measures in the period 2020-2022 and the war between Russia and Ukraine
are only two examples where we must understand, analyze, and improve the real-
life situations using such data. Finally, these efforts respond to “growing
public interest in up-to-date information on crowds” as well.
111https://sites.google.com/view/crowdcountingconsortium/faqs

The workshop Challenges and Applications of Automated Extraction of Socio-
political Events from Text (CASE 2023) is held in the scope of the conference
Recent Advances in Natural Language Processing (RANLP). CASE 2023 is the sixth
edition of a workshop series Hürriyetoğlu et al. (2022b, 2021b); Hürriyetoğlu
et al. (2020).

We provide brief notes about the accepted papers, shared tasks, and keynote
speeches in the following sections.

##  2 Accepted papers

This year, all seven submissions were accepted by the program committee. A
quick summary of these papers are provided below.

  * •

Osorio and Vásquez (2023) collect and annotate a dataset in 3 different
granularity: whether a document is related to criminal activities, whether
each sentence is related to lethal behaviors, and each sentence’s membership
to 11 predefined categories of events. Following these granularities, the
authors design three binary classification tasks and apply numerous non-neural
and neural models to the annotated dataset; they observe good performance and
provide analysis for data slices with lower performance.

  * •

Tanev and Longueville (2023) discern “main” location, where the event in
question occurred, versus “secondary” locations, that provide extra context
such as the origin of the protestors and the first event location in a series
of protests. They accomplish this by training a BERT model on news articles
annotated with the main event location (CASE 2021 shared task news dataset).
The secondary locations are all other mentioned locations. They compared their
results to that of two SVM models and a baseline that assumes only the first
sentence contains the main location. Their model outperformed all other
systems with an F1 of 0.8 and accuracy 0.73.

  * •

Delucia et al. (2023) proposes an extension of the Multiple-instance learning
(MIL) framework to better handle a common problem in computational social
science: given noisy reports from text sources, how can researchers identify
if a bag of reports (here, tweets within a country-day) report a true event.
The authors show that MIL improves civil unrest detection over methods based
on simple aggregation. The experiments conducted on hyperparameters (key
instance ratio n and instance supervision b) show an improvement from MIL-n
compared to vanilla MIL and other variants by model selection.

  * •

Mbouadeu et al. (2023) This paper studies the news headlines event linking
task that given a title of news (typically a sentence), maps it to an event
concept from a knowledge graph. The challenge is how to compare different
(zero-shot) models’ performance. They propose a benchmark for the evaluation
and compare multiple models. By comparing three families of approaches (a)
similarity based on rule or embeddings, (b) off-the-shelf entity linking
tools, and (c) prompting Large Language Models, the authors show that the
approach (c) has the best performance even though different approaches could
be complementary to each other.

  * •

Slavcheva et al. (2023) presents a semantic model to structure protest event
ontology, and provides some general description of the practical work with the
Bulgarian data. The paper presents both the modelling framework and the
implementation. The model is a practical application of the Unified Eventity
Representation (UER) formalism, which is based on the Unified Modeling
Language (UML), whose four-layer architecture (i.e., user objects, model,
metamodel, and meta-metamodel) provides flexible means for building the
semantic representations of the language units along a scale of generality and
specificity.

  * •

Tuparova et al. (2023) proposes a method that extends the detection capability
of existing event detection models to new event types. The authors have an
experimental setup for few-shot learning when there is a limited training
sources. Moreover, they provide several analyses on the experimental results.
The main strength is in using low resource (single GPU) in order to fine tune
the model in a reasonable time by providing a small set of samples by
leveraging the transfer learning feature of the pre-trained model. The
performance on the detection of the already known events tends to improve as
well.

  * •

Mutlu and Hürriyetoğlu (2023) The paper propose a solution to address the
issue of data scarcity in closed-domain event extraction. The proposed
solution leverages on the use of a side-product of data annotation campaigns
that are the data containing no annotation, by considering this information as
a discriminant that improves the extraction performance. The authors propose a
multi-task model where they leverage additional data present after the token
annotation process. Experiments are well conducted, by showing the efficacy of
the method using different gradually decreasing dataset dimensions.

##  3 Shared tasks

###  3.1 Task 1: Multilingual Protest News Detection

The performance of an automated system depends on the target event type as it
may be broad or potentially the event trigger(s) can be ambiguous. The context
of the trigger occurrence needs to be considered as well. For instance,
depending on the context, the ‘protest’ event type may or may not be
synonymous with ‘demonstration’. Moreover, the hypothetical cases such as
future protest plans may need to be excluded from the results. Finally, the
relevance of a protest depends on the actors, since only citizen-led events
are in the scope of contentious political events. This challenge becomes even
harder in a cross-lingual and zero-shot setting where training data are not
available in new languages. We tackle the task in four steps and hope state-
of-the-art approaches will yield optimal results.

This shared task was announced as a re-run of the same tasks from CASE 2021
Hürriyetoğlu et al. (2021a) and CASE 2022 Hürriyetoğlu et al. (2022a).
Although it attracted some interest, we did not receive any task description
papers for this edition.

###  3.2 Task 2: Automatically Replicating Manually Created Event Datasets

The purpose of Task 2 is to test the abilities of the event extraction systems
to map events on the World map, by finding the locations where they have taken
place. This year the subtitle of the task was ”Detecting and Geocoding Battle
Events from Social Media Messages on the Russo-Ukrainian War”: The purpose of
the task was to detect armed clash events in Russian and Ukrainian Telegram
messages and to geocode them, i.e., find the location of the battles at the
level of a populated place Tanev et al. (2023a).

Until recently, event geocoding has been considered a topic, covered by the
works in the area of Geoinformation systems. Only in the last years the NLP
community started to consider ML algorithms for geocoding Halterman (2023),
also in the context of event detection Tanev et al. (2023b). In this context,
we can consider our shared task as an evaluation exercise for such event
geocoding systems.

The evaluation of the systems participating at shared task 2 relies on an
original evaluation methodology which compares the battle coordinates, found
by the systems with the locations of such events from a Gold standard data
set.

As a Gold standard this year we used a selected subset of the ACLED event
database Raleigh et al. (2010), covering the first six months start of the
Russo - Ukrainian conflict, namely 24 February 2022 - 24 August 2022 and
considering only the events of type battle.

We provided the participants with English language text messages from Telegram
channels originating from Russia and Ukraine. We gathered nearly 326K original
English Telegram Massages from six Telegram channels.

The Telegarm data is available from a public Github repository
222https://github.com/htanev/RussoUkrainianWarTelegram.

Two systems participated in this year’s evaluation: The top ranked system
relied on a combination of a XLMRoberta Liu et al. (2019) classifier, trained
on ACLED, and a rule based geocoder using the JRC NEROne named entity
recognition system Jacquet et al. (2019). The second ranked system used the
NEXUS Tanev et al. (2008) keyword based event classifier and Mordecai3
geoparser Halterman (2023).

The first system, based on XLMRoberta, achieved a moderate level of
correlation with the ACLED dataset, which is a good result, considering the
possibly low coverage of this year Telegram dataset, regarding Russo Ukrainian
war.

We hypothesize that there is a low coverage of the Telegram dataset this year,
since in the 2021 issue of this task Giorgi et al. (2021), participating
systems achieved much higher correlation with the ACLED Gold standard,
including the NEXUS system. The low correlation can be explained with the low
coverage of the war, where battles are not always reported in the media,
especially in Telegram. There is a lot of imprecise information on the social
media and in contrast ACLED gold standard relies on a wide range of verified
sources, including radio and TV, and manually curates the data.

Our conclusion from this year database replication task was that social media
is not the best source of information about armed conflicts.

###  3.3 Task 3: Event Causality Identification

Causality is a core cognitive concept and appears in many natural language
processing (NLP) works that aim to tackle inference and understanding. We are
interested in studying event causality in the news and, therefore, introduce
the Causal News Corpus Tan et al. (2022b). The dataset comprises of 3,767
event sentences extracted from protest event news, that have been annotated
with sequence labels on whether it contains causal relations or not.
Subsequently, causal sentences are also annotated with Cause, Effect and
Signal spans. Two corresponding subtasks were involved in our shared task: In
Subtask 1, participants were challenged to predict if a sentence contains a
causal relation or not. In Subtask 2, participants were challenged to identify
the Cause, Effect, and Signal spans given an input causal sentence. We hope
that our shared task promotes research on the topic of detection and
extraction of causal events in news.

This year’s competition is the second iteration of the shared task, first
introduced in 2022 Tan et al. (2022a), and uses the latest version of the
Causal News Corpus (CNC-V2), also known as RECESS Tan et al. (2023a). As
compared to V1 comprising of 160 sentences and 183 relations, the V2 contains
1,981 sentences and 2,754 causal relations for Subtask 2. Annotations were
also revised for some examples across both subtasks.

Tan et al. (2023b) provides an overview of the work of the ten teams that
submitted their results to our competition and the six system description
papers that were received. The top F1 score for Subtask 1 was 84.66% by Team
DeepBlueAI, who did not submit a description paper. Team InterosML Patel
(2023) scored a similar high score of 84.36%, and used a two step approach:
first pre-training a baseline RoBERTa model with supervised contrastive loss,
then fine-tuning the model on Subtask 1 itself. The top F1 score for Subtask 2
was 72.79% by Team BoschAI Schrader et al. (2023), who used a sequence tagging
approach to fine-tune BERT-large and RoBERTa-large, and adapted the target
labels to allow prediction of up to three different causal relations per
sentence.

###  3.4 Task 4: Multimodal Hate Speech Event Detection

Hate speech detection is one of the most important aspects of event
identification during political events like invasions Thapa et al. (2022). In
the case of hate speech event detection, the event is the occurrence of hate
speech, the entity is the target of the hate speech, and the relationship is
the connection between the two. Since multimodal content is widely prevalent
across the internet, the detection of hate speech in text-embedded images is
very important.

Given a text-embedded image, task 4 aims to automatically identify the hate
speech and its targets333https://codalab.lisn.upsaclay.fr/competitions/13087.
This task had two subtasks Thapa et al. (2023). In subtask 1, participants
were given a dataset of text-embedded images and the participants had to
classify whether the given image contained hate speech or not. It was tasked
as a binary classification problem of classifying hate speech and non-hate
speech. Similarly, in subtask 2, the participants were given a dataset of
hateful text-embedded images where they had to classify what the targets of
hate speech were. This subtask was posed as a multi-class classification
problem where targets were individual, community, and organization. The
dataset curated by Bhandari et al. (2023) was used in this task. More than 50
participants registered for the competition.

Thapa et al. (2023) presents the overview of the performance of 13 teams who
submitted their scores in subtask 1 and 10 teams who submitted their scores in
subtask 2. The ranking was done on the basis of the macro F1-score. The
competition saw a wide range of methodologies ranging from traditional machine
learning models to powerful transformer architectures. The first team ARC-NLP
Sahin et al. (2023) proposed an ensemble of multilayer perceptions (for
representations from textual and visual encoders) and various boosting
algorithms (using syntactical and Bag-of-words representations) for subtask 1.
The team was able to score an F1-score of 85.65%. Similarly, for subtask 2,
they used Named Entity Recognition (NER) features along with CLIP
representations. An ensemble approach similar to subtask 1 was able to give
them the first position with an F1-score of 76.34%.

Similarly, many teams used transformer-based approaches. Out of the submitted
papers for subtask 1, IIC_Team Singh et al. (2023) ranked at rank 3 (F1-score
84.63%), Ometeotl Armenta-Segura et al. (2023) at rank 6 (F1-score of 80.97%),
and VerbaVisor Esackimuthu and Balasundaram (2023) at rank 8 (F1-score of
78.21%) were able to get the best performances with XLM-Roberta-base,
BertForSequence classification, and ALBERT models respectively. All of them
used the text extracted from the given dataset of text-embedded images using
Google Vision API. In subtask 2, IIC_Team, VerbaVisor, and Ometeotl were able
to get the rank of 3 with an F1-score of 69.73%, rank 4 with an F1-score of
68.05% and rank 7 with F1-score of 56.88% respectively with same models used
in subtask 1.

Often, the visual information is also necessary. The first team utilized both
textual and visual information effectively to get a high F1-score. Two teams
viz. CSECU-DSG and LexicalSquad leveraged both textual and visual information.
CSECU-DSG Aziz et al. (2023) used a combination of BERT and vision
transformers (ViT) Dosovitskiy et al. (2020) to leverage textual and visual
information respectively. They were able to get an F1-score of 82.48% and
65.30% in subtask 1 and subtask 2 respectively. They were placed at the fifth
rank in both subtasks. Similarly, LexicalSquad Kashif et al. (2023)
participated only in subtask 1 where they used XLNet and BERT for textual
features and Inception-V3 for visual features. With this combined
representation, they were able to get an F1-score of 74.96%. This ranked them
at the tenth position in the leaderboard.

Traditional machine learning algorithms were also used by some teams where
they performed decently well. SSN-NLP-ACE and ML_Ensemblers used various
traditional machine learning approaches. SSN-NLP-ACE K et al. (2023) used TF-
IDF features with SVM (with RBF kernel) to get an F1-score of 78.80% in
subtask 1 ranking them in the seventh position in the leaderboard. They used
TF-IDF features with logistic regression for subtask 2 which ranked them at
eighth position with an F1-score of 52.58%. Similarly, ML_Ensemblers used a
variety of algorithms like Naive Bayes, KNN, SVM, and Decision Trees out of
which Naive Bayes performed the best in both subtasks with an F1-score of
42.94% and 43.32% in subtask 1 and subtask 2 respectively. They were able to
secure the rank of 13 and 9 in subtask 1 and subtask 2 respectively.

##  4 Keynotes

Three scholars delivered three keynote speeches that are summarized below.

###  4.1 Using Automated Text Processing to Understand Social Movements and
Human Behaviour

Erdem Yörük’s keynote will describe two large-scale ERC-funded projects that
employs computational social science methods to extract data on protests and
public opinion. The first is the Global Contentious Politics Dataset (GLOCON)
Project. 444https://glocon.ku.edu.tr/ is the first automated comparative
protest event database on emerging markets using local news sources Duruşan et
al. (2022). The countries included in the GLOCON dataset are India, South
Africa, Argentina, Brazil and Turkey. Glocon has been created by using natural
language processing, and machine learning in order to extract protest data
from online news sources. The project develops fully automated tools for
document classification, sentence classification, and detailed protest event
information extraction that performs in a multi-source, multi-context protest
event setting with consistent performances of recall and precision for each
country context. GLOCON counts the number of events such as strikes, rallies,
boycotts, protests, riots, and demonstrations, i.e. the “repertoire of
contention,” and operationalizes protest events by various social groups. The
project has developed a novel bottom-up methodology that is based on a random
sampling of news archives, as opposed to keyword filtering. The high-quality
Gold standard corpus is designed in a way that can accommodate context
variability from the outset as it is compiled randomly from a variety of news
sources from different countries Hürriyetoğlu et al. (2021); Yörük et al.
(2021). The second one, Politus Project, aims at scaling up traditional survey
polls for public opinion research with AI-based social data analytics. Politus
develops an AI-based innovation that combines quantitative and computational
methods to create a data platform that delivers representative, valid,
instant, real-time, multi-country, and multi-language panel data on key
political and social trends. The project will collect content information from
Twitter and process it with AI tools to generate a large set of indicators on
political and social trends through its data platform. The deep learning
models and NLP tools will be designed from the ground up as language-
independent and generalizable systems. The platform will deliver geolocated
hourly panel data on demography, ideology, topics, values, and beliefs,
behavior, sentiment, emotion, attitudes, and stance of users aggregated at the
district level. In this keynote, Dr. Yörük will describe the general
methodology of the projects, including data collection, data analysis, and
their approach for representativeness, which is based on multilevel regression
with post-stratification.

###  4.2 Bulgarian Event Corpus for the Construction of a Bulgaria-centric
Knowledge Graph

The Bulgarian Event Corpus is being constructed within the CLaDA-BG (Bulgarian
National Interdisciplinary Research E-Infrastructure for Bulgarian Language
and Cultural Heritage Resources and Technologies. In the spirit of European
CLARIN and DARIAH) we aim to support researchers in Humanities and Social
Sciences (H&SS) to access the necessary datasets for their research. The
different types of objects of study, representation and search are integrated
on the basis of common metadata and content categories. The approach for
interlinking of the datasets is called contextualization. The implementation
of contextualization in CLaDA-BG will utilize a common Bulgaria-centered
knowledge graph - BGKG. The knowledge facts within BGKG are constructed around
events of different types. Thus, construction of BGKG requires a set of
appropriate language resources for training of Bulgarian language pipeline for
extraction of events from text documents. A key element within these language
resources is the Bulgarian Event Corpus. In the talk I will present the design
of the annotation schema, the annotation process, relation to ontologies and
RDF representation. We have started with the CIDOC-CRM ontology for the
construction of the annotation schema. This ontology provides a good
conceptualization of events motivated by the domain of museums which is
appropriate for our goals. During the design of the annotation schema, we
extended the ontology with new events depending on the content of the corpus.
The documents to be annotated were selected from scientific and popular
publications of the partners within CLaDA-BG and articles from Bulgarian
Wikipedia. The annotation is done on several layers: Named Entities, Events,
Roles, Linking, terms and keywords.

###  4.3 With a little help from NLP: My Language Technology applications with
impact on society

Ruslan Mitkov will present original methodologies developed by the speaker,
underpinning implemented Language Technology tools which are already having an
impact on the following areas of society: e-learning, translation and
interpreting and care for people with language disabilities.

The first part of the presentation will introduce an original methodology and
a tool for generating multiple-choice tests from electronic textbooks. The
application draws on a variety of Natural Language Processing (NLP) techniques
which include term extraction, semantic computing and sentence transformation.
The presentation will include an evaluation of the tool which demonstrates
that generation of multiple-choice tests items with the help of this tool is
almost four times faster than manual construction and the quality of the test
items is not compromised. This application benefits e-learning users (both
teachers and students) and is an example of how NLP can have a positive
societal impact, in which the speaker passionately believes. The latest
version of the system based on deep learning techniques will also be briefly
introduced.

The talk will go on to discuss two other original recent projects which are
also related to the application of NLP beyond academia. First, a project,
whose objective is to develop next-generation translation memory tools for
translators and, in the near future, for interpreters, will be briefly
presented. Finally, a project will be outlined which focuses on helping users
with autism to read and better understand texts. The speaker will put forward
ideas as to what we can do next.

The presentation will finish with a brief outline of the latest (and
forthcoming) research topics (to be) which the speaker plans to pursue and his
vision on the future NLP applications. In particular, he will share his views
as to how NLP will develop and what should be done for NLP to be more
successful, more inclusive and more ethical.

##  5 Conclusion

Many aspects of event information modeling and collection are reported in the
scope of CASE 2023. Hosting a shared task that is on multimodal problem and
having submissions about languages other than English (e.g., Bulgarian) are
distinguishing aspects of this edition.

## References

  * Armenta-Segura et al. (2023) Jesus Armenta-Segura, César Jesús Núñez-Prado, Grigori Olegovich Sidorov, Alexander Gelbukh, and Rodrigo Francisco Román-Godínez. 2023.  Ometeotl@multimodal hate speech event detection 2023: Hate speech and text-image correlation detection in real life memes using pre-trained bert models over text.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Aziz et al. (2023) Abdul Aziz, Md. Akram Hossain, and Abu Nowshed Chy. 2023.  Csecu-dsg@multimodal hate speech event detection 2023: Transformer-based multimodal hierarchical fusion model for multimodal hate speech detection.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Bhandari et al. (2023) Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023.  Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1993–2002.
  * Coleman et al. (2014) Peter T Coleman, Morton Deutsch, and Eric C Marcus. 2014.  _The handbook of conflict resolution: Theory and practice_.  John Wiley & Sons.
  * Della Porta and Diani (2015) Donatella Della Porta and Mario Diani. 2015.  _The Oxford handbook of social movements_.  Oxford University Press.
  * Delucia et al. (2023) Alexandra Delucia, Mark Dredze, and Anna L. Buczak. 2023.  A multi-instance learning approach to civil unrest event detection on twitter.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_.
  * Duruşan et al. (2022) Fırat Duruşan, Ali Hürriyetoğlu, Erdem Yörük, Osman Mutlu, Çağrı Yoltar, Burak Gürel, and Alvaro Comin. 2022.  Global contentious politics database (glocon) annotation manuals.
  * Esackimuthu and Balasundaram (2023) Sarika Esackimuthu and Prabavathy Balasundaram. 2023.  Verbavisor@multimodal hate speech event detection 2023: Hate speech detection using transformer model.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Giorgi et al. (2021) Salvatore Giorgi, Vanni Zavarella, Hristo Tanev, Nicolas Stefanovitch, Sy Hwang, Hansi Hettiarachchi, Tharindu Ranasinghe, Vivek Kalyan, Paul Tan, Shaun Tan, et al. 2021.  Discovering black lives matter events in the united states: Shared task 3, case 2021.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 218–227.
  * Halterman (2023) Andrew Halterman. 2023.  Mordecai3: A neural geoparser and event geolocator.  _working paper_.
  * Hürriyetoğlu et al. (2022a) Ali Hürriyetoğlu, Osman Mutlu, Fırat Duruşan, Onur Uca, Alaeddin Gürel, Benjamin J. Radford, Yaoyao Dai, Hansi Hettiarachchi, Niklas Stoehr, Tadashi Nomoto, Milena Slavcheva, Francielle Vargas, Aaqib Javid, Fatih Beyhan, and Erdem Yörük. 2022a.  Extended multilingual protest news detection - shared task 1, CASE 2021 and 2022.  In _Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , pages 223–228, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2021a) Ali Hürriyetoğlu, Osman Mutlu, Erdem Yörük, Farhana Ferdousi Liza, Ritesh Kumar, and Shyam Ratan. 2021a.  Multilingual protest news detection - shared task 1, CASE 2021.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 79–91, Online. Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2021b) Ali Hürriyetoğlu, Hristo Tanev, Vanni Zavarella, Jakub Piskorski, Reyyan Yeniterzi, Osman Mutlu, Deniz Yuret, and Aline Villavicencio. 2021b.  Challenges and applications of automated extraction of socio-political events from text (CASE 2021): Workshop and shared task report.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 1–9, Online. Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2022b) Ali Hürriyetoğlu, Hristo Tanev, Vanni Zavarella, Reyyan Yeniterzi, Osman Mutlu, and Erdem Yörük. 2022b.  Challenges and applications of automated extraction of socio-political events from text (CASE 2022): Workshop and shared task report.  In _Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , pages 217–222, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Hürriyetoğlu et al. (2021) Ali Hürriyetoğlu, Erdem Yörük, Osman Mutlu, Fırat Duruşan, Çağrı Yoltar, Deniz Yüret, and Burak Gürel. 2021.  Cross-Context News Corpus for Protest Event-Related Knowledge Base Construction.  _Data Intelligence_ , 3(2):308–335.
  * Hürriyetoğlu et al. (2020) Ali Hürriyetoğlu, Vanni Zavarella, Hristo Tanev, Erdem Yörük, Ali Safaya, and Osman Mutlu. 2020.  Automated extraction of socio-political events from news (AESPEN): Workshop and shared task report.  In _Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020_ , pages 1–6, Marseille, France. European Language Resources Association (ELRA).
  * Jacquet et al. (2019) Guillaume Jacquet, Jakub Piskorski, Hristo Tanev, and Ralf Steinberger. 2019.  Jrc tma-cc: Slavic named entity recognition and linking. participation in the bsnlp-2019 shared task.  In _Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing_ , pages 100–104.
  * K et al. (2023) Avanthika K, Mrithula Kl, and Thenmozhi D. 2023.  Ssn-nlp-ace@multimodal hate speech event detection 2023: Detection of hate speech and targets using logistic regression and svm.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Kashif et al. (2023) Mohammad Kashif, Mohammad Zohair, and Saquib Ali. 2023.  Lexical squad@multimodal hate speech event detection 2023: Multimodal hate speech detection using fused ensemble approach.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized bert pretraining approach.  _arXiv:1907.11692_.
  * Mbouadeu et al. (2023) Steve Fonin Mbouadeu, Martin Lorenzo, Ken Barker, and Oktie Hassanzadeh. 2023.  An evaluation framework for mapping news headlines to event classes in a knowledge graph.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Mutlu and Hürriyetoğlu (2023) Osman Mutlu and Ali Hürriyetoğlu. 2023.  Negative documents are positive: Improving event extraction performance using overlooked negative data.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Osorio and Vásquez (2023) Javier Osorio and Juan Vásquez. 2023.  Classifying organized criminal violence in mexico using ml and llms.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Patel (2023) Rajat Patel. 2023.  Interosml @ causal news corpus 2023: Understanding causal relationships: Supervised contrastive learning for event classification.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Raleigh et al. (2010) Clionadh Raleigh, rew Linke, Håvard Hegre, and Joakim Karlsen. 2010.  Introducing acled: An armed conflict location and event dataset.  _Journal of peace research_ , 47(5):651–660.
  * Sahin et al. (2023) Umitcan Sahin, Izzet Emre Kucukkaya, Oguzhan Ozcelik, and Cagri Toraman. 2023.  Arc-nlp at multimodal hate speech event detection 2023: Multimodal methods boosted by ensemble learning, syntactical and entity features.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Schrader et al. (2023) Timo Pierre Schrader, Simon Razniewski, Lukas Lange, and Annemarie Friedrich. 2023\.  Boschai @ causal news corpus 2023: Robust cause-effect span extraction using multi-layer sequence tagging and data augmentation.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Singh et al. (2023) Karanpreet Singh, Vajratiya Vajrobol, and Nitisha Aggarwal. 2023.  IIC_Team@multimodal hate speech event detection 2023: Detection of hate speech and targets using xlm-roberta-base.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Slavcheva et al. (2023) Milena Slavcheva, Hristo Tanev, and Onur Uca. 2023.  On the road to a protest event ontology for bulgarian: Conceptual structures and representation design.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Tan et al. (2022a) Fiona Anting Tan, Hansi Hettiarachchi, Ali Hürriyetoğlu, Tommaso Caselli, Onur Uca, Farhana Ferdousi Liza, and Nelleke Oostdijk. 2022a.  Event causality identification with causal news corpus - shared task 3, CASE 2022.  In _Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , pages 195–208, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Tan et al. (2023a) Fiona Anting Tan, Hansi Hettiarachchi, Ali Hürriyetoğlu, Nelleke Oostdijk, Tommaso Caselli, Tadashi Nomoto, Onur Uca, Farhana Ferdousi Liza, and See-Kiong Ng. 2023a.  RECESS: Resource for extracting cause, effect, and signal spans.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics_ , Bali, Indonesia. Association for Computational Linguistics.
  * Tan et al. (2023b) Fiona Anting Tan, Hansi Hettiarachchi, Ali Hürriyetoğlu, Nelleke Oostdijk, Onur Uca, Surendrabikram Thapa, and Farhana Ferdousi Liza. 2023b.  Event causality identification - shared task 3, case 2023.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Tan et al. (2022b) Fiona Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk, Tadashi Nomoto, Hansi Hettiarachchi, Iqra Ameer, Onur Uca, Farhana Ferdousi Liza, and Tiancheng Hu. 2022b.  The causal news corpus: Annotating causal relations in event sentences from news.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2298–2310, Marseille, France. European Language Resources Association.
  * Tanev and Longueville (2023) Hristo Tanev and Bertrand De Longueville. 2023.  Where ”where” matters : Event location disambiguation with a bert language model.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Tanev et al. (2008) Hristo Tanev, Jakub Piskorski, and Martin Atkinson. 2008.  Real-time news event extraction for global crisis monitoring.  In _Natural Language and Information Systems: 13th International Conference on Applications of Natural Language to Information Systems, NLDB 2008 London, UK, June 24-27, 2008 Proceedings 13_ , pages 207–218. Springer.
  * Tanev et al. (2023a) Hristo Tanev, Nicolas Stefanovitch, Andrew Halterman, Onur Uca, Vanni Zavarella, Ali Hürriyetoğlu, Bertrand De Longueville, and Leonida Della Rocca. 2023a.  Detecting and geocoding battle events from social media messages on the russo-ukrainian war: Shared task 2, case 2023.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_. Association for Computational Linguistics (ACL).
  * Tanev et al. (2023b) Hristo Tanev, Nicolas Stefanovitch, Andrew Halterman, Onur Uca, Vanni Zavarella, Ali Hürriyetoğlu, Bertrand De Longueville, and Leonida Della Rocca. 2023b.  Where ”where”’ matters : Event location identification with a BERT language model.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_. Association for Computational Linguistics (ACL).
  * Thapa et al. (2023) Surendrabikram Thapa, Farhan Jafri, Ali Hürriyetoğlu, Francielle Vargas, Roy Ka-Wei Lee, and Usman Naseem. 2023.  Multimodal hate speech event detection - shared task 4, case 2023.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Thapa et al. (2022) Surendrabikram Thapa, Aditya Shah, Farhan Ahmad Jafri, Usman Naseem, and Imran Razzak. 2022.  A multi-modal dataset for hate speech detection on social media: Case-study of russia-ukraine conflict.  In _CASE 2022-5th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text, Proceedings of the Workshop_. Association for Computational Linguistics.
  * Tuparova et al. (2023) Elena Tuparova, Petar Ivanov, Andrey Tagarev, Svetla Boytcheva, and Ivan Koychev. 2023.  Next: An event schema extension approach for closed-domain event extraction models.  In _Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)_ , Varna, Bulgaria (Hybrid). Association for Computational Linguistics.
  * Yörük et al. (2021) Erdem Yörük, Ali Hürriyetoğlu, Fırat Duruşan, and Çağrı Yoltar. 2021.  Random sampling in corpus design: Cross-context generalizability in automated multicountry protest event collection.  _American Behavioral Scientist_ , 0(0):00027642211021630.