11institutetext: Indraprastha Institute of Information Technology, Delhi,
India
<EMAIL_ADDRESS>
# PicHunt: Social Media Image Retrieval for Improved Law Enforcement

Sonal Goel  Niharika Sachdeva  Ponnurangam Kumaraguru
A V Subramanyam  Divam Gupta

###### Abstract

First responders are increasingly using social media to identify and reduce
crime for well-being and safety of the society. Images shared on social media
hurting religious, political, communal and other sentiments of people, often
instigate violence and create law & order situations in society. This results
in the need for first responders to inspect the spread of such images and
users propagating them on social media. In this paper, we present a comparison
between different hand-crafted features and a Convolutional Neural Network
(CNN) model to retrieve similar images, which outperforms state-of-art hand-
crafted features. We propose an Open-Source-Intelligent (OSINT) real-time
image search system, robust to retrieve modified images that allows first
responders to analyze the current spread of images, sentiments floating and
details of users propagating such content. The system also aids officials to
save time of manually analyzing the content by reducing the search space on an
average by 67%.

##  1 Introduction

First responders across the globe are increasingly using Online Social
Networks (OSN) for maintaining safety and law & order situations in society.
Prior work shows the role of social media to aid first responders like police,
for instance; police can use OSN to obtain actionable information like
location, place, and evidence of the crime [24]. Police have realized the
effectiveness of OSN in various activities such as investigation, crime
identification, intelligence development, and community policing [1, 6, 23].
However, in order to accomplish these goals on OSN, they often needs to
identify tweets and images causing safety issues [6, 36].

Researchers have explored the utility of social media platforms like Twitter
to identify text and network of people leading to law enforcement help [1,
36], similarly images on social media sites also often yield information of
interest to investigators [10]. A study shows that people don’t engage equally
with every tweet, Twitter content of over 2 million tweets by thousands of
verified users over the course of a month was analyzed, and it showed that
tweet with an image present can increase user engagement by 35% [20]. It is
said that an image is worth a thousand words. People with different
backgrounds can easily understand the main content of an image thus,
increasingly becoming preferred media to reach and effect masses.

Often images shared on social media have the potential to hurt religious,
political, communal, caste and other sentiments of a certain section of
society. Such images intimidate people, instigate anger among them which
further leads to law and order situations and security critical scenarios in
the society. For example, in April 2016, a journalist tweeted a morphed image
of an Indian politician touching feet of a foreign country’s king, the image
invited anger and backlash on all social media networks and the political
party filed a complaint against the journalist for misleading public [27].
Another example, in June 2014, obscene pictures of Indian warrior-king
Chhatrapati Shivaji and a late political party chief were posted on Facebook
leading to riots in Maharastra, India. People went on rampage damaging public
and private vehicles, pelting stones leading to severe injuries and even
beating a person death [29].

With this arises the need for first responders to understand, patrol, prevent
the spread of such images for maintaining law and order in the society.
However, police personnel has limited exposure to technology [30] and this
limitation makes it difficult to adopt findings of OSN use by police to
facilitate policing needs [23]. Moreover, researchers have focussed more on
textual content to aid the first responders but analyzing multimedia content
like images and videos on OSN is still largely unexplored [13]. Also, the
tools necessary to retrieve, filter, integrate and intelligently present
relevant images and their information for better safety during security
critical scenarios need to be leveraged [2, 12]. These initiatives motivated
us to propose a real-time image-search system which can serve the first
responders by bridging the gap between research in technologies and solving
real-world problems to improve security during such critical scenarios. But
the modifications done on images like cropping, scaling, stitching image,
wrapping around text, contrast enhancement and changing colors are one of the
biggest challenges faced in image analysis. These challenges often create
barriers to directly access these images and utilize information like image
spread, etc., which makes image retrieval a complex and daunting process. This
paper lies in the intersection of crisis informatics and social media image
utility in the understudied & novel context of law enforcement. The main
contributions of this work are:

  1. 1.

We develop a real-time image search system, which is robust to detect similar
images which are cropped, scaled, blurred, stitched with other images, wrapped
around with text, brightened, or modified using other similar image processing
techniques.

  2. 2.

The system aims at data management for first responders, helping them reduce
search space for images.

  3. 3.

We analyze different techniques for retrieving similar images and
experimentally show that ORB (Oriented Fast and Rotated Brief) in combination
with RANSAC is the state-of-art technique in hand-crafted image features for
identifying similar images.

  4. 4.

We propose a supervised deep CNN model that outperforms state-of-art hand-
crafted techniques to retrieve similar images.

  5. 5.

We created a new human-annotated dataset of images from incidents that created
law and order situations in the
society.111http://precog.iiitd.edu.in/resources.html

##  2 Problem Statement

We now formally define the problem definition and notations. Given an image
$I_{A}$ and set of keywords $K$ by a user, find a set of similar images
$I_{B}=\\{I_{b1},I_{b2},\dots I_{bn}\\}$ using a comparison function $C$, from
a search set of images $I_{C}=\\{I_{c1},I_{c2},\dots I_{cm}\\}$ formed using a
search function $S$ on a social network $S_{N}$, where $n\leq m$:

$I_{C}=S(S_{N},K)$ and $I_{B}=C(I_{A},I_{C})$

The search function $S$ takes a set of keywords, and a search space $S_{N}$ as
input and gives a search set of images $I_{C}$. The comparison function takes
the given image $I_{A}$ and $I_{C}$ as input and returns the required similar
images set $I_{B}$.

We model the image similarity as a classification problem with four phases -
database formation, feature extraction of images, feature comparison and
finally similarity classification (see Fig. 1).

Figure 1: Proposed methodology to find image similarity. The system takes
keywords and an image from the user, using the search API of Twitter, creates
a database of related images. To compare two images, their features are
extracted and fed to a comparison function. Comparison score classifies image
similarity.

##  3 Related Work

Recent studies show an increase in need of OSM as a plausible resource for
first responders [3]. Police have realized the effectiveness of OSM in various
activities such as investigation, identifying crime, intelligence development,
and community policing [3, 22, 26]. Research shows the effectiveness of OSN
during events involving law and order issues like the Boston bombings, Sichuan
earthquake (2008), Haiti earthquake (2009), Oklahoma grassfires (2009), and
Chile earthquake (2010) [8, 9, 15, 28, 35]. Studies have shown how OSN has
been effectively used to aid police to increase community engagement, reduce
crime by getting on actionable information from OSN and improve coordination
between police and citizen [23, 24, 25]. But, these studies focus on the
textual aspect of OSN. Research shows that the role of social media text has
received attention but the role of social media images remains largely
unstudied [13]. However, a report shows that if there is an image attached to
a tweet it increases user engagement by 35% [20].

There are also some studies and tools on social media image analysis in
general context; like, the analysis of 581 Twitter images of the 2011 Egyptian
revolution revealed that more efficacy-eliciting (crowds, protest activities,
national and religious symbols) content is posted by Egyptian users than
emotionally arousing (violent) content [13] . Though there have been studies
on social media image search, but in most of them query is text based; for
instance, a study describes how to use Wikipedia and Flickr content to improve
the match of query text with database vocabulary [18]. They used a combination
of Flickr and Wikipedia query model for query expansion to improve the
accuracy of image retrieval systems [18]. Images are also used to visually
summarise events. In [14], authors propose a technique to find most popular
unique images shared on Twitter related to events like sports, law & politics,
art, culture, etc.. In another study, a tool was developed to show trending
images of an event by extracting images from Twitter’s streaming API using
text as query and then detecting near duplicates using locality sensitive
hashing [11]. Another hybrid approach for image retrieval combines social
relevance by understanding the user’s interest using social site Flickr and
visual relevance by ranking the images in Google search result according to
the interests of the user [2]. A study shows that image search tools such as
TinEye and Google Reverse Image Search are used by journalists to find
duplicates, such as other posts of the same image, and near duplicates, such
as posts before or after potential Photoshop manipulations, to help find fake
posts [37]. Though, some of these studies discuss image retrieval aspects but
most of them take text as the query to retrieve related images, and systems
like TinEye and Google Reverse Image Search gives better results for searches
on the entire web than specifically for social media sites like Twitter in
specific. Further, these tools do not provide knowledge management to first
responders, like users propagating the visual content and sentiments floating
with them. Also, most of the image-retrieval tools discussed in these studies
use basic hand-crafted features for finding image similarity thus, resulting
in comparatively lower accuracy.

Our work focuses on emphasizing on the needs of first responders to analyze
image spread on social media, for which we deeply analyzed different
techniques that can be used for image retrieval with the query as an image and
developed a system to aid them to find & analyze similar images on social
media.

##  4 Data Collection and Annotation

From October, 2015 to February, 2016, we collected data related to 5 events
that created law and order issues in society. To create this dataset, we
collected data from Twitter using Twitter’s search API
222https://dev.twitter.com/rest/public/search, filtering tweets that contain
images, counting to a total of 3,725 images. Fig. 2 shows images which were
viral from these events and are taken as input images for evaluating different
image similarity models to be studied. The details of the events are discussed
below:

  1. 1.

Kulkarni Ink: Black ink was sprayed on an Indian technocrat-turned-columnist
Sudheendra Kulkarni by the members of a political party, ahead of the launch
of former Pakistan’s Foreign Minister book, ‘Neither a Hawk nor a Dove: An In-
sider’s Account of Pakistan’s Foreign Policy’. FIR was lodged against the
political party workers and six workers got arrested [16]. This incidence was
slammed on social media arousing political issues and the images of the man
with black ink on face went viral. We collected 1,905 tweets with images using
the keyword “#Kulkarni”.

  2. 2.

Baba Ram Rahim: An Indian self-proclaimed Godman Baba Ram Rahim posted
pictures posing as Hindu God Vishnu on social media. He was accused by All
India Hindu Student Federation for insulting Lord Vishnu and hurting religious
sentiments of Hindus by dressing up as Lord Vishnu and lodged a complaint
against him [32]. We collected 408 tweets with images using the keyword
“#RamRahim”.

  3. 3.

Lord Hanuman Cartoon: A cartoon tweeted by an Indian politician drew much flak
from other political parties of India and its affiliates for allegedly hurting
religious sentiments. The cartoon purportedly showed Hindu Lord Hanuman clad
in saffron robes and leader of another political party. The political party
lodged a complaint alleging that the cartoon image was posted with the intent
to hurt religious sentiments of Hindus [31]. We collected 664 tweets with
images using keywords like “#JNU” and “Insults Hanuman”.

  4. 4.

Charlie Hebdo cartoon: A cartoon in the French satirical magazine Charlie
Hebdo sparked outrage by publishing a cartoon attempting to satirize the
Syrian refugee crisis. The cartoon imagines Alan Kurdi, the three-year-old
Syrian who died in the sea in September 2015, on the way to Europe, has grown
up to be a sexual abuser [7]. Many called the cartoon was racist and said it
was incredibly bad. We collected 568 tweets with images using the keyword
“#CharlieHebdo”.

  5. 5.

Shani Shingnapur protest: As many as 1,500 women, mostly homemakers and
college students, planned to storm the Shani Shingnapur temple in India on
Republic Day. The protesters wanted to end the age-old humiliating practice of
not allowing women to enter the core shrine area [4]. The image of one of the
activist, giving an interview about the protest on Republic Day to media went
viral. We collected 180 tweets with images related to the keyword
‘#ShaniShingnapur”.

Figure 2: Input images set. These images represents image which could be of
interest to first responders and were viral from the event (a) Kulkarni Ink
(b) Baba Ram Rahim (c) Lord Hanuman cartoon (d) Charlie Hebdo cartoon (e)
Shani Shingnapur, respectively.

Data Annotation
To create the ground truth for testing the models, we obtained labels for all
the 3,725 images. The annotators were given an image from each event (see Fig.
2) and from the respective event’s image database they had to mark images that
were similar and dissimilar to the given image. The broad definition of image
similarity given to them was: images that are cropped, scaled, blurred,
wrapped with text, stitched with other images, having color changes,
brightness changes or contrast enhancements are considered similar. The images
which had agreement from at least two annotators were marked in the final
results. Table 1 shows the dataset after annotation was completed.

Keyword | Total Images | Similar Images | Dissimilar Images
---|---|---|---
#Kulkarni | 1,905 | 354 | 1,551
#RamRahim | 408 | 97 | 311
#Insults Hanuman | 664 | 277 | 387
#CharlieHebdo | 568 | 118 | 450
#ShaniShingnapur | 180 | 70 | 110
Table 1: Data after annotation shows the number of similar and dissimilar
images for each event.

##  5 Similarity Modelling

The retrieval performance of CBIR (Content Based Image Retrieval) system
crucially depends on the features that represent images and their similarity
measurement. In this section, we’ll discuss some of the techniques we studied
to compare the similarity of two images.

###  5.1 Hand-Crafted Features

  1. 1.

Color-based feature similarity

Color histograms are frequently used to compare images [17]. It is often done
by comparing color histograms of images, which eliminates information on the
spatial distribution of colors. The image descriptor is a 3D RGB color
histogram with 8 bins per channel and we compare the descriptor using a
similarity metric. After extracting image descriptors, we calculated
Bhattacharyya distance for comparing the two image’s histogram descriptors.
The lesser is the distance between two color histograms, more similar the two
images are.

  2. 2.

Keypoint-Descriptor-based similarity

Certain parts of an image have more information than others, particularly at
edges and corners. Keypoints can be generated using this information. After
finding the keypoints of an image, next step is to find their descriptors.
Descriptors are fixed length vectors that describe some characteristics about
the keypoints. Next, we compare each keypoint descriptors of one image to each
keypoint descriptors of the other image. Since, the descriptors are vectors of
numbers, we can compare them using different distance metrics. We studied
techniques like DAISY [33], ORB [21] and Improved ORB [38] for computing
keypoint descriptors and then comparing them.

     * –

The _DAISY_ dense image descriptor is based on gradient orientation histograms
similar to the SIFT (Scale Invariant Feature Transform) descriptor. It is
formulated in a way that allows for fast dense extraction which is useful for
bag-of-features image representations [34]. After extracting DAISY descriptors
we calculate the distance between the descriptor vectors of two images using
Brute Force matcher with KNN (K-Nearest Neighbors) as the distance metric. We
use this distance as the score to measure the similarity of two images.

     * –

_ORB_ uses improved FAST for feature detection, and these features are
described using an improved Rotated BRIEF feature descriptor [38]. Since the
speed of FAST and BRIEF are very fast this can be the choice for real-time
systems. ORB is rotational invariant, noise invariant and uses image pyramids
for scale invariance [21]. It returns binary strings to describe feature
points, which is used for feature point matching using Hamming distance. We
use this hamming distance as the score to evaluate the similarity of two
images.

     * –

In order to improve the matches given by ORB we add one extra step. After
comparing the keypoint descriptors given by ORB, we find homography of two
images using RANSAC [38]. We extract top 30 matches having the least distance
and pass these matches to RANSAC to weed out wrong matching points. This
technique is called as _Improved ORB_. RANSAC returns a binary array equal to
size of input matches array, where 0 represents a false match and 1 represents
a true match. We define true ratio as the ratio of true matches returned by
RANSAC to total matches given to it. We use this true ratio to evaluate the
similarity of two images. In the given equation, $t_{r}$ denotes true ratio,
$n$ is the size of matches set passed to RANSAC, $A$ is the array returned by
RANSAC and $A_{i}$ represents the value of array returned by RANSAC at
$i^{th}$ index.

$t_{r}=\frac{\sum_{i=0}^{n-1}A_{i}}{n}$

###  5.2 Trained Features

In this section we discuss one important technique “deep learning” which is
organised in a deep architecture and processes information through multiple
stages. To that end, inspired by recent advances in neural architectures and
deep learning, we choose to address CBIR using deep convolutional neural
network.

For implementation of _Convolutional Neural Network (CNN)_ , we took
inspiration from the architecture discussed in [19]. To create the dataset for
training, we used an unsupervised approach. We extracted around 25K images
from popular news websites and then on each of 25K image, we applied various
image processing operations like cropping, distorting, blurring, scaling,
adding text, stitching image, adding noise, and changing color, to generate
the corresponding similar image. Thus, we created a dataset of 50K images. To
train the model we had 25K pairs of similar and dissimilar images. Similar
approach of transforming images to create dataset was used by Fischer et al.,
2014 [5]. As a pre-processing step we first align the images. We extract SIFT
descriptors to map the matching features of both the images and align the
images by applying a perspective transformation using the homography. We then
club the two aligned images to form a 6 channel image which is passed to the
CNN model.

The model contains 15 layers out of which 6 are trainable. First 5 of
trainable layers are convolutional and one is fully connected. The output of
the last fully connected layer is a 1D vector and is fed to a sigmoid
activation which produces the final output between 0 and 1. Our network
minimizes binary cross entropy loss objective and uses adam optimizer as the
optimizer. The convolutional layers contain 64 filters $122\times 122$ pixels,
32 filters $61\times 61$ pixels, 16 filters $31\times 31$ pixels, 6 filters
$16\times 16$ pixels, 1 filter $16\times 16$ pixels. All the convolution
layers are applied with a stride of 4 pixels. The fully connected layer
contains 1 neuron. All the convolution layers use LeakyRelu and batch
normalization except the first layer which doesn’t use batch normalization and
the last convolution layer directly feeds output to the fully connected layer.
All the layers use a subsampling of $2\times 2$ except the last layer, which
uses a subsampling of $1\times 1$. The initial convolution layers extract the
low level features like edges and gradients. The other convolution layers
learn to compare the extracted features of the two images. The similar areas
of the images get propagated till top and fully connected layers calculate a
similarity score using the high level features.

##  6 Hand-crafted vs. Trainable Features

In this section, we evaluate different techniques discussed in the above
section on the annotated test data described in section 4. To evaluate the
performance of the above-discussed image similarity models we calculate
accuracy of their classification results, which is the ratio of true
classification to the total population.

###  6.1 Histogram

For each of the five events, we plotted the accuracy for Bhattacharyya
distance between color histogram of two images, ranging from 0.1 to 0.9. Fig.
3 (a) shows accuracy for different events at different histogram distance. The
graph shows a lot of variation in the accuracy corresponding to the histogram
distance values. We calculated average variance in accuracy at 3 distance
points on the x-axis (0.2, 0.4, 0.5) as 104.24, which is high making it tough
to choose a distance value as the threshold. For example, If we choose
threshold distance as 0.4, the accuracy for ShaniShignapur and Lord Hanuman
cartoon is close to 60% whereas for Charlie Hebdo it is 90%. Another drawback
of this method is, it lacks spatial information, so images with very different
appearances can also have similar histogram [17].

###  6.2 DAISY

The mean distance between DAISY descriptors of both similar and dissimilar
images was in the range of 0.0 to 0.10. To choose the threshold distance
values between this range, we plotted the accuracy graph for each event at
different threshold value. Fig. 3(b) shows the accuracy vs. threshold plot,
where the threshold is the mean distance between DAISY descriptors of two
images. Since the range of distance returned by DAISY descriptors is very
small there was a very high overlap between distance values of similar and
dissimilar images leading to high error. Due to this high error, the maximum
accuracy achieved overall is less than 88%, at a distance of 0.06 but, at this
point accuracy for remaining events 4 events is less than 50%. We also
calculated the average variance in accuracy at 3 distance points on the x-axis
(0.02, 0.05, 0.06) to be 196.58, which is even higher than the Histogram
technique. Thus, making it hard to choose an optimal threshold value.

Figure 3:  (a) Accuracy for different events at different distance between
histograms of two images. (b) Accuracy for different events for different
distance between DAISY descriptors of two images. Both graphs shows high
variance in accuracy.

###  6.3 ORB

For all the 5 events, we plotted the accuracy for different values of mean
distance between ORB descriptors. We use this plot to choose an optimal
threshold distance value. Fig. 4(a) represents the accuracy plot for different
events at different distance between ORB descriptors. The variation in
accuracy is less as compared to the Histogram and DAISY technique. Average
variance in accuracy at 3 distance points on the x-axis (29, 32, 35) is 17.6.
Also, the accuracy for all the events at threshold distance 29 is above 88%.
This shows that ORB is certainly a better choice than Histogram and DAISY for
our data.

###  6.4 Improved ORB

In Improved ORB, after getting the match set of the descriptors from ORB, we
pass the top thirty matches having least distance to RANSAC, which filters
matches which are true. The threshold value here is the ratio of true matches
returned by RANSAC to the total matches passed (thirty in our case), defined
as true match ratio. Fig. 4(b) shows the accuracy of Improved ORB for
different true match ratio taken as the threshold. Average variance in
accuracy at 3 true match ratio values on the x-axis (0.33, 0.35, 0.37) is 6.2,
which is least among all the techniques discussed till now. Also, after true
match ratio of 0.3, the accuracy for each event is almost constant and is
above 90% for all events. If compared with above methods Improved ORB is
giving best results. Hence, Improved ORB can be termed the state-of-art
technique in hand-crafted features discussed.

Figure 4: (a) Accuracy for different events at different distance values
between ORB descriptors, at distance value 29 on x-axis all the events have
accuracy more than 88% and (b) Accuracy for different events at different true
match ratio returned by Improved ORB, at true ratio of 0.35 all events have
accuracy above 90%.

###  6.5 Trainable Features

We trained a deep CNN model for 25K pairs of similar and dissimilar images,
and to test the model we gave input image and images from annotated set to the
model for different epoch values. To choose optimal epoch for the model, we
plot the accuracy for different events, for models trained on different epoch
values. Fig. 5 shows the accuracy for each event at different epoch value. The
model trained for epoch value 35 is showing an accuracy above 97% for all
events. Also, average variance at three epoch values on the x-axis (30, 35,
40) is 0.71, which is even lower than what we achieved in Improved ORB. Since
the overall accuracy for CNN model is higher and the variance is lesser than
Improved ORB, we can conclude that our proposed CNN model outperforms state-
of-art Improved ORB. In the next section, we’ll see how these two techniques
perform when the input image is modified.

Figure 5: CNN models trained for different epochs and their accuracy for five
test events. For model trained for 35 epochs, accuracy for all events is above
97%.

##  7 Competing on Modified Image

In this section, we analyze the performance of Improved ORB (state-of-art
technique in hand-crafted features) and CNN when the input image is modified.
For all the 5 events, we take different cases of modified images usually seen
on social media sites like Twitter and compare the accuracy of both the image
retrieval techniques. Fig. 6 shows modified images taken from all the events
and Table 2 shows the accuracy of the image retrieval models for these images.
For each event, we picked three kinds of modified images: one with single
modification like cropped or scaled or color changes, second with dual
modification together like cropping & scaling or adding text & scaling, etc.,
and third with multiple modifications together like scaling, cropping, adding
text and stitching image, represented as All in Table 2.

Among different modifications seen, the most common modification technique was
scaling the image. Hence, we calculated the scaling factors of the modified
images and found that as the scaling factor increases, the accuracy dip for
Improved ORB also increases. For example, in Table 2 image ‘a’ is modified by
a scaling factor of $7.42\times 5.23$ and the accuracy of Improved ORB is
84.1% while, for the same image CNN is showing an accuracy of 99.4%. Likewise,
for image ‘c’, ‘m’, ‘n’, ‘o’ also the scaling factor is high leading to low
accuracy in Improved ORB but, the accuracy of CNN in all these cases
outperforms Improved ORB. Another common modification affecting accuracy is
cropping of an image, for instance; image ‘f’ has only the cropped face from
the complete body of Baba Ram Rahim (original picture see in Fig. 2), and
again the accuracy dip in Improved ORB is much higher than CNN. Likewise, is
the case for image ‘d’, cropping reduced the accuracy of Improved ORB while
CNN still outperforms it. Hence, we can conclude that the proposed CNN model
is more robust to modifications like high scaling and high cropping than
state-of-art Improved ORB.

Figure 6: Event-wise modified images. By Row: (1) Kulkarni Ink Event (2) Baba
Ram Rahim (3) Lord Hanuman Cartoon (4) Charlie Hebdo Cartoon (5) Shani
Shingnapur.

Image-Id | Improved ORB | CNN | Modification | Scaling Factor | Event
---|---|---|---|---|---
(a) | 84.1 | 99.4 | Scaled | $7.42\times 5.23$ | Kulkarni Ink
(b) | 95.6 | 99.5 | Text added & scaled | $1.23\times 1.08$ | Kulkarni Ink
(c) | 80.7 | 87.3 | All | $9.61\times 3.87$ | Kulkarni Ink
(d) | 91.2 | 99.5 | Cropped | – | Baba Ram Rahim
(e) | 88.2 | 92.6 | Stitched & scaled | $2.31\times 0.92$ | Baba Ram Rahim
(f) | 77.4 | 93.8 | All | $2.65\times 1.56$ | Baba Ram Rahim
(g) | 96.9 | 99.3 | Scaled | $1.04\times 1.09$ | Lord Hanuman Cartoon
(h) | 90.3 | 99.5 | Text added & scaled | $2.32\times 2.47$ | Lord Hanuman Cartoon
(i) | 92.2 | 92.7 | All | $1.27\times 1.25$ | Lord Hanuman Cartoon
(j) | 96.8 | 97.2 | Background color | – | Charlie Hebdo Cartoon
(k) | 97.2 | 97.6 | Text added & scaled | $1.04\times 1.28$ | Charlie Hebdo Cartoon
(l) | 94.5 | 95.3 | All | $2.02\times 1.54$ | Charlie Hebdo Cartoon
(m) | 90.4 | 95.3 | Scaled | $4.98\times 6.09$ | Shani Shingnapur
(n) | 62.1 | 98.2 | Cropped & scaled | $5.88\times 5.91$ | Shani Shingnapur
(o) | 93.7 | 98.3 | All | $3.24\times 2.35$ | Shani Shingnapur

Table 2:  Table showing the accuracy comparison of different modified images
for test events. In all the test cases, CNN has out-performed state-of-art
Improved ORB.

##  8 Benefits: Improved Understanding of Law & Order Scenario

The system is built with the aim to aid first responders to find the spread of
images that can create law and order situations by retrieving similar images,
find & analyze the users propagating the content, sentiments floating, etc.
Thus, helping in reducing their human efforts in identifying the visual
content of the interest. In the previous sections, we saw comparison between
different hand-crafted methods and CNN to compare image similarity. The
experimental results show that CNN outperforms Improved ORB and hence, CNN is
a more suitable technique to find image spread for the system. We now explain
various other measures offered by the system that help in achieving goals for
better law enforcement using similar image retrieval process discussed above.

Search Space Reduction: The system takes a set of keywords from the user to
create a database of images and an image using which it retrieves a set
similar images from the database created. Thus, reducing the search space for
first responders hence, saving their time and efforts to analyse the overall
dump. We were able to reduce the search space on an average by 67% and by 65%,
using CNN and Improved ORB as image retrieval technique respectively. The
maximum reduction in search space was seen for Baba Ram Rahim event followed
by Charlie Hebdo cartoon. Table 3 shows the details of search space reduction
for all the events using the two techniques.

Users Analysis: Users play a vital role in spreading the content on social
media. Thus, it is important for security analysts to identify people who are
spreading the images. The system tries to deliver this need by producing a
list of users who spread the visual content and their details like, their
Twitter usernames, profile pictures, description, location, and links to their
Twitter profiles. In our dataset, we found maximum users for Lord Hanuman
cartoon event propagating the content, for images retrieved using both CNN and
Improved ORB. Table 3 shows the number of users listed when we used Improved
ORB and CNN as image retrieval techniques.

Sentiment Analysis & Tweets vs. Retweets Analysis: Sentiments of the tweets
play an important role in spreading the content and eliciting the reaction of
people. Thus, making it an important analysis to find the percentage of
content having positive, negative and neutral sentiments. The system currently
analyzes the sentiment of the textual content having images using Sentiment140
API 333http://help.sentiment140.com/home, widely used in the literature to
study social media data. In our dataset, we found 7% of tweets have negative
sentiments and 5% have positive sentiments and remaining have neutral
sentiments which might be due to broken language, non-english, hinglish
language or absence of text. Tweets vs. Retweets analysis shows the percent of
retweets and original tweets. The analysis shows, most of the images in the
result were spread by retweeting, average retweeted posts were 89.4%.

Event |  | Search Space Reduction (%)
---
| Number of Users
---
Improved ORB | CNN | Improved ORB | CNN
Kulkarni ink | 51.1 | 54.3 | 770 | 1,144
Baba Ram Rahim | 83.1 | 90.7 | 75 | 96
Lord Hanuman cartoon | 60.1 | 61.6 | 2,210 | 2,657
Charlie Hebdo cartoon | 74.1 | 75.2 | 2,132 | 2,037
Shani Shingnapur | 57.9 | 57.9 | 186 | 186
Table 3: Event-wise reduction in search space and analysis on number of users
after retrieving similar images using Improved ORB and CNN.

##  9 Implementation and Response Time

###  9.1 Implementation

The proposed system takes an image and keywords as input. These keywords are
then given to Twitter’s Search API (Application Program Interface), part of
Twitter’s REST API. The API returns a set of tweets related to the keywords it
takes as the query, and the system then filters and saves the tweets
containing images in a database. As the images get stored in the database, the
image comparison model computes the similarity score between the input image
and the images in the database. We then set a threshold $t$, if the similarity
score is above $t$, images are marked similar else dissimilar. After getting
similar images, the system does analysis on the retrieved set of images and
finds the users propagating them, the sentiment analysis and retweets
analysis. Fig. 7 shows input and different output and analysis screens of the
proposed system.

Figure 7: Moving in anti-clockwise from top-left image the images represent
(a)Input to the system is a image url and a keyword. (b) Output screen: The
system retrieves similar images (c) Analysis screen: Users propagating (d)
Analysis screen: Sentiment analysis on text of the resultant images (e)
Analysis Screen: Retweets analysis on text of the resultant images.

###  9.2 Response Time

Since our aim is to build a real-time image search system, apart from the
accuracy of the model we also need to select the technique that is time
efficient. We compare the approximate time consumed by Improved ORB and CNN to
compare one pair of image and we found that Improved ORB took 0.2 seconds and
CNN model took 0.55 seconds.

##  10 Conclusion

In our study, we proposed a OSINT real-time image search system robust to find
modified images, that aids first responders to analyze and find the current
spread of images creating law & order situations in society, analyze users
propagating such images and sentiments floating with them. The system performs
a hybrid search by taking a keyword to create an image database and an image
as input and returns similar images. The system also aims to reduce search
space for first responders on average by 67%. To find similar images we
conducted an in-depth experimental analysis of different CBIR techniques. We
did the experimental study on the data collected for the 5 events which
created law and order situations in society and whose images were viral during
the duration of the study. We compared different hand-crafted techniques and
found that Improved ORB (ORB + RANSAC) is the state-of-art technique for the
hand-crafted approach. We also proposed a CNN model which outperforms the
accuracy of state-of-art hand-crafted technique. Future work will focus on
reducing the response time of the proposed system, improving results of
textual sentiment analysis that takes broken or non-english language into
consideration, including more image analytics features like, sentiments of
images in result to aid maintaining safety in society during security critical
scenarios.

## References

  * [1] Jeremy Crump.  What are the police doing on twitter? social media, the police and the public.  3(4):1–27, 2011.
  * [2] Peng Cui, Shao-Wei Liu, Wen-Wu Zhu, Huan-Bo Luan, Tat-Seng Chua, and Shi-Qiang Yang.  Social-sensed image search.  ACM Transactions on Information Systems (TOIS), 32(2):8, 2014.
  * [3] S Denef, N Kaptein, and PS Bayerl.  Ict trends in european policing. composite project, 2011.
  * [4] The Indian Express.  Over 1500 women plan to storm shani shingnapur temple today, January 2016\.
  * [5] Philipp Fischer, Alexey Dosovitskiy, and Thomas Brox.  Descriptor matching with convolutional neural networks: a comparison to sift.  arXiv preprint arXiv:1405.5769, 2014.
  * [6] Matthew S Gerber.  Predicting crime using twitter and kernel density estimation.  Decision Support Systems, 61:115–125, 2014.
  * [7] The Guardian.  Charlie hebdo cartoon depicting drowned child alan kurdi sparks racism debate, January 2016.
  * [8] A Gupta and P Kumaraguru.  Misinformation on twitter during crisis events. encyclopedia of social network analysis and mining (esnam), 2012.
  * [9] Aditi Gupta, Hemank Lamba, and Ponnurangam Kumaraguru.  $1.00 per rt #bostonmarathon #prayforboston: Analyzing fake content on twitter.  In eCrime Researchers Summit (eCRS), 2013, pages 1–12. IEEE, 2013\.
  * [10] WAYNE HANSON.  How social media is changing law enforcement, December 2011.
  * [11] Jonathon S Hare, Sina Samangooei, David P Dupplaw, and Paul H Lewis.  Twitter’s visual pulse.  In Proceedings of the 3rd ACM conference on International conference on multimedia retrieval, pages 297–298. ACM, 2013.
  * [12] Steven CH Hoi and Pengcheng Wu.  Sire: a social image retrieval engine.  In Proceedings of the 19th ACM international conference on Multimedia, pages 817–818. ACM, 2011.
  * [13] Tamara Kharroub and Ozen Bas.  Social media and protests: An examination of twitter images of the 2011 egyptian revolution.  New Media & Society, page 1461444815571914, 2015.
  * [14] Philip J McParlane, Andrew James McMinn, and Joemon M Jose.  Picture the scene…;: Visually summarising social media events.  In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 1459–1468. ACM, 2014\.
  * [15] Marcelo Mendoza, Barbara Poblete, and Carlos Castillo.  Twitter under crisis: can we trust what we rt?  In Proceedings of the first workshop on social media analytics, pages 71–79. ACM, 2010.
  * [16] BBC News.  Mumbai ink attack: India police arrest six shiv sena workers, October 2015\.
  * [17] Greg Pass, Ramin Zabih, and Justin Miller.  Comparing images using color coherence vectors.  In Proceedings of the fourth ACM international conference on Multimedia, pages 65–73. ACM, 1997.
  * [18] Adrian Popescu and Gregory Grefenstette.  Social media driven image retrieval.  In Proceedings of the 1st ACM International Conference on Multimedia Retrieval, page 33. ACM, 2011.
  * [19] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [20] Simon Rogers.  What fuels a tweet’s engagement, March 2014.
  * [21] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In 2011 International conference on computer vision, pages 2564–2571. IEEE, 2011.
  * [22] Rick Ruddell and Nicholas Jones.  Social media and policing: matching the message to the audience.  Safer Communities, 12(2):64–70, 2013.
  * [23] Niharika Sachdeva and Ponnurangam Kumaraguru.  Online social networks and police in india -understanding the perceptions, behavior, challenges.  In ECSCW 2015: Proceedings of the 14th European Conference on Computer Supported Cooperative Work, 19-23 September 2015, Oslo, Norway, pages 183–203. Springer, 2015.
  * [24] Niharika Sachdeva and Ponnurangam Kumaraguru.  Social networks for police and residents in india: exploring online communication for crime prevention.  In Proceedings of the 16th Annual International Conference on Digital Government Research, pages 256–265. ACM, 2015.
  * [25] Niharika Sachdeva and Ponnurangam Kumaraguru.  Social media-new face of collaborative policing?  In International Conference on Social Computing and Social Media, pages 221–233. Springer, 2016.
  * [26] Lexis Nexis Risk Solutions.  Survey of law enforcement personnel and their use of social media in investigations.  Lexis Nexis, 2012.
  * [27] Business Standard.  Bjp files complaint against journalist for morphed modi photo, April 2016\.
  * [28] Kate Starbird and Leysia Palen.  Voluntweeters: Self-organizing by digital volunteers in times of crisis.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1071–1080. ACM, 2011.
  * [29] Sakal Times.  180 held for circulating obscene photos on internet, June 2014.
  * [30] The Economics Times.  Ncrb to connect police stations and crime data across country in 6 months, November 2014.
  * [31] India Today.  Arvind kejriwal slammed on twitter again! twitterati trend #kejriwalinsultshanuman, February 2016.
  * [32] India Today.  Case against gurmeet ram rahim for posing as vishnu, will he be arrested?, January 2016.
  * [33] Engin Tola, Vincent Lepetit, and Pascal Fua.  A fast local descriptor for dense matching.  In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1–8. IEEE, 2008.
  * [34] Engin Tola, Vincent Lepetit, and Pascal Fua.  Daisy: An efficient dense descriptor applied to wide-baseline stereo.  IEEE transactions on pattern analysis and machine intelligence, 32(5):815–830, 2010.
  * [35] Sarah Vieweg, Amanda L Hughes, Kate Starbird, and Leysia Palen.  Microblogging during two natural hazards events: what twitter may contribute to situational awareness.  In Proceedings of the SIGCHI conference on human factors in computing systems, pages 1079–1088. ACM, 2010.
  * [36] Xiaofeng Wang, Matthew S Gerber, and Donald E Brown.  Automatic crime prediction using events extracted from twitter posts.  In International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction, pages 231–238. Springer, 2012\.
  * [37] Stefanie Wiegand and Stuart E Middleton.  Veracity and velocity of social media content during breaking news: Analysis of november 2015 paris shootings.  In Proceedings of the 25th International Conference Companion on World Wide Web, pages 751–756. International World Wide Web Conferences Steering Committee, 2016.
  * [38] Lei Yu, Zhixin Yu, and Yan Gong.  An improved orb algorithm of extracting and matching.  2015\.