# IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes
in Unstructured Traffic and Adverse Weather

Furqan Ahmed Shaik∗1 Abhishek Reddy Malreddy1 Nikhil Reddy Billa1
Kunal Chaudhary2 Sunny Manchanda2 Girish Varma1
$1.$ Machine Learning Lab, IIIT Hyderabad $2.$ DYSL-AI, DRDO
<EMAIL_ADDRESS>
###### Abstract

Large-scale deployment of fully autonomous vehicles requires a very high
degree of robustness to unstructured traffic, weather conditions, and should
prevent unsafe mispredictions. While there are several datasets and benchmarks
focusing on segmentation for drive scenes, they are not specifically focused
on safety and robustness issues. We introduce the IDD-AW dataset, which
provides 5000 pairs of high-quality images with pixel-level annotations,
captured under rain, fog, low light, and snow in unstructured driving
conditions. As compared to other adverse weather datasets, we provide i.) more
annotated images, ii.) paired Near-Infrared (NIR) image for each frame, iii.)
larger label set with a 4-level label hierarchy to capture unstructured
traffic conditions. We benchmark state-of-the-art models for semantic
segmentation in IDD-AW. We also propose a new metric called “Safe mean
Intersection over Union (Safe mIoU)” for hierarchical datasets which penalizes
dangerous mispredictions that are not captured in the traditional definition
of mean Intersection over Union (mIoU). The results show that IDD-AW is one of
the most challenging datasets to date for these tasks. The dataset and code
will be available here: http://iddaw.github.io.

##  1 Introduction

Figure 1: Examples of RGB/NIR/Annotations triplets in four adverse weather
conditions from IDD-AW. The color legend for annotation is given in Figure 2.

Semantic Segmentation of driving scenes is one of the major tasks in the
development of autonomous driving vehicles. Even though there has been much
work in terms of datasets [5, 4, 10, 17], there are still many challenges when
it comes to the representation of rare events and conditions. One of the major
challenges is the robustness of the model to high amounts of traffic
participants and adverse weather conditions. While state-of-the-art models
give high prediction accuracies, it’s unclear how they will perform in such
cases. Vehicles with full autonomy cannot be deployed safely unless these
corner cases are analyzed.

Secondly, given the focus on safety in autonomous vehicles, it is essential to
check if the metrics used accurately represent the safety concerns in road
scenes. Traditionally segmentation models have been evaluated based on the
mIoU which is a generic metric for the quality of the segmentation. However,
mIoU doesn’t capture safety concerns specific to road scenes. For example,
accurate prediction of traffic participants (vehicles, living things) and
roadside objects are more essential compared to far objects. Also for
autonomous driving, certain types of mispredictions are more tolerable than
others. For example, a car being mispredicted as a bus is more tolerable than
a car being mispredicted as a road. mIoU treats both these mispredictions in
the same way while they represent different severity of safe mispredictions.

In this paper, we introduce the IDD-AW dataset which captures highly
unstructured traffic in adverse weather conditions. Here, highly unstructured
traffic refers to situations where road conditions are intricate and
challenging to predict, often characterized by a diverse mix of vehicles,
pedestrians, and other dynamic elements. The dataset captures driving
scenarios under various adverse weather conditions, encompassing rain, fog,
low light, and snow. The distribution of pixels and instances of traffic
participants is higher when compared to existing datasets. Additionally, we
provide paired Near Infrared (NIR) images for every frame, thereby increasing
the information available to models and enhancing their ability to make
precise predictions.

We also propose a new metric called ”Safe mIoU (SmIoU)” which penalizes
dangerous mispredictions that are not captured in the traditional mIoU metric.
We propose this metric for all hierarchical datasets where the severity of the
penalty is based on the tree distance. We also calculate this metric over
various label sets varying from the complete set and narrowing it down to
several important classes such as traffic participants and roadside objects
which directly impact autonomous driving.

We benchmark models on the IDD-AW dataset and compare them with results on
other similar datasets like Cityscapes[4], IDD [25], ACDC [20]. We show that
appending the NIR image to the input increases the accuracy of our dataset. We
also show high contrast in results when comparing mIoU and SmIoU using the
same trained models. We provide qualitative and logical explanations, that the
lowering of Safe mIoU as compared to mIoU exposes safety concerns in existing
models.

### Main Contributions.

  1. 1.

We propose an adverse weather conditions dataset for robust segmentation in
unstructured traffic scenes (see Section 3).

  2. 2.

We also propose a new metric called ”Safe mIoU” to ensure the penalizing of
misclassified pixels in hierarchical datasets (see Section 4).

  3. 3.

We show both qualitative and quantitative results to prove our dataset is more
diverse and challenging than other existing datasets (see Section 5).

##  2 Related Work

Dataset |  | Labeled
---
images

Rain |  Fog |  Snow |  Lowlight |  Labels |  NIR
| Foggy
---
Driving
101 | 0 | 101 | 0 | 0 | 19 |
| Foggy
---
Zurich
40 | 0 | 40 | 0 | 0 | 19 |
| Nighttime
---
Driving
50 | 0 | 0 | 0 | 50 | 19 |
| Dark
---
Zurich
201 | 0 | 0 | 0 | 201 | 19 |
Raincouver | 326 | 326 | 0 | 0 | 95 | 19 |
WildDash | 226 | 13 | 10 | 26 | 13 | 19 |
BDD100K | 1346 | 213 | 23 | 345 | 765 | 19 |
ACDC | 4006 | 1000 | 1000 | 1000 | 1006 | 19 |
IDD-AW | 5000 | 1500 | 1500 | 1000 | 1000 | 30 | ✓

Table 1: Comparison of IDD-AW with other adverse weather condition datasets.
IDD-AW provides i.) more labeled images ii.) with more (30) labels that
capture unstructured driving conditions and iii.) NIR images.

### Road scene segmentation datasets.

The Cityscapes Dataset[4, 5] is one of the most popular datasets for driving
scene segmentation. Along with it, there are other driving datasets such as
KITTI [10], Pascal VOC[8], Mapillary vistas[17] etc which contain a large
collection of images. However, most of these datasets are captured in
structured environments and have label sets that cannot completely satisfy the
unstructured scenarios. IDD[25] introduced a large-scale annotated dataset in
unstructured driving conditions with more than 10K annotated images. It
contains drive scenes with a larger label set when compared to Cityscapes and
includes 2-wheelers, pedestrians, and roadside objects. However, none of the
above datasets have images in adverse weather conditions. All the datasets
have been captured in clear daylight conditions with minimum image distortion.

Figure 2: IDD & IDD-AW hierarchical labels (x-axis) and pixel count (y-axis)
comparison. IDD provides data in unstructured driving conditions and our pixel
composition is similar to it. Both datasets have 30 labels and a 4-level label
hierarchy. The colors used will serve as legends for all plots of annotations
and predictions in this paper. Normalization is done relative to image
resolution and number of images in the datasets.

### Adverse weather road scene datasets.

Oxford Robotcar [16] was the first real-world large-scale dataset in which
adverse visual conditions such as nighttime, rain, and snow were significantly
represented, but it did not feature semantic annotations. Foggy Zurich[18],
Foggy Cityscapes[19] have synthesized fog images generated by applying a mask
over the original images. Rain1400[9], RainyCityscapes [22] Rain100H[27], and
Rain12[15] are prime examples for synthetic rain datasets. Comprehensive Snow
Dataset [3] is another example of synthetic snow images. Datasets such as
Rainy Wcity [31] BDD100K[28], ACDC[20], WildDash [29], WildDash2 [30] and
Ithaca365[6] have real images which have been captured in various adverse
conditions like rain, fog, snow and low light/night time. However, none of
these datasets have both NIR and RGB components which could help us give a
better insight into the images in these adverse conditions. We compare these
datasets with IDD-AW in Table 1.

### Datasets with NIR/Multispectral Data.

Nuscenes [2], the first dataset to carry the fully autonomous vehicle sensor
suite: 6 cameras, 5 radars, and 1 lidar, all with a full 360-degree field of
view. The Multi-Spectral Road Scenarios (MSRS) dataset [23] focuses on road-
related scenarios by combining NIR and RGB modalities. The TNO Hyperspectral
Dataset [24] provides NIR and RGB images obtained from an airborne platform.
Similarly, the KAIST Multispectral Pedestrian Dataset [12] offers synchronized
NIR and RGB images captured in urban settings. IDD-3D [7], which consists of
multimodal data from multiple cameras and LiDAR sensors across various traffic
scenarios. However, most of these datasets do not cover adverse weather
images. IDD-AW dataset provides synchronized RGB and NIR images in adverse
weather and unstructured driving scenarios.

### Safe & Robust Segmentation.

The robustness of a dataset can also be viewed in terms of the ambiguity of
labels in the dataset and the misclassification of corner cases. Disambiguity
of labels can be achieved by introducing a hierarchy in the dataset labels.
Hierarchical semantic segmentation explains visual scenes with multi-level
abstraction by considering a structured class label hierarchy.

Li et al. [14] addresses hierarchical segmentation at both scene level and
object level. Shaik et al. [21] aims to accelerate semantic segmentation and
other computer vision tasks on GPUs using sparsity frameworks. In the work of
Li et al. [13], the authors aim to enhance hierarchical semantic segmentation
by modifying the cross-entropy loss and penalizing it over various class
hierarchies. In the research by Bar et al. [1], the ‘teacher-student
framework’ is employed, where a pre-trained model guides the training of
another model, a common practice to improve segmentation. In the study by
Kamann et al. [11], the authors examine the robustness of semantic
segmentation networks concerning a wide range of image corruptions based on
their architecture.

IDD-AW contains images in unstructured driving scenarios which are also
captured in various adverse weather conditions. This signifies the robustness
of the dataset along various conditions. Additionally, we introduce a metric
called Safe-mIoU which penalizes unsafe mispredictions more than traditional
mIoU. This helps in improving the safety of the models especially for
hierarchical driving datasets.

Figure 3: Comparison of ACDC & IDD-AW: Pixel wise (left), traffic participants
instances count (right). Pixel counts are normalized by resolution and the
number of images in the datasets. IDD-AW has more pixels (left) as well as
more instances per image (right) of traffic participants (TP) which includes
all vehicles and living things that represent unstructured traffic. Note that
there are over 300 images in IDD-AW with more than 20 instances of TP while
ACDC has only around 10.

##  3 The IDD-AW Dataset

### Data Collection or Acquisition.

The dataset primarily focuses on capturing traffic scenarios in adverse
weather conditions such as rain, fog, snow, and low light, with an emphasis on
unstructured driving scenarios. Our dataset comprises a total of 202 driving
sequences, providing a comprehensive collection of challenging driving
scenarios. The dataset is captured at different locations in India comprising
urban, rural, and even hilly areas to ensure diversity across various
conditions.

For data capture, we used JAI’s FS-3200D-10GE camera. This is a 2-CMOS
multispectral prism camera that captures images in different spectral bands
simultaneously. Specifically, it provides a visible color channel from 400-670
nm and a near-infrared (NIR) channel from 740-1000 nm. To ensure a clear
separation between different adverse conditions, each capture was made under
only one adverse scenario to make sure the images were diverse between various
conditions.

### Frame Selection.

The frames captured by the camera are filtered using a 3-second threshold to
eliminate any repetitive frames. The goal of this threshold is to ensure that
unique frames are used for subsequent processing. After the filtering step,
the RGB-NIR pairs are matched based on their time stamps. However, to ensure
accuracy, the matches are also manually verified by reviewing all frames. From
the matched pairs, visually appealing frames that feature a diverse range of
traffic participants and showcase adverse driving conditions are shortlisted.
This process allows for the selection of high-quality frames for the dataset.

### Label hierarchy and Annotation.

The label set used is the same as in IDD [25]. This label set has a
hierarchical structure with four levels, consisting of 7 labels at level 1 and
30 labels at level 4 (see x-axis of Figure 2). This adds a higher level of
complexity to our dataset when compared to existing datasets like Cityscapes,
and even when compared to adverse weather datasets like ACDC or Foggy
Cityscapes. For labelling the dataset, we had a team of highly skilled
annotators. The annotation process for each image typically requires 1.5 to
2.5 hours to complete, encompassing the initial annotation and quality check.
To ensure the correctness of the annotations, multiple steps are taken, such
as verifying annotations against predefined criteria and conducting a final
quality check by experienced annotators.

### Dataset Splits.

IDD-AW is split into four sets corresponding to the examined conditions. We
manually selected 1500 rainy, 1500 foggy, and 1000 lowlight and 1000 snowy
images from the recordings for dense pixel-level semantic annotation, for a
total of 5000 adverse-condition images. The selection process aimed at
maximizing the complexity and diversity of captured scenes.

We used a thorough train-test split technique with set limits to permit robust
evaluations. All images from a drive sequence are either completely in train
or in test to ensure that the test is truly unseen and distinct. For each
weather condition, the test set includes drive sequences in a ratio of 0.18 to
0.22 of total sequences, ensuring adequate representation of varied driving
scenarios. The average number of frames per drive sequence in the test set was
kept within the ratio of 0.9 to 1.2 of the average frames per drive sequence
of the entire dataset. Class-specific fairness is maintained by ensuring that
the average instances per image in the test set vary between 0.8 and 1.2 of
the overall dataset. For precise pixel-level accuracy evaluation, the test
dataset must meet strict criteria: an average pixel ratio of 0.8 to 1.2 for at
least 18 classes, and 0.7 to 1.3 for at least 22 classes.

### Comparison with Other Datasets.

In Table 1, we compare IDD-AW to existing datasets that address semantic
segmentation under adverse conditions. Most of these datasets are specialized
in a single condition and are relatively small in scale. IDD and ACDC are the
two primary large-scale datasets, focusing on unstructured driving scenes and
adverse weather, respectively. IDD-AW is unique in its focus on drive scenes
that are both unstructured and affected by adverse weather. Additionally, we
provide a Near-Infrared (NIR) component, recognizing its potential to enhance
segmentation accuracy and safety. NIR images tend to exhibit higher contrast
and sharpness, while RGB images provide color and luminescence information
that enhances the visual appeal.

Figure 2 illustrates the pixel-wise comparison between IDD and IDD-AW
datasets. Although our dataset features images captured in adverse weather
conditions, most of the classes exhibit pixel counts similar to those of the
IDD dataset which represents unstructured traffic.

Additionally, Figure 3 provides a pixel-wise comparison, as well as traffic
participant, counts when compared with ACDC. Our dataset notably contains more
pixels for crucial traffic participants such as roads, pedestrians, riders,
and cars. Similarly, when comparing instance counts for traffic participants
per image, nearly every image in our dataset has more instances than those in
ACDC. Moreover, over 300 images in our dataset contain more than 20 instances,
while ACDC only includes approximately 10-20 images with the same number of
instances. This observation highlights that our dataset features numerous
dense images in terms of both pixels per class and traffic participants per
image.

Figure 4: Example of tree distances between labels. Tree distance captures the
severity of the misprediction using the label hierarchy. The
$\text{td}(\text{sidewalk},\text{motorcycle})=3$ since the length of the path
is 6 and td is length/2. Similarly $\text{td}(\text{person},\text{rider})=2$
and $\text{td}(\text{truck},\text{bus})=1$.

##  4 Safe-mIoU: A metric for safe segmentation

Traditionally, mIoU (mean of Intersection over Union) is a widely used metric
for semantic segmentation tasks. It reflects the quality of segmentation by
considering both false positives (pixels mistakenly classified as a class) and
false negatives (pixels of the class not detected). It captures the balance
between precision and recall. However, while mIoU is an excellent general-
purpose metric, its limitations become apparent when attempting to quantify
the safety of driving scenes. However, mIoU’s strength lies in its generality,
and this generality becomes its limitation in contexts where class importance
as well as the severity of the mispredictions varies. In driving scenes, the
safety of pedestrians, vehicles, traffic signs, and other critical elements
holds paramount importance. Misclassifying these safety-critical classes can
lead to dire consequences. Also, certain misclassifications are dangerous,
while others are tolerable. For example, it is tolerable to misclassify one
vehicle as another type, while it is dangerous to misclassify a pedestrian or
the vehicle as the road (see Figure 5). Unfortunately, traditional mIoU does
not inherently capture these safety concerns. It treats all classes as equal
entities, disregarding their impact on real-world driving scenarios.

To address this gap, a more refined metric is needed – one that can not only
assess the quality of segmentation results but also consider the severity of
unsafe mispredictions. We propose a new Safe mIoU (SmIoU) metric. The
calculation of Safe mIoU involves selecting a subset of critical/important
classes $C_{\text{imp}}$ from the entire class set $C$. These chosen classes
typically encompass those that have a direct impact on safety within driving
scenes, such as pedestrians, vehicles, and traffic signs. Furthermore, we use
the distance between the predicted and the actual label in the hierarchical
label tree of IDD-AW, to measure the severity of misclassifications in
$C_{\text{imp}}$. It aims to provide a more holistic evaluation that aligns
with the priorities of driving scene applications and the overall safety of
the environment.

The essence of SmIoU lies in the introduction of hierarchical penalty, a
strategy that takes into account the semantic relationships between classes.
Misclassifications within critical classes, and non-critical classes
classified as critical, are penalized based on their distance in the class
hierarchy. The _tree distance_ (td) between a pair of labels is the length of
the shortest path in the class hierarchy tree divided by 2 (see Figure 4). So
the td between person and rider is 1, while sidewalk and motorcycle is 3 (see
the class hierarchy in 2), denoting that the latter is a more severe
misprediction than the former. The calculation of SmIoU involves computing the
$I_{c}^{\text{safe}}$ for each class as shown in equation 1. The safe IoUs are
the IoUs with a penalty for misclassification of that class weighted by the
tree distance. The final SmIoU score is obtained by taking the mean of these
$I_{c}^{\text{safe}}$ as shown in equation 2, resulting in a metric that
accurately measures both the segmentation quality and the safety-related
considerations of driving scene applications.

Figure 5: Examples from IDD-AW, showing a pedestrian in the rain, a truck in
the fog, a rider/bike in low light, and an animal in the snow are mispredicted
respectively. In the Severity Column, Red, orange, and yellow colors denote
the level of danger posed by mispredictions. In the third row, a bike being
mispredicted as a truck is less dangerous (in orange) than a rider being
predicted as a vehicle (in red). mIoU considers these different colored
mispredictions in the same way, while the proposed safe mIoU gives a higher
penalty for orange and red mispredictions. In these images, the proposed safe
mIoU is less than mIoU by more than 20%.

Let $C$ be the set of all classes at the bottom level of the hierarchy,
$d(c,s)$ be the tree distance between class $c$, $s$, and $n$ be the number of
levels in the hierarchy, $\text{gt}_{c}$ the set of pixels in the ground truth
with label $c$ and $\text{pred}_{s}$ the set of pixels in the prediction with
label $s$. We define the following quantities:

$\displaystyle I_{c,s}^{\text{safe}}$
$\displaystyle=\frac{|\text{gt}_{c}\cap\text{pred}_{s}|}{|\text{gt}_{c}\cup\text{pred}_{c}|}\qquad
I_{c,c}$
$\displaystyle=\frac{|\text{gt}_{c}\cap\text{pred}_{c}|}{|\text{gt}_{c}\cup\text{pred}_{c}|}$
(a)

Now, we define SmIoU as follows:

$I_{c}^{\text{safe}}=\begin{cases}\begin{aligned} I_{c,c}&-\sum_{s\in C,s\neq
c}\frac{d(c,s)}{n}I_{c,s}^{\text{safe}}&\text{ if }c\in C_{\text{imp}}\\\
I_{c,c}&-\sum_{s\in
C_{\text{imp}}}\frac{d(c,s)}{n}I_{c,s}^{\text{safe}}&\text{ else.
}\end{aligned}\end{cases}$ (1)

$\text{SmIoU}=\frac{\sum_{c\in C}I_{c}^{\text{safe}}}{|C|}$ (2)

Note that the definition of SmIoU requires the definition of important classes
$C_{\text{imp}}$. When SmIoU is mentioned without specifying $C_{\text{imp}}$,
we take it to be the union of traffic participants and roadside object classes
(all classes except Far Objects and Sky in Figure 2).

Dataset [0.3em] Test $\rightarrow$ [0.3em] Train $\downarrow$ |  CS |  ACDC |  IDD |  Rain |  Fog |  LL |  Snow |  IDD-AW
---|---|---|---|---|---|---|---|---
CS RGB | 83 | - | - | 46 | 45 | 42 | 43 | 46
ACDC RGB | - | 75 | - | 47 | 51 | 42 | 38 | 48
IDD RGB | - | - | 73 | 52 | 55 | 50 | 33 | 54
IDD-AW RGB | 49 | 51 | 51 | 62 | 64 | 62 | 53 | 64
IDD-AW NIR | - | - | - | 61 | 58 | 57 | 51 | 61
IDD-AW NIR+RGB | - | - | - | 66 | 65 | 63 | 53 | 67

Table 2: Comparison of mIoU scores (%) of InternImage-b model trained on
CityScapes (CS), ACDC, IDD, and IDD-AW datasets (RGB, NIR, and Combined) for
IDD-AW test set in individual conditions and jointly for all conditions. The
model trained on IDD-AW NIR+RGB gives the highest accuracy which is 20% more
than CS, ACDC, and 14% more than IDD. The NIR+RGB model gives 3% more accuracy
compared to the RGB model, which indicates that the NIR image adds useful
information for prediction. Also, note that the model trained on IDD-AW gives
3% better accuracy when tested on CS and ACDC as compared to the models
trained on the respective datasets tested on IDD-AW (in grey). This trend is
reversed in IDD since its training set is double the size of IDD-AW (in darker
grey).

### Properties of SmIoU.

The following are the important properties of the proposed SmIoU when compared
with traditional mIoU, which show their relationship and motivate the
definition of SmIoU for capturing unsafe mispredictions.

  1. 1.

$\text{SmIoU}=\text{mIoU}$, when $C_{\text{imp}}=\emptyset$. When there are no
important classes, SmIoU matches traditional mIoU. This is because the penalty
becomes zero from the above formula, resulting in SmIoU becoming equal to the
original mIoU.

  2. 2.

$\text{SmIoU}\in[-1,1]$. As we are penalizing the mIoU with the tree distance
divided by the number of levels in the hierarchy, it always ranges between -1
to 1 inclusive of both extremes.

  3. 3.

$\text{SmIoU}=1$ if and only if all pixels in an image are accurately
classified into their respective classes.

  4. 4.

$\text{SmIoU}=-1$ if and only if all pixels in an image are misclassified with
the farthest classes in the tree hierarchy.

eval $\rightarrow$ | cross |  | same
---|---|---|---
Test $\rightarrow$ [0.3em] Train $\downarrow$ |  Rain |  Fog |  LL |  Snow |  |  mIoU |  SmIoU (tp) |  SmIoU
IDD | 52 | 55 | 50 | 33 |  | - | - | -
IDD-AW | - | - | - | - |  | 64 | 60 | 51
Rain | - | 55 | 40 | 29 |  | 64 | 58 | 48
Fog | 51 | - | 53 | 29 |  | 64 | 58 | 47
LL | 52 | 57 | - | 30 |  | 62 | 58 | 48
Snow | 35 | 38 | 33 | - |  | 53 | 43 | 28
Table 3: Cross-evaluation of InternImage-b condition experts on the various
conditions of IDD-AW are shown in the first 4 columns. Comparison of mIoU (%)
with SmIoU (%) metric at different levels and label sets for various adverse
weather conditions shown in the last 3 columns. Here, SmIoU refers to Safe
mIoU and tp refers to setting the important classes to be just the traffic
participants (ie Living Things $\cup$ Vehicles in Figure 2).

##  5 Results

### Training on CS, IDD, and ACDC.

We have performed several experiments on Cityscapes, ACDC, IDD, and IDD-AW
datasets using the InternImage [26] model. InternImage was chosen due to its
state-of-the-art performance currently in semantic segmentation benchmarks. We
have taken InternImage-b backbones of the InternImage [26]. Regarding the
results from Table 2, we can see the mIoU when trained on Cityscapes does not
even cross 45% mIoU when tested on IDD-AW. As the underlying dataset does not
have any images with different weather conditions in it, the performance of a
CS-trained model on IDD-AW is very low. Similarly, a model trained on ACDC
does not cross 50%, which shows the dataset is that much harder and has a
larger label set. Even though ACDC is also a driving dataset that is captured
in multiple weather conditions, there’s still not much improvement from
Cityscapes. This already shows that our dataset is much harder than both
Cityscapes and ACDC datasets.

When using pre-trained models of the IDD dataset, it performs much better than
Cityscapes and ACDC despite testing with a larger label set. This is due to
the unstructured images being present in the IDD dataset. The IDD pre-trained
models performed efficiently in well-lit conditions like rain and fog,
nonetheless the performance did not cross the 50% mIoU mark in low-light
conditions. Also, the major concern is the performance in snow conditions,
where the best IDD pre-trained model, gives only 32% mIoU which is a
significant drop and is not usable for this condition at all. So even though
the final mIoU reaches around 53%, it cannot be generalized over each
condition and overly fails to perform in certain conditions.

### Training and Evaluation on IDD-AW.

We perform the training in three different ways on IDD-AW. First, we train the
network only on the RGB images of our dataset. We can observe that training on
IDD-AW gives more than 10% mIoU score when compared to IDD and almost 20% when
taking the best performing Cityscapes or the ACDC pre-trained model. This is
because of the variety of images in our dataset in various weather conditions
which are not present in regular driving datasets like IDD or Cityscapes, as
well as the unstructured nature of images and varying traffic when compared to
other adverse datasets like ACDC. From Figure 6, we can see qualitative
results from each pre-trained model on our dataset.

We conducted a similar experiment using only the NIR components of the images,
yielding a mIoU of over 60% across various models and highlighting the quality
of both the RGB and NIR components. Furthermore, we include training with
stacked RGB+NIR images as input. Notably, the combination of RGB and NIR
images through stacking led to a significant increase in mIoU by more than 3%
when compared to using just RGB or NIR components alone (refer to Table 2),
underscoring the critical role of NIR components within our dataset.

### Condition Expert Training.

In condition expert training, we train specifically on each condition
separately and then test on each condition to assess the quality of images
across each set. In Table 3, we can see the differences between training on
one condition and testing on each of them. We can see that training on any
other condition except Snow, does not yield any significant mIoU increase in
the snow images.

### Safe mIoU on IDD-AW test set.

Figure 6: Qualitative examples from each condition with predictions using
models trained on ACDC, IDD, and IDD-AW Datasets and the ground truth at the
end. The mIoUs (%) for each pre-trained model on the IDD-AW test set are
46.57, 53.4, and 64.5 using ACDC, IDD, and IDD-AW pre-trained models
respectively. Figure 7: Distribution of Safe mIoU (%) vs mIoU (%) for IDD-AW
test set. y-axis represents the number of images with a particular value of
SmIoU/mIoU. The SmIoU distribution is shifted to the lower side, indicating
that it is finding a significant amount of dangerous mispredictions that are
not accounted for in mIoU.

Metric |  Condition |  road |  | drivable
---
fallback

sidewalk |  person |  rider |  bike |  bicycle |  rickshaw |  car |  truck |  bus |  | vehicle
---
fallback

curb |  wall |  guard rail |  billboard |  | traffic
---
sign

| traffic
---
light

mIoU % | All | 95 | 51 | 48 | 76 | 72 | 68 | 5 | 83 | 85 | 74 | 76 | 45 | 78 | 52 | 74 | 62 | 58 | 52
Rain | 96 | 49 | 49 | 48 | 73 | 68 | 2 | 86 | 87 | 79 | 52 | 47 | 81 | 47 | 46 | 64 | 54 | 52
Fog | 97 | 64 | 24 | 62 | 75 | 73 | 8 | 67 | 91 | 82 | 80 | 49 | 79 | 57 | 78 | 61 | 69 | 47
LL | 95 | 60 | 54 | 73 | 73 | 68 | 38 | 75 | 80 | 69 | 86 | 29 | 65 | 50 | 72 | 58 | 35 | 53
Snow | 85 | 42 | - | 80 | 62 | 40 | 0 | - | 82 | 58 | 70 | 48 | 23 | 56 | 64 | 60 | 37 | -
SmIoU % | All | 92 | 32 | 16 | 64 | 58 | 52 | \- 22 | 77 | 81 | 68 | 70 | 21 | 69 | 32 | 61 | 42 | 40 | 27
Rain | 94 | 28 | 25 | 15 | 59 | 54 | -7 | 81 | 84 | 74 | 46 | 23 | 74 | 22 | 20 | 46 | 32 | 26
Fog | 95 | 51 | -48 | 41 | 62 | 58 | -63 | 43 | 87 | 78 | 77 | 28 | 71 | 52 | 69 | 45 | 53 | 12
LL | 92 | 46 | 22 | 60 | 59 | 52 | 14 | 70 | 76 | 61 | 82 | 2 | 48 | 26 | 52 | 35 | 15 | 32
Snow | 79 | 20 | - | 70 | 44 | -2 | -99 | - | 76 | 48 | 56 | 23 | -46 | 23 | 42 | 38 | 5 | -

Table 4: Comparison of class-wise labels for important classes between mIoU vs
SmIoU for InternImage-b model on segmentation.

In Table 3, we can see the drop in SmIoU when we consider the important
classes. For the InternImage-b framework, the SmIoU again drops more than 15%
at least for each individual condition. This clearly shows the discrepancy and
the dangers of traditional mIoU in hierarchical autonomous driving datasets.
In Figure 7, we can see the distribution of SmIoU vs mIoU over the test set
has shifted toward the lower side which some images exhibiting even negative
SmIoU.

Notice that SmIoU = mIoU if $C_{\text{imp}}=\emptyset$. When we add more
classes to $C_{\text{imp}}$ the metric value decreases. This can be observed
in the last 3 columns of Table 3. They correspond to
$C_{\text{imp}}=\emptyset$, traffic participants and traffic participants
$\cup$ roadside objects respectively. The SmIoU (tp) has values between mIoU
and SmIoU. This shows our new metric is flexible and very effective at
identifying the images in which the important classes are misclassified.

When comparing class-wise mIoUs and SmIoUs for the InternImage-b framework as
in Table 4, we can see several classes showing great disparities between mIoU
and SmIoU. There is a significant difference between things like bicycles,
traffic signals, and sidewalks. It can be dangerous to misclassify these
classes, especially when driving. When compared to conventional mIoU, our
suggested SmIoU metric more accurately measures this risk. SmIoU also shows
several classes with negative values showing dangerous misclassification,
especially bicycle, sidewalk, and curb. The danger level associated with
misclassification that could be brought on by these objects in the drive
scenes is represented by classes like person in rain, sidewalk, and rickshaw
in fog, vehicle fallback in lowlight, bike, and curb in snow that is well
captured by SmIoU but missed by the traditional mIoU.

##  6 Conclusion

We have presented IDD-AW, a large-scale dataset and a benchmark suite for
semantic driving scene understanding in adverse weather and unstructured
driving conditions. We also present a new metric called Safe mIoU which
incorporates safety concerns in the definition of mIoU. We benchmark state-of-
the-art models for semantic segmentation in IDD-AW and also show the
differences between traditional mIoU and safe mIoU while considering important
classes. Finding appropriate loss functions, which can better optimize safe
mIoU more efficiently is an interesting direction for future work.

##  7 Acknowledgement

We acknowledge the funding from DYSL-AI, DRDO for this project. The dataset
was collected in collaboration with iHub Data, IIIT Hyderabd.

## References

  * [1] Andreas Bar, Marvin Klingner, Serin Varghese, Fabian Huger, Peter Schlicht, and Tim Fingscheidt.  Robust semantic segmentation by redundant networks with a layer-specific loss contribution and majority vote.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 332–333, 2020.
  * [2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [3] Wei-Ting Chen, Hao-Yu Fang, Cheng-Lin Hsieh, Cheng-Che Tsai, I Chen, Jian-Jiun Ding, Sy-Yen Kuo, et al.  All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4196–4205, 2021.
  * [4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset.  In CVPR Workshop on the Future of Datasets in Vision, volume 2. sn, 2015.
  * [6] Carlos A. Diaz-Ruiz, Youya Xia, Yurong You, Jose Nino, Junan Chen, Josephine Monica, Xiangyu Chen, Katie Luo, Yan Wang, Marc Emond, Wei-Lun Chao, Bharath Hariharan, Kilian Q. Weinberger, and Mark Campbell.  Ithaca365: Dataset and driving perception under repeated and challenging weather conditions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21383–21392, June 2022.
  * [7] Shubham Dokania, A. H. Abdul Hafez, Anbumani Subramanian, Manmohan Chandraker, and C. V. Jawahar.  IDD-3D: Indian driving dataset for 3d unstructured road scenes.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4482–4491, January 2023.
  * [8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  International journal of computer vision, 88:303–338, 2010.
  * [9] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley.  Removing rain from single images via a deep detail network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3855–3863, 2017.
  * [10] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The KITTI dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [11] Christoph Kamann and Carsten Rother.  Benchmarking the robustness of semantic segmentation models with respect to common corruptions.  International journal of computer vision, 129:462–483, 2021.
  * [12] Chengyang Li, Dan Song, Ruofeng Tong, and Min Tang.  Multispectral pedestrian detection via simultaneous detection and segmentation.  arXiv preprint arXiv:1808.04818, 2018.
  * [13] Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang.  Deep hierarchical semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1246–1257, 2022.
  * [14] Qing Li, Aihua Liang, and Hongzhe Liu.  Hierarchical semantic segmentation of image scene with object labeling.  EURASIP Journal on Image and Video Processing, 2018:1–10, 2018\.
  * [15] Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown.  Rain streak removal using layer priors.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2736–2744, 2016.
  * [16] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman.  1 year, 1000 km: The oxford robotcar dataset.  The International Journal of Robotics Research, 36(1):3–15, 2017\.
  * [17] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In Proceedings of the IEEE international conference on computer vision, pages 4990–4999, 2017.
  * [18] Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool.  Model adaptation with synthetic and real data for semantic dense foggy scene understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [19] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Semantic foggy scene understanding with synthetic data.  International Journal of Computer Vision, 126(9):973–992, Sep 2018\.
  * [20] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10765–10775, 2021.
  * [21] Furqan Ahmed Shaik, Thejasvi Konduru, Girish Varma, and Kishore Kothapalli.  Accelerating computer vision tasks on gpus using ramanujan graph product framework.  In Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD), pages 113–117, 2023.
  * [22] Vishwanath A Sindagi, Poojan Oza, Rajeev Yasarla, and Vishal M Patel.  Prior-based domain adaptive object detection for hazy and rainy conditions.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 763–780. Springer, 2020.
  * [23] Linfeng Tang, Jiteng Yuan, Hao Zhang, Xingyu Jiang, and Jiayi Ma.  Piafusion: A progressive infrared and visible image fusion network based on illumination aware.  Information Fusion, 83:79–92, 2022.
  * [24] Alexander Toet.  The tno multiband image data collection.  Data in Brief, 15:249–251, 2017.
  * [25] Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawahar.  IDD: A dataset for exploring problems of autonomous navigation in unconstrained environments.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1743–1751. IEEE, 2019.
  * [26] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.  Internimage: Exploring large-scale vision foundation models with deformable convolutions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14408–14419, 2023.
  * [27] Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu.  Joint rain detection and removal from a single image with contextualized deep networks.  IEEE transactions on pattern analysis and machine intelligence, 42(6):1377–1393, 2019.
  * [28] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020.
  * [29] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez.  Wilddash - creating hazard-aware benchmarks.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [30] Oliver Zendel, Matthias Schörghuber, Bernhard Rainer, Markus Murschitz, and Csaba Beleznai.  Unifying panoptic segmentation for autonomous driving.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21351–21360, 2022.
  * [31] Xian Zhong, Shidong Tu, Xianzheng Ma, Kui Jiang, Wenxin Huang, and Zheng Wang.  Rainy wcity: A real rainfall dataset with diverse conditions for semantic driving scene understanding.  In International Joint Conference on Artificial Intelligence, pages 1743–1749, 2022.


# Learning Unbiased News Article Representations:
A Knowledge-Infused Approach* *This paper is partly supported by NSF: Research
Experience for Teachers (RET); Award Number:2055557

Sadia Kamal  Department of Computer Science
Oklahoma State University
<EMAIL_ADDRESS>Jimmy Hartford  Department of Mathematics
Cushing Public Schools
<EMAIL_ADDRESS> Jeremy Willis  Arunkumar Bagavathi
Department of Science & Technology
Sand Springs Public Schools
<EMAIL_ADDRESS>Department of Computer Science
Oklahoma State University
<EMAIL_ADDRESS>
###### Abstract

Quantification of the political leaning of online news articles can aid in
understanding the dynamics of political ideology in social groups and measures
to mitigating them. However, predicting the accurate political leaning of a
news article with machine learning models is a challenging task. This is due
to (i) the political ideology of a news article is defined by several factors,
and (ii) the innate nature of existing learning models to be biased with the
political bias of the news publisher during the model training. There is only
a limited number of methods to study the political leaning of news articles
which also do not consider the algorithmic political bias which lowers the
generalization of machine learning models to predict the political leaning of
news articles published by any new news publishers. In this work, we propose a
knowledge-infused deep learning model that utilizes relatively reliable
external data resources to learn unbiased representations of news articles
using their global and local contexts. We evaluate the proposed model by
setting the data in such a way that news domains or news publishers in the
test set are completely unseen during the training phase. With this setup we
show that the proposed model mitigates algorithmic political bias and
outperforms baseline methods to predict the political leaning of news articles
with up to $73\%$ accuracy.

###### Index Terms:

Fair model, Political leaning prediction, Mitigating news domain bias

##  I Introduction

News media houses have endured through time to disseminate political news to
the people while also influencing their political perceptions. They play a
vital role to impersonate public opinion on several issues like diseases [1],
elections [2], and natural calamities [3]. Recent events from COVID-19 prove
that news media bias is capable of polarizing individual ideologies [1, 4].
Moreover, the rise of the web and social media has also increased the capacity
of news media to propagate information at a rapid pace. Over the last decade,
several small-scale and focused online news forums have harnessed online
platforms to spread news content across the general public. News articles
reflect the news forums’ opinion on politicians, laws, and policies which in
turn defines the _political bias_ of a news media house. A news forum is
politically biased in the scale of _far-left_ to _far-right_ based on the
political ideology (republican/democrat in the USA) that they approve and
criticize in their news articles. An example of news domains from different
political leaning covering the story of ’attack on Mr. Pelosi’ news from
multiple angles is given in Figure 1. _Right-leaning_ news domains attribute
the problem of illegal immigrants, _left-leaning_ domains emphasize
Mrs.Pelosi’s criticisms on republicans, and _center-leaning_ news domains
cover details of the attack at a high-level.

Figure 1: Biased projections of news contents on the same story of ’Nancy
Pelosi Discusses Attack on Husband’ covered by news media houses with
political leaning of _left_ (blue), _center_ (white), and _right_ (red)

Politically biased news contents are some of the well known sources for the
widespread increase in filter bubbles [5], echo-chambers [6], misinformation
[7], and propaganda in online communities [8]. Our primary motivation in this
work is echo-chambers created by biased news recommendations in online forums
[9, 10]. In other words, user profiles with conservative ideology would prefer
news articles from right leaning news media like Fox News and CBN, whereas
profiles with liberal ideology may prefer news from left leaning news domains
like CNN and CBS News. Some of the current AI based recommendation models
intensify the echo chamber effect by giving news article suggestions to users
that align with their ideology and preferences. Thus there are works to refine
such recommender models to expose online users to news articles from news
domains of diverse political ideologies [11, 12, 13]. Thus characterizing
political bias in news articles is crucial to quantify its effects and take
measures to mitigate them in online forums like Twitter, Reddit, and Youtube.
Accurately mapping the bias of news articles will also provide social media
platforms and news recommendation platforms like Google News to show
personalized news from multiple political ideologies [14].

Algorithmic bias is one of the primary concerns in existing real-world AI
applications. The machine learning models by default undergo algorithmic bias
as the training data given to such models are collected on human assumptions
[15, 16]. Although these biased models give good performance on new data that
have similar characteristics to the training set, the models discriminate
features that are completely unseen during the training phase. Such model
biases are prevalent in many real-world scenarios like gender, race, and
religion [17]. Such unfairness in machine learning models reduces the human
reliability of model outcomes and their decision-making capabilities in
various domains like medicine, agriculture, and autonomous systems. In this
paper, we analyze machine learning classifiers undergoing _algorithmic
political bias_ of news domains that are available during model training and
make discriminatory predictions. We demonstrate in upcoming sections that
existing language models are politically biased on news domains and fail to
map the political leaning of news articles published by news domains that are
unseen during the training phase. It is important to mitigate such bias in the
learning models to (i) learn unbiased feature representations of news
articles, and (ii) predict political bias of news articles from news media
sources whose bias is not evidenced during model training.

The existing work [14] showcases the effect of political bias present in news
domains to predict the political leaning of news articles. Mitigating such
political bias in news articles is important to understand the political bias
of a news article from any new news domain. We introduce such a bias
mitigation method in a machine learning setup to learn news article
representations by infusing external knowledge. Our method is an improvement
over all the prior work on political leaning prediction of news articles, as
summarized in Table I. The proposed method of encoding features of topics
learned from external data resources can enable the machine learning models to
mitigate political bias in classification tasks. Overall, we have the
following _three-fold_ contributions in this paper:

  1. 1.

Proposed work: We propose a novel framework that mitigates political bias in
machine learning classifier models by infusing external knowledge sources for
political leaning prediction on news articles. Most importantly, the proposed
model uses representations of topics in news articles and knowledge of news
domains from external data sources in the prediction task

  2. 2.

Experiments: We give in-depth experiments on the performance of the proposed
framework using multiple training and test sets, multiple language models and
external knowledge sources. We show that the proposed model achieves up to
_73%_ prediction accuracy

  3. 3.

Mitigating political bias: We demonstrate the efficacy of the proposed
framework to mitigate political bias of news domains even when they are not
available during the model training. Specifically, we demonstrate robustness
by evaluating the proposed model with multiple splits of training and test
data based on news domains in our experiments

TABLE I: Qualitative comparison of aspects in the proposed work with existing methods Method |  Knowledge Encoding |  Multiple Data Sources |  Topic Encoding |  Bias Mitigation
---|---|---|---|---
Baly et al. [14] |  ✓ |  - |  - |  ✓
KGAP [18] |  ✓ |  - |  ✓ |  -
KCD [19] |  ✓ |  - |  ✓ |  -
Devatine et al. [20] |  - |  - |  - |  ✓
KHAN [21] |  ✓ |  ✓ |  ✓ |  -
Ours |  ✓ |  ✓ |  ✓ |  ✓

##  II Related Work

###  II-A Political Leaning Detection

Identifying political leaning has several applications in online social media
like echo chamber detection [22], quantifying controversy [23], and partisan
segregation identification [24]. Several research ideas are emerging to
utilize linguistic and temporal features that exist in online content to
characterize political bias and leaning. Few recent works [25, 26] study the
identification of political ideology of media houses from social media
networks like Twitter. Analyzing text data with deep learning methods have
been widely studied to capture political viewpoints. Methods like attention
mechanism [27] have been utilized with deep language encoder models on
contextual information to capture political perspective. An attention-based
multi-view model is proposed in [28] to identify the ideology of news articles
from the article details such as title, content and link structure. Opinion-
aware knowledge graph [29] has been proven efficient specially for graph based
approaches to infer and predict the ideology. MEAN [30] detects bias in terms
of entities that use entity information as external knowledge and leverage
that information in multiple ways to capture political perspective. KCD [19]
also detects political perspective with knowledge reasoning for graph level
representation learning methods. Political bias detection has been studied at
multiple levels like word-level, sentence-level [31], news article-level [32,
33] and news domain level [34]. Few recent works [28, 14, 35] try to infer the
political bias associated with the news article by using multiple methods such
as deep neural network architectures. Although the work proposed in [14] is
closely similar to our work, there are major differences. In this paper, we
develop a novel knowledge-infused deep learning model to extract news article
representations to predict their political leaning. We use two external
sources to add both global and fine-grained contextual information to the news
articles.

###  II-B Model Fairness

Traditional machine learning and deep learning models are mostly designed to
optimize based on any performance metric such as accuracy, which can lead to
learning bias or unfairness [16] to the model which can affect the prediction
task. Previously, there has been work like elimination and measuring of gender
bias [36], neuro-imaging dataset bias [37] from machine learning models and
deep neural network frameworks. The bias in the texts associated with the
sources [38] and users can be directly attributed to those sources and users.
There are few works that try to detect the media bias, particularly the work
given in [39] predicts the media bias by link-based approach. Having articles
from the same news domain can make the model biased [40] because the
classifiers may start making predictions based on the political leaning of
news domain itself rather than the content of the article. We attempt to
mitigate such model’s algorithmic political bias in our work by _media-based
split_ datasets and a proposed knowledge infusion method.

##  III Datasets

In this work, we utilize datasets from multiple resources for knowledge
infusion on news article representations. In this section, we give details of
all the datasets.

TABLE II: Demonstration of algorithmic political bias in classifier models and the importance of external data sources - Wikipedia Dataset | Acc. | Macro F1 | MAE
---|---|---|---
NewsA (Random split) | 0.749 | 0.7456 | 0.3774
NewsA (Media split) | 0.516 | 0.4804 | 0.5907
WikiA | 0.562 | 0.4040 | 0.5625
NewsA (Media split) + WikiA | 0.685 | 0.6290 | 0.6011

###  III-A Labeled News Articles

We utilize labeled news articles from the existing research [14] in our
experiments. This dataset comprises _37,554_ news articles from _389_ news
domains and each news article is associated with one of the three political
leaning labels: _left_ , _center_ , and _right_. Prediction models trained
using the traditional approach of randomly splitting the data into training
and test datasets will be unfair. Meaning that such models cannot generalize
to predict the political leaning of news articles from an unknown news domain,
as given in _row 2_ (Madia split) of Table II.

To analyze and mitigate the model bias of learning models, we split the news
articles by news domains into training and test sets. Such data splits are
represented as _Media split_ throughout this paper and this data split will
test the performance of classifier models to predict the political leaning of
news articles from completely unknown news domains during the model testing
phase. Unlike the existing work [14], which hand pick the news domains for the
test set we randomly choose $7\%$ of the news domains from our data and their
corresponding news articles for our test set. Table II demonstrates the
performance of a classifier model built on top of a BERT language model to
predict the political bias of news articles. Our experiments aligns with the
existing work [14], although ours perform better than their results, to show
that the learning models underperform when they get articles from unknown news
domains and it requires mitigation measures. The distribution of 4 _Media
split_ data and one _random split_ data used in our experiments is given in
Figure 2(a). It is notable from the Figure 2(b) that the number of news
domains in our news articles is relatively minimum. We claim that this
imbalanced distribution does not bias the proposed model as the number of
articles in the _right_ political ideology is approximately equal to the
number of news articles in other political ideologies from Figure 2(a).

(a) Distribution of political leaning labels training and test sets in one
random split and four media-based data splits

(b) Distribution on the number of news domains in the news articles dataset,
the number of Wikipedia articles and the number of speech in presidential
debates in _left_ , _center_ , and _right_ ideologies

Figure 2: Data distribution of news articles, news domains, Wikipedia
articles, and Presidential debates characterized by their political ideologies
Figure 3: Architecture of our proposed knowledge-infused deep learning model.
The proposed model infuse weighted Wikipedia representations ($\omega$) and
topics representations ($\tau$) to the corresponding base representations
($\delta$) of a news article $n_{j}$

###  III-B External Data Sources

It is well studied research that adding contextual information from external
data sources improve the performance of supervised machine learning models
[14, 26, 41, 21]. Our proposed method use two external datasets to mitigate
algorithmic political bias in political leaning prediction models. We have
released the datasets to the research community
here111https://github.com/sadiakamal/Learning-Unbiased-News-Article-
Representation. Figure 2(b) gives the distribution of the number of documents
per class label.

####  III-B1 Wikipedia Articles (WKB)

One of the datasets that we use to mitigate model bias are Wikipedia pages of
news domains. We use the Wikipedia API to collect the title and body of _324_
news domain that exists in our dataset. There are no wikipedia pages for the
other 65 news domains. We used wikipedia articles in the proposed model to
capture global context of news articles in the form of news domains. The
observations in _row 3_ of Table II clearly demonstrates that Wikipedia
article representations, learned from a pre-trained BERT model without
incorporating any news articles representations, can effectively mitigate
algorithmic political bias in the prediction model. This approach resulted in
a $5\%$ improve in performance when evaluated with the test data of the _media
split_. We can also notice a significant performance spike of $68.5\%$ model
accuracy when using Wikipedia representations along with news article
representations on the media split.

####  III-B2 Presidential Debates (PDB)

In addition to Wikipedia articles, we use presidential debates [42] to capture
contextual representations of topics that are present in news articles. This
dataset consists of debates of presidential candidates from both the
republican and democratic parties. The main motivation to utilize this dataset
is two-fold: (i) to capture the fine-grained representations of topics that
appear in news articles, and (ii) to rely on relatively reliable external
knowledge sources instead of utilizing topic representations derived from much
unreliable and noisy data sources such as social media posts [14, 26, 21]. As
depicted in Figure 2(b), our presidential debates dataset (PDB) comprises of
an almost balanced distribution of 478 speeches of Democrats and 389 speeches
of Republicans. We also highlight that the proposed approach extracts topic
representations from the entire debate data rather than handling them
separately by their political ideology. So, the absence of speeches from the
_center_ ideology and the presence of a minor imbalance in the debates data
cannot bias the proposed model.

##  IV Methodology

Our proposed framework is a knowledge infused deep network architecture, as
illustrated in Figure 3, that mitigates algorithmic political bias that exists
in classifier models to predict the political leaning of news articles. Also,
we summarize the list of notations used in this paper in Table III.

TABLE III: Description of notations used in this paper Notation |  Description
---|---
$\mathcal{N}$ |  Set of news articles
l |  Number of news articles
$\mathcal{D}$ |  Set of news domains
m |  Number of news domains
$T_{n_{j}}$ |  Set of topics in a news article $n_{j}$
$C_{n_{j}}$ |  Political leaning of news article $n_{j}$
$\delta$ |  Feature representation of news articles
$\omega$ |  Feature representation of wikipedia articles
$q$ |  Number of dimensions in $\delta$ and $\omega$
$\tau$ |  Feature representation of topics
$r$ |  Number of dimensions in $\tau$
$\beta$ |  Weight of external knowledge representations
$\lambda$ |  Weighted external knowledge representations
$\Theta$ |  Knowledge infused news article representations
p |  Number of dimensions in $\Theta$

Assume the training set consists of $l$ news articles which are represented as
$\mathcal{N}=\\{n_{1},n_{2},n_{3},...,n_{l}\\}$, where each article
$n_{j}\in\mathcal{N}$ is published by a news domain $d_{k}\in\mathcal{D}$
where $\mathcal{D}=\\{d_{1},d_{2},d_{3},...,d_{m}\\}$, given that $m<<<l$, and
$n_{j}\in\mathcal{N}$ is associated to one of the political leaning classes
$\mathcal{C}_{n_{j}}\in\\{0,1,2\\}$ (0: Left, 1: Center, 2: Right). Each news
article $n_{j}\in\mathcal{N}$ is in turn consists of a series of topics
represented as $T_{n_{j}}=\\{t_{1},t_{2},t_{3},\ldots\\}$, given that our
model does not set any limits on the number of topics in the news article
$n_{j}$. In addition to the news articles, we also have Wikipedia knowledge
base (WKB) with $m$ wikipedia pages and debates database (PDB). The WKB
provides contextual information about news domains that publish news articles
$\mathcal{N}$, while the PDB focuses on the topics discussed in news articles
$\mathcal{N}$.

Given a set of news articles $\mathcal{N}$ and their corresponding external
data sources, the proposed model learns a $p-$dimensional representation
$\Theta_{n_{j}}\in\mathbb{R}^{l\times p}$ of a news article $n_{j}$ where
$p=2q+r$. We learn the news article representation as
$\Theta_{n_{j}}=\delta_{n_{j}}\oplus\omega_{n_{j}}\oplus\tau_{n_{j}}$, where
$\delta_{n_{j}}\in\mathbb{R}^{l\times q}$ is the base representation of
$n_{j}$, $\omega_{n_{j}}\in\mathbb{R}^{l\times q}$ is the representation of a
Wikipedia article that corresponds to the news domain $d_{k}$ which published
$n_{j}$, and $\tau_{n_{j}}\in\mathbb{R}^{l\times r}$ is the aggregated
representation of topics in $n_{j}$. The proposed model learns all the above
representations jointly in a supervised fashion as $f(\Theta)\rightarrow
Pr(\mathcal{C}_{n_{j}}|\Theta)$ where $\mathcal{C}_{n_{j}}\in$ $\\{0,1,2\\}$
is the political leaning of news article $n_{j}\in\mathcal{N}$. We give
details about the proposed model to learn $\Theta_{n_{j}}$, $\delta_{n_{j}}$,
$\omega_{n_{j}}$, and $\tau_{n_{j}}$ below.

###  IV-A Base Representation $\delta_{n_{j}}$

We use the backbone transformer models to extract the base news article
representations $\delta_{n_{j}}\in\mathbb{R}^{l\times q}$. We choose
transformer models over classic text representation models because of its
ability to learn contextualized representations rather than static word
embeddings. Given that there are $w$ words in a news article $n_{j}$, we
aggregate the word representations from the transformer model using a MEAN
operator. We fine-tune the parameters of the backbone model in a supervised
setup $f(\Theta)$ as mentioned before.

1:Topics of news article $n_{j}\in N$: $T_{n_{j}}$, Pre-trained word embedding
model: $\mathcal{W}$, knowledge weight variable: $\beta$

2:Initialize $E\leftarrow\phi$

3:for topic $t\in T_{n_{j}}$ do

4: $E\leftarrow E+\mathcal{W}[t]$

5:end for

6:$M\leftarrow\frac{E}{|T_{n_{j}}|}$ $\triangleright$ Mean of all word vectors

7:$\tau_{n_{j}}\leftarrow$ Encode($M$) to $r$-dimensional vector

8:$\tau_{n_{j}}\leftarrow(1-\beta)\times\tau_{n_{j}}$ $\triangleright$
weighted topic representation of $n_{j}$

9:return $\tau_{n_{j}}$

Algorithm 1 Topic Knowledge Extraction

###  IV-B Knowledge Infusion

We further enhance the base representations of news articles
($\delta_{n_{j}}$) with the global contextual representations of news articles
($\lambda_{n_{j}}$) collected from external data sources to debias the
learning model $f(\Theta)$. In this paper, we utilize Wikipedia data (WKB) and
Presidential debates data (PDB) to learn the global representations
$\lambda_{n_{j}}$ of a news article $n_{j}\in\mathcal{N}$. Each of our
external data covers a different context of global representations. That is
the representations $\omega_{n_{j}}$ from WKB consists of global context of
the news domain $d_{n_{j}}$ and the representations $\tau_{n_{j}}$ from PDB
captures the unmanipulated context of topics $T_{n_{j}}$ present in news
articles. We primarily use political debates to learn topic representations,
rather than general sources like social media posts or Wikipedia, as they are
significant to the political polarization problem and they are relatively
reliable to perceive the political context of topics.

####  IV-B1 Wikipedia Representation $\omega_{n_{j}}$

We first map the news domain $\mathcal{D}_{n_{j}}$ of a news article $n_{j}$
to its corresponding Wikipedia article $W_{n_{j}}$ from the Wikipedia data
base WKB using the mapping function $g_{wkb}:\mathcal{D}_{n_{j}}\rightarrow
W_{n_{j}}$. We learn the Wikipedia representations $\omega_{n_{j}}$ of news
article $n_{j}$ with the same backbone transformer model used to learn the
base representations of $n_{j}$. The proposed model optimizes the backbone
transformer model parameters based on the classification task $f(\Theta)$ to
learn $\omega_{n_{j}}$.

(a) The Encoder model receives r=300-dimensional input text representation and
maps it to a lower dimensional (r=200)representation

(b) The Autoencoder consists of two main components: an encoder and a decoder.
The encoder part works the same. The decoder part then reconstructs the input
to the original 300-dimensional representation.

Figure 4: We using an Autoencoder and Encoder model to learn topic
representations $\tau_{n_{j}}$, where each $\tau_{n_{j}}$ represents a topic.
Autoencoder model to learn topic representations $\tau_{n_{j}}$ and we use
ReLU() as the activation function in the hidden layers.

####  IV-B2 Topic representation $\tau_{n_{j}}$

Algorithm 9 gives the complete pipeline to extract the topics representation
$\tau_{n_{j}}$ of a news article $n_{j}$ with topics
$T_{n_{j}}=\\{t_{1},t_{2},t_{3},\ldots\\}$ from our presidential speeches
knowledge base (PDB). We collect topic representation $\tau_{n_{j}}$ for each
base representation $\delta_{n_{j}}$ and this is our another infused knowledge
base to enhance the learning. We do not set any limits on the number of topics
to appear in the news article. Capturing this information allow us to predict
the political leaning of a article with political speech based topic
representation learning context. First, we pre-train a skip-gram word
embedding model $\mathcal{W}$ (_Word2Vec_ [43] in this paper) on the PDB data
to get contextual text features of words in the context of PDB. We then
collect representations for each topic $t\in T_{n_{j}}$ using the skip-gram
model and take mean of representations from all topics in news article
$n_{j}$. We further enhance topics representation with two types of encoding
in the proposed model: i) Encoder model ii) Autoencoder model, as given in
Figures 4(a) and 4(b) respectively. For encoder model we use two linear layers
and one ReLU layer. Similarly for the autoencoder model we use two linear
layer and one ReLU layer in both encoder and decoder part. As given in
Algorithm 9, we extract weighted topic representation $\tau_{n_{j}}$ using our
knowledge weight variable $(1-\beta)$.

####  IV-B3 Knowledge representations $\lambda_{n_{j}}$

We propose the knowledge representations of $n_{j}$ as the weighted
aggregation of Wikipedia representations ($\omega_{n_{j}}$) and presidential
debates representations ($\tau_{n_{j}}$). We use the weight variable
$\beta\in[0,1]$ to optimize the importance of representations from two
different knowledge bases as given in Equation 1.

$\lambda_{n_{j}}=[\beta\times\omega_{n_{j}}]\bigoplus[(1-\beta)\times\tau_{n_{j}}]$
(1)

###  IV-C News Article Representations $\Theta$

With base representations $\delta$ learned from a news article and its
corresponding knowledge representations $\lambda$, we extract the knowledge-
infused news article representations as $\Theta=\delta\bigoplus\lambda$. We
further fine-tune all three representations of news articles ($\delta$,
$\omega$, and $\tau$) by training a supervised model $f(\Theta)$ to predict
political leaning of a news article.

###  IV-D Training the Proposed Model

The concatenated knowledge-infused representations $\Theta$ captures the high
level information. Then $\Theta$ is fed to a fully connected linear layer
followed by a ReLU layer as shown in Equation 2 to predict the political
leaning of news articles. Here _W_ is parameter matrix and _b_ is the bias
associated.

$Pr(\mathcal{C}_{n_{j}}|\Theta)=ReLU(W\Theta+b)$ (2)

We optimize the knowledge-infused news article representations for the
supervised task using the multi-class cross entropy loss function, as given in
Equation 3, to optimize model parameters.

$\mathcal{L}=-\sum_{\mathcal{C}_{n_{j}}=1}^{M}y_{\mathcal{C}_{n_{j},\Theta}}log(Pr(\mathcal{C}_{n_{j}}|\Theta))$
(3)

where $\mathcal{C}_{n_{j}}$ is the training target label with respect to
$\theta$, M is the number of classes and y is binary indicator that represents
correct or incorrect classification of the class label. In this paper, we set
$M=3$ (_left_ , _center_ , and _right_).

TABLE IV: Impact of infusing external knowledge to news article representations in the prediction task. All model results are given in Accuracy(%), Precision, Recall, Macro F1, and Mean Average Error (MAE). The proposed knowledge-infused approach with encoder module to learn topic representations ($\tau$) gives significant performance boost over the baseline approaches in terms of F1 score. Experiment Name |  Acc. |  Precision |  Macro F1 |  Recall |  MAE
---|---|---|---|---|---
NewsA(RandomSplit) |  0.749 |  0.7350 |  0.7456 |  0.8145 |  0.3774
Baly et al.(2020)(NewsA) [14] |  0.3675 |  - |  0.3553 |  - |  0.90
Devatine et al.(2022)(Bi-LSTM) [20] |  0.4697 |  - |  0.4441 |  - |  0.69
NewsA(Our method) |  0.516 |  0.5325 |  0.4804 |  0.5319 |  0.5907
WikiA |  0.562 |  0.4722 |  0.4040 |  0.3650 |  0.5625
Baly et al.(2020)(NewsA+WikiA) [14] |  0.4975 |  - |  0.5116 |  - |  0.32
Devatine et al.(2022)(Bi-LSTM + SA/Sent) [20] |  0.4876 |  - |  0.4584 |  - |  0.67
Liu et al.(2018)+SA/EDU [44] |  0.5101 |  - |  0.4861 |  - |  0.72
Devatine et al.(2022)(Bi-LSTM + SA/EDU) [20] |  0.5439 |  - |  0.5136 |  - |  0.57
NewsA+WikiA |  0.685 |  0.5535 |  0.6290 |  0.6835 |  0.6011
Khan et al.(2023)) [21] |  0.3240 |  0.33 |  0.161 |  0.10 |  1.03
Baly et al.(2020)(NewsA+Twitter) [14] |  0.72 |  - |  0.6429 |  - |  0.2900
NewsA+WikiA+Topic(AE) |  0.5867 |  0.6067 |  0.5344 |  0.5936 |  0.6184
NewsA+WikiA+Topic(E) |  0.73 |  0.7232 |  0.7288 |  0.7584 |  0.3676

##  V Experiments and Results

In this section, we evaluate our proposed knowledge-infused model with
multiple experimental setup for political leaning detection task. We give
results based on multiple transformer-based text representation models,
assessed the model fairness with multiple datasets, and analyze the impact of
weight parameter $\beta$ on knowledge representations infused with base news
article representations $\delta_{n_{j}}$.

###  V-A Experimental Setup

Before presenting our experimental results, we will detail the implementation
details and notations of the baseline models in the below sections.

####  V-A1 Implementation details

In this work, we use PyTorch 1.12.1 [45] to implement our proposed model. The
machine we use for all the experiments is equipped with an Intel X710 quad
port CPU with 64.14 GB memory and an NVIDIA Ampere A10 GPU with 24 GB memory,
the GPU is installed with CUDA 11.3 and cuDNN8.3.2. We set the batch size as 2
and we use Adam optimizer [46] with learning rate = 1e-6. Lastly, we set the
number of epochs as 3.

####  V-A2 Baseline models

We compare the performance of our proposed model with the performance reported
in the following state-of-the-art baselines on the same media split dataset.
Since there is not much work in the literature to predict the political
leaning of news articles, we use multiple modules in our proposed model as
baselines as well. Unless or otherwise specified as _Random Split_ , all the
models used in our experiments are trained with _Media Based_ data split. All
the baseline models used in this paper are given below:

  * •

Devatine et al.(2022) (Bi-LSTM) [20] This model is their base model and it
leverages sequences of static word embedding to generate hidden
representations. This model is modified to incorporate the enhancements
proposed in [47] and it achieves $46.97\%$ accuracy.

  * •

Devatine et al.(2022) (Bi-LSTM+SA/Sent) [20] This model is an extension of the
Bi-LSTM model where it leverages structural attention and sentence
segmentation to the existing Bi-LSTM model to give $48.76\%$ accuracy.

  * •

Devatine et al.(2022) (Bi-LSTM+SA/EDU) [20] In particular, the model performs
better when they incorporate Elementary Discourse Units(EDU) segmentation(Bi-
LSTM + SA/EDU) to the base Bi-LSTM model. Identifying text spans(EDU) is
linked with discourse relations to improve the overall model performance.

  * •

Liu et al.(2018) + SA/EDU The original model proposed in [44] leverages
structural attention and sentence segmentation performs well without the
improvements introduced in [20].

  * •

Baly et al.(2020)(NewsA) [14], Baly et al.(2020) (NewsA+ WikiA) [14], and Baly
et al.(2020)(NewsA+Twitter) [14] This work is the closest resemblance to the
proposed model. But this work utilizes Twitter follower Bios, which can be
uncertain on political leaning, and Wikipedia articles to predict the
political leaning of news articles. Also, this work has not given any
experiments to evaluate the model fairness on multiple datasets.

  * •

WikiA: We report the performance of the proposed model only with Wikipedia
articles representation from WKB and not including any base representations
$\delta_{n_{j}}$.

  * •

NewsA+WikiA: This is the proposed model which learns the Wikipedia
representation $\omega_{n_{j}}$ together with news article representations
$\delta_{n_{j}}$ to predict the political leaning of the news article $n_{j}$.

  * •

NewsA+WikiA+Topic(E) and NewsA +WikiA+Topic(AE): These are the proposed models
which utilizes Wikipedia $\omega_{n_{j}}$ and topic representations
$\tau_{n_{j}}$ with the base news article representations where _E_ and _AE_
denotes Encoder and Autoencoder versions of the model respectively.

####  V-A3 Language Models

We utilize the existing transformer-based text representation learning models
in our proposed model. We primarily use BERT [48] in all our experiments.
However, we also give results by replacing BERT with other popular language
models like _RoBERTa_ [49] and _DistillBERT_ [50] to learn base
representations $\delta_{n_{j}}$ and Wikipedia representations
$\omega_{n_{j}}$ of a news article $n_{j}$.

###  V-B Results

Table IV gives an overview of the results of all our experiments using BERT to
obtain base representations $\delta_{n_{j}}$ and Wikipedia representations
$\omega_{n_{j}}$ of a news article $n_{j}$. We replicate some of the results
in Table II in our results for extensive comparison of our model performance.
this experiment, we set $\beta=0.5$ to give more importance to topics and all
the results in Table IV are based on the Media-based split(1) dataset.

We can note from Table IV that the performance of the base BERT model on only
news articles differs significantly for the random and media-based data
splits. This corroborates our claim that models are not able to generalize the
political learning from unseen news domains. It is worth mentioning here that
minor parameter tuning on BERT model increase the model accuracy by $15\%$
compared to the existing work [14]. To reduce the model bias we evaluated
several experiments by infusing external knowledge sources.

We first compare the model performance by only utilizing Wikipedia
representations $\omega_{n_{j}}$ to predict the political leaning of a news
article $n_{j}$. Similar to the previous experiment, we notice that the
parameter fine-tuning on the BERT model increases the model accuracy by
$16.9\%$ compared to the model that uses only base representations while the
existing method [14] under performs in a same setup. Also, our proposed model
outperforms all other baseline works in terms of performance measures. Our
experiments improves the accuracy by $17\%$ than Liu et al.(2018)+SA/EDU [44].
Furthermore, it increases accuracy by $19.8\%$ and $14\%$ than Devatine et
al.(2022) (Bi-LSTM+SA/Sent) and Devatine et al.(2022) (Bi-LSTM+SA/EDU) [20]
respectively, when we utilize Wikipedia representations $\omega_{n_{j}}$.
Next, we compare the proposed model that infuses external knowledge
representations $\lambda_{n_{j}}$ of a news article $n_{j}$ with the base
representations $\delta_{n_{j}}$, where $\lambda_{n_{j}}$ is the weighted
aggregation of Wikipedia representations $\omega_{n_{j}}$ from WKB and topic
representations $\tau_{n_{j}}$ from PDB. It is evident from Table IV that the
proposed model with _Encoder_ setup outperforms the existing work [14] and the
_Autoencoder_. The encoder version of the proposed model gives an accuracy of
$73\%$ to predict the political leaning of news articles whose news domains
are completely invisible during the training phase. Even though the
performance of the Auto Encoder version of the proposed model is almost equal
to that of the existing work [14], we emphasize that the proposed model gives
a significantly better performance in terms of _F1 score_.

TABLE V: Impact of varying the importance of representations $\omega_{n_{j}}$ and $\tau_{n_{j}}$ with the weight parameter $\beta$ on the proposed model performance Model | $\beta$
---|---
0.0 | 0.1 | 0.5 | 0.7 | 1.0
E | Acc. | 0.5044 | 0.5704 | 0.73 | 0.6850 | 0.6474
| Mac. F1 | 0.4801 | 0.5885 | 0.7232 | 0.6321 | 0.6452
AE | Acc. | 0.5066 | 0.5824 | 0.5867 | 0.4191 | 0.4027
| Mac. F1 | 0.4675 | 0.5196 | 0.5344 | 0.3906 | 0.3587
TABLE VI: The proposed model performance with different backbone language models. Both BERT and RoBERTa models are able to mitigate algorithmic political bias with the proposed knowledge-infusion approach Model |  Acc. (%) |  Precision |  Macro F1 |  Recall |  MAE
---|---|---|---|---|---
RoBERTa |  (NewsA) |  0.361 |  0.333 |  0.176 |  0.120 |  0.963
|  NewsA+WikiA+Topic(E) |  0.584 |  0.607 |  0.510 |  0.710 |  0.637
|  NewsA+WikiA+Topic(AE) |  0.525 |  0.544 |  0.451 |  0.672 |  0.511
DistilBERT |  (NewsA) |  0.497 |  0.511 |  0.470 |  0.497 |  0.670
|  NewsA+WikiA+Topic(E) |  0.323 |  0.333 |  0.163 |  0.107 |  1.036
|  NewsA+WikiA+Topic(AE) |  0.323 |  0.333 |  0.163 |  0.107 |  1.036
BERT |  (NewsA) |  0.516 |  0.532 |  0.480 |  0.531 |  0.590
|  NewsA+WikiA+Topic(AE) |  0.5867 |  0.6067 |  0.5344 |  0.5936 |  0.6184
|  NewsA+WikiA+Topic(E) |  0.73 |  0.7232 |  0.7288 |  0.7584 |  0.3676

In Table V we show the impact of varying the value of weight parameter $\beta$
on both Encoder and AutoEncoder versions of the proposed model. We notice that
both models give their best performance with $\beta=0.5$, which means we set
equal priority to both knowledge sources. We verify the efficacy of our
external knowledge bases, and the results illustrate the both global knowledge
base and topic based knowledge plays an important role in terms of reducing
the algorithmic political bias and improve in predicting the political leaning
of news articles.

Figure 5: Accuracy (%) of the proposed model with only base representations
(_NewsA_) and with knowledge-infused representations (_NewsA+WikiA+Topic(E)_)
on multiple media-based data splits. The knowledge-infused approach has a
significant performance improvement over the base model in mitigating
algorithmic political bias

In order to demonstrate the generalizing capabilities of the proposed model,
we evaluate the proposed model on multiple Media-based data splits. From the
Figure 5, we notice that the algorithmic political bias is present in the base
representation without any external knowledge on all the Media-based split
datasets. Figure 5 also displays the effectiveness of our proposed framework
across multiple train and test set combination from our news article
dataset.It is important to analyze the performance of the model by varying the
dataset during evaluation. We show the performance of our model on different
Media-based splits to illustrate robustness and generalizability of the model.

To demonstrate the efficacy of our proposed model we use multiple language
models like RoBERTa and DistillBERT. Table VI confirms that among all the
models, BERT representation model outperforms other language models in both
Encoder and AutoEncoder versions. BERT is a strong baseline model, as it was
pre-trained on a large corpus of text and has been fine-tuned on various
downstream tasks. RoBERTa and DistillBERT are both variations of BERT,
RoBERTa, for example, uses additional pre-training data and modified pre-
training techniques, while DistillBERT uses a knowledge distillation approach
to reduce the computational resources required for training. Although
efficient, we noticed that the proposed model trained with DitillBERT gives
poor performance and it is unable to mitigate the algorithmic political bias
in our prediction task. Table VI also corroborates the fact that BERT is
better suited than RoBERTa or DistillBERT for capturing certain linguistic
features or for making political leaning prediction.

##  VI Conclusion and Future Work

In this work, we attempt to mitigate algorithm political bias of machine
learning algorithms by infusing external knowledge sources like Wikipedia and
political debates to predict the political leaning of news articles. We
proposed a novel way to learn weighted feature representations of entities or
topics present in presidential debate records by carefully mapping them in
news articles. With a series of experiments, we notice that external knowledge
sources can debias base feature representations of news articles and thereby
improving the performance of the prediction model by outperforming the
baseline approaches in terms of prediction accuracy (%) and F1 score. We
demonstrate the effectiveness of our proposed knowledge-infused model by
conducting several experiments on a variety of media-based data splits, and
with multiple base language models.

The proposed model and the problem of predicting the political leaning of news
articles have several future directions. We discuss some of them below:

###  VI-A Quantifying Political Bias on Topics

The proposed work and existing work characterize political bias on fine-
grained entities like users, social media posts, and news articles. An
interesting research direction can be identifying and forecasting political
bias of topics and events with news articles as surrogate information using
machine learning. Such models can quantify social responses to news content on
a given topic or story which can help social media engineers to give
supporting information on topics for news articles from a given news domain.

###  VI-B Mitigating echo chambers

It is well known that the political bias in news domains creates echo chamber
effect in social media communities. Identifying the political leaning of news
articles can assist analysts to engineer measures to mitigate echo chamber
effect in social media and online discussion forums. This will open
opportunities to develop ML algorithms that recommend news contents which are
similar in story that the user prefers but in different projections of the
story. Another potential research direction can be mitigating polarized online
discussions among multiple users who argues about political scenarios that
appeared on news articles.

###  VI-C Dynamics of polarization in online forums

Since the political leaning of news domains change over time, mitigating the
political bias in a temporal aspect can also be another interesting future
direction in this work. Understanding the dynamics of political bias of topics
in online forums can also be another potential research direction given the
importance of quantifying political bias of topics.

## References

  * [1] M. A. AlAfnan, “Covid 19-the foreign virus: Media bias, ideology and dominance in chinese and american newspaper articles,” _International Journal of Applied Linguistics and English Literature_ , vol. 9, no. 1, pp. 56–60, 2020.
  * [2] K. Beckers, “The voice of the people in the news: A content analysis of public opinion displays in routine and election news,” _Journalism Studies_ , vol. 21, no. 15, pp. 2078–2095, 2020.
  * [3] F. Ghassabi, F. Zare-Farashbandi _et al._ , “The role of media in crisis management: A case study of azarbayejan earthquake,” _International Journal of Health System and Disaster Management_ , vol. 3, no. 2, p. 95, 2015\.
  * [4] W. Chipidza, “The effect of toxicity on covid-19 news network formation in political subcommunities on reddit: An affiliation network approach,” _International Journal of Information Management_ , vol. 61, p. 102397, 2021\.
  * [5] P. Liu, K. Shivaram, A. Culotta, M. A. Shapiro, and M. Bilgic, “The interaction between political typology and filter bubbles in news recommendation algorithms,” in _ACM WWW_ , 2021, pp. 3791–3801.
  * [6] K. Garimella, G. De Francisci Morales, A. Gionis, and M. Mathioudakis, “Political discourse on social media: Echo chambers, gatekeepers, and the price of bipartisanship,” in _ACM WWW_ , 2018, pp. 913–922.
  * [7] K. Shu, S. Wang, and H. Liu, “Beyond news contents: The role of social context for fake news detection,” in _ACM WSDM_ , 2019, pp. 312–320.
  * [8] M. Osmundsen, A. Bor, P. B. Vahlstrup, A. Bechmann, and M. B. Petersen, “Partisan polarization is the primary psychological motivation behind political fake news sharing on twitter,” _American Political Science Review_ , vol. 115, no. 3, pp. 999–1015, 2021.
  * [9] G. Pennycook and D. G. Rand, “Fighting misinformation on social media using crowdsourced judgments of news source quality,” _PNAS_ , vol. 116, no. 7, pp. 2521–2526, 2019.
  * [10] F. Huszár, S. I. Ktena, C. O’Brien, L. Belli, A. Schlaikjer, and M. Hardt, “Algorithmic amplification of politics on twitter,” _PNAS_ , vol. 119, no. 1, p. e2025334119, 2022.
  * [11] N. Gillani, A. Yuan, M. Saveski, S. Vosoughi, and D. Roy, “Me, my echo chamber, and i: Introspection on social media polarization,” in _ACM WWW_ , 2018, pp. 823–831.
  * [12] R. Levy, “Social media, news consumption, and polarization: Evidence from a field experiment,” _American economic review_ , vol. 111, no. 3, pp. 831–870, 2021.
  * [13] Y. Jeon, B. Kim, A. Xiong, D. Lee, and K. Han, “Chamberbreaker: mitigating the echo chamber effect and supporting information hygiene through a gamified inoculation system,” _ACM SIGCHI_ , vol. 5, pp. 1–26, 2021.
  * [14] R. Baly, G. Da San Martino, J. Glass, and P. Nakov, “We can detect your bias: Predicting the political ideology of news articles,” in _EMNLP_ , 2020, pp. 4982–4991.
  * [15] R. Baeza-Yates, “Data and algorithmic bias in the web,” in _Web Science_ , 2016, pp. 1–1.
  * [16] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” _ACM CSUR_ , vol. 54, no. 6, pp. 1–35, 2021.
  * [17] I. Solaiman and C. Dennison, “Process for adapting language models to society (palms) with values-targeted datasets,” _NIPS_ , vol. 34, pp. 5861–5873, 2021.
  * [18] S. Feng, Z. Chen, W. Zhang, Q. Li, Q. Zheng, X. Chang, and M. Luo, “Kgap: Knowledge graph augmented political perspective detection in news media,” _arXiv preprint arXiv:2108.03861_ , 2021.
  * [19] W. Zhang, S. Feng, Z. Chen, Z. Lei, J. Li, and M. Luo, “Kcd: Knowledge walks and textual cues enhanced political perspective detection in news media,” in _NAACL_ , 2022.
  * [20] N. Devatine, P. Muller, and C. Braud, “Predicting political orientation in news with latent discourse structure to improve bias understanding,” in _3rd Workshop on Computational Approaches to Discourse (CODI 2022)_. ACL, 2022, pp. 77–85.
  * [21] Y. Ko, S. Ryu, S. Han, Y. Jeon, J. Kim, S. Park, H. Han, Kyungsik Tong, and S.-W. Kim., “Khan: Knowledge-aware hierarchical attention networks for accurate political stance prediction,” in _ACM WWW_ , 2023.
  * [22] P. Barberá, J. T. Jost, J. Nagler, J. A. Tucker, and R. Bonneau, “Tweeting from left to right: Is online political communication more than an echo chamber?” _Psychological science_ , vol. 26, no. 10, pp. 1531–1542, 2015\.
  * [23] K. Garimella, G. D. F. Morales, A. Gionis, and M. Mathioudakis, “Quantifying controversy on social media,” _ACM Transactions on Social Computing_ , vol. 1, no. 1, pp. 1–27, 2018.
  * [24] M. Conover, J. Ratkiewicz, M. Francisco, B. Gonçalves, F. Menczer, and A. Flammini, “Political polarization on twitter,” in _AAAI_ , vol. 5, no. 1, 2011, pp. 89–96.
  * [25] P. Stefanov, K. Darwish, A. Atanasov, and P. Nakov, “Predicting the topical stance and political leaning of media using tweets,” in _Proceedings of ACL_ , 2020, pp. 527–537.
  * [26] R. Baly, G. Karadzhov, J. An, H. Kwak, Y. Dinkov, A. Ali, J. Glass, and P. Nakov, “What was written vs. who read it: News media profiling using text analysis and social media context,” in _Proceedings of ACL_ , 2020, pp. 3364–3374.
  * [27] C. Li and D. Goldwasser, “Using social and linguistic information to adapt pretrained representations for political perspective identification,” in _ACL-IJCNLP 2021_ , 2021, pp. 4569–4579.
  * [28] V. Kulkarni, J. Ye, S. Skiena, and W. Y. Wang, “Multi-view models for political ideology detection of news articles,” in _ACL EMNLP_ , 2018, pp. 3518–3527.
  * [29] W. Chen, X. Zhang, T. Wang, B. Yang, and Y. Li, “Opinion-aware knowledge graph for political ideology detection.” in _IJCAI_ , vol. 17, 2017, pp. 3647–3653.
  * [30] C. Li and D. Goldwasser, “MEAN: Multi-head entity aware attention networkfor political perspective detection in news media,” in _Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda_. ACL, 2021, pp. 66–75.
  * [31] S. Bhatia and D. P, “Topic-specific sentiment analysis can help identify political ideology,” in _the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , Oct. 2018, pp. 79–84.
  * [32] R. R. R. Gangula, S. R. Duggenpudi, and R. Mamidi, “Detecting political bias in news articles using headline attention,” in _the ACL Workshop BlackboxNLP_ , 2019, pp. 77–84.
  * [33] B. D. Horne, W. Dron, S. Khedr, and S. Adali, “Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news,” in _ACM WWW_ , 2018, pp. 235–238.
  * [34] R. Baly, G. Karadzhov, A. Saleh, J. Glass, and P. Nakov, “Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media,” in _NAACL: Human Language Technologies_ , pp. 2109–2116.
  * [35] K. Alzhrani, “Political ideology detection of news articles using deep neural networks,” _Intelligent Automation and Soft Computing_ , vol. 33, pp. 483–500, 01 2022.
  * [36] S. Bordia and S. R. Bowman, “Identifying and reducing gender bias in word-level language models,” in _ACL_ , 2019, pp. 7–15.
  * [37] N. K. Dinsdale, M. Jenkinson, and A. I. Namburete, “Deep learning-based unlearning of dataset bias for mri harmonisation and confound removal,” _NeuroImage_ , vol. 228, p. 117689, 2021.
  * [38] R. Baly, G. Karadzhov, D. Alexandrov, J. R. Glass, and P. Nakov, “Predicting factuality of reporting and bias of news media sources,” in _EMNLP_ , 2018\.
  * [39] V. Patricia Aires, F. G. Nakamura, and E. F. Nakamura, “A link-based approach to detect media bias in news websites,” ser. ACM WWW, 2019, p. 742–745.
  * [40] W.-F. Chen, K. Al Khatib, B. Stein, and H. Wachsmuth, “Detecting media bias in news articles using Gaussian bias distributions,” in _ACL EMNLP_ , 2020, pp. 4290–4300.
  * [41] S. A. Farimani, M. V. Jahan, A. M. Fard, and G. Haffari, “Leveraging latent economic concepts and sentiments in the news for market prediction,” in _IEEE DSAA_ , 2021, pp. 1–10.
  * [42] “Ontheissues,” https://www.ontheissues.org, 2021, [Online; accessed 11-July-2021].
  * [43] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _NIPS_ , C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds., vol. 26, 2013.
  * [44] Y. Liu and M. Lapata, “Learning structured text representations,” _Transactions of the Association for Computational Linguistics_ , vol. 6, pp. 63–75, 2018.
  * [45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, _PyTorch: An Imperative Style, High-Performance Deep Learning Library_. Proceedings of the 33rd International Conference on NeurIPS, 2019.
  * [46] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _CoRR_ , vol. abs/1412.6980, 2014.
  * [47] E. Ferracane, G. Durrett, J. J. Li, and K. Erk, “Evaluating discourse in structured text representations,” in _Proceedings of ACL_ , 2019, pp. 646–653.
  * [48] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _ACL_ , J. Burstein, C. Doran, and T. Solorio, Eds., 2019, pp. 4171–4186.
  * [49] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [50] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _ArXiv_ , vol. abs/1910.01108, 2019.