Paper Title: SmileyNet -- Towards the Prediction of the Lottery by Reading Tea Leaves with AI.




.jpg,.pdf,.mps,.png

The Bulk Densities of Small Solar System Bodies as a Probe of Planetesimal Formation
    [
    August 5, 2024
====================================================================================



§ ABSTRACT 
We introduce SmileyNet, a novel neural network with psychic abilities. It is inspired by the fact that a positive mood can lead to improved cognitive capabilities including classification tasks. The network is hence presented in a first phase with smileys and an encouraging loss function is defined to bias it into a good mood. SmileyNet is then used to forecast the flipping of a coin based on an established method of Tasseology, namely by reading tea leaves. Training and testing in this second phase are done with a high-fidelity simulation based on real-world pixels sampled from a professional tea-reading cup. SmileyNet has an amazing accuracy of 72% to correctly predict the flip of a coin. Resnet-34, respectively YOLOv5 achieve only 49%, respectively 53%. It is then shown how multiple SmileyNets can be combined to win the lottery.     






§ INTRODUCTION

Modern AI is one of the greatest if not the greatest scientific breakthrough of all times. The publication of the structure of DNA by Watson and Crick in Nature <cit.> has only achieved slightly more than 19,000 citations by summer 2024 according to GoogleScholar. And it has taken 71 years. AI <cit.> has led to many important papers, which received many more citations within a much shorter time-frame; see, e.g., Tab. <ref> for a few examples based on citations according to GoogleScholar.



Given the relevance of AI for our modern world, it is surprising that there are areas of high economic relevance where it has not been applied, yet. We explore here - up to our knowledge for the very first time - the potential of AI for the psychic services industry, which is a multi-billion dollar business. This industry is estimated to have reached a revenue of well above $2 billion in the USA in 2023. And it experiences a steady further growth. This holds especially for psychic online services that managed an exponential growth in recent years according to market-reports.       

We suspect that AI may already be used in some real-world psychic services, especially in the online branches. But the methods and the results are not academically published up to our knowledge - likely due to the large economic potential they possess. While this is somewhat understandable, we believe in Open Science <cit.> and we share our insights due to the large benefits Psychic AI can provide to humanity.

A first medium-term goal is to win the lottery to further fund research in the novel field of Psychic AI. A well-established method of Tasseology (also known as Tasseomancy or Tasseography) <cit.> is used to this end, namely reading tea leaves <cit.>. In the spirit of divide and conquer, the simpler problem of predicting the outcome of flipping a coin is tackled first. We then show how this can be used to predict the numbers for winning the lottery jackpot. 

As an important basic research result, it is proven in this paper, up to our knowledge for the very first time, that a neural network can possess psychic powers. Concretely, it is shown that the novel SmileyNet can predict the flipping of a coin with a precision well above chance. 




     


§ SMILEYNET


SmileyNet is a novel bio-inspired network. It is motivated by the fact that cognitive processes can be positively influenced by a positive mood 
<cit.>.
This includes especially also classification tasks <cit.>. 



Furthermore, it is well-known that visual stimuli are well suited to elicit emotions <cit.>. This includes in particular a smiling face <cit.>. We hence first present the network with smileys as input to bias it into a good mood. More precisely, the Unicode emoticon "U+1F642" for the classical smiley, officially known as "Slightly Smiling Face" is used to this end (Fig.<ref>).

But stress is a negative factor for cognition <cit.>. We do not want to loose the positive effects of the good mood by putting the network under stress in the smiley training phase. Hence, it gets positive feedback <cit.> that it is doing fine, i.e., that it is doing nothing wrong.

More precisely, the training in the smileyfication phase is done as follows. A quadratic loss function L_smile() is used. To avoid putting the network under stress and to give a further positive endorsement, the expected output ŷ is set to be the actual output y whenever a smiley is presented to the net, i.e., ŷ = f_I(y) where f_I() is the identity function. So,


    L_smile()= 1/2 (y - ŷ)^2  with ŷ = f_I(y)


This method of smileyfication can be applied to any network N_old to turn it into a novel one N_new, which is expected to perform better than the previous N_old. 

Often, there may only be small improvements and it can happen that multiple trials are needed until some improvement is seen for the first time, which should then be immediately documented. Also, it can be useful to take additional factors into account. For example, the effects of smileyfication may only be seen on a sunny day as, e.g., the weather has an influence on the mood <cit.>.  



§ RELATED WORK


YOLOv5 <cit.> and Resnet34 <cit.> are well-known networks for image classification. They are used for comparison to SmileyNet in Sec. <ref>. There are also many other networks. But as YOLOv5 and Resnet32 are very popular, we concentrate here on those two. 





YOLOv5 is the fifth iteration of the revolutionary "You Only Look Once" object detection model. It is designed to deliver high-speed, high-accuracy results in real-time. Built on PyTorch, this powerful deep learning framework has garnered immense popularity for its versatility, ease of use, and high performance. Fig. <ref> shows how it works.


ResNet-34 is a convolutional neural network in PyTorch. It is has 34 layers. It addresses the problem of vanishing gradient. This is done with the identity shortcut connection that skips one or more layers. 
Fig. <ref> shows how it works.




§ HIGH-FIDELITY SIMULATION


The constellation of tea leaves and the predicted event are linked by cosmic energy. We use this fact to generate high-fidelity training data. This is not easy in the context of psychic powers and it may be one reason why AI has not been used - up to our knowledge - in this economically highly relevant business up to now.

The core idea of our approach for training data is to use the reverse direction of the link. The coin is flipped in a deterministic way, i.e., it is so to say turned by hand in rounds r from one side to the other. The related constellation of tea-leaves, i.e., the image I_r is then generated afterwards. 

In the software implementation, "heads" is presented with a "1" and "tails" with a "0" for the sake of convenience and wlog. Also wlog, we start with "tails" facing up as initialization of the flipping process. In the very first round r=1, the coin is turned and it hence shows "heads". The coin is then turned again and it now shows "tails" in round r=2, and so on. More formally, it holds for the outcome o_flip(r) in round r that


    o_flip(r) = 
    heads    if  (r+1)  mod  2 = 0
    tails    otherwise



The images I_r in our high-fidelity simulation are based on a real-world image (Fig. <ref>)[wikimedia.org; curid=17659832; user: MochaSwirl; Public Domain] of tea-leaves in a cup used by a fortune-telling professional. From this professional image, real-world pixels p_tea belonging to tea and real-world pixels p_cup belonging to the cup are extracted. This forms the base image I_0 with 100 × 100 pixels (Fig. <ref>). For the sake of simplicity, p_tea-pixels are shown in black and white is used for p_cup-pixels . 




The first image I_1 is then generated right after the coin was flipped for the first time. So, I_1 is linked to producing "head" according to the laws of Tasseology. In general, each image I_r is generated from I_r-1 by randomly selecting k_change coordinates [x,y]_j : x,y ∈{0,...,99} that are altered as follows 


    I_r[x][y] = 
    
        p_tea   if  I_r-1[x][y] = p_cup
    
        p_cup   if  I_r-1[x][y] = p_tea











Each I_r is generated right after doing the flip of the coin in round r, i.e., it is linked to the outcome o_flip(r) of flipping the coin in that round. According to our experience, small to medium values for k_change work best. The process is illustrated in Fig. <ref>. As mentioned, the p_cup, respectively p_tea pixel are shown as white, respectively black pixel in Fig. <ref> for the sake of simplicity. 



§ EXPERIMENTS AND RESULTS

  
The high-fidelity simulation presented in Sec. <ref> is used to generate 500 images. When it is needed by the network, the images are scaled to fit the requirements of PyTorch. The set is split into the first 300 images that are used for training and the following 200 images are used for validation. 


 


Tab. <ref> shows the results for YOLOv5 and Resnet-34. Resnet-34 has only 98 out of 200 cases correctly predicted. YOLOv5 does a bit better and it manages 106 correct predictions on the testing set.

Due to the better performance of YOLOv5 compared to Resnet-34, it is used for a smileyfication to generate SmileyNet version 1. As SmileyNet is expected to have an improved performance, it is trained with less data. Only every fifth image from the data-set is used. Note that this is still an alternating sequence of "heads" and "tails", i.e., it is exactly the same principle as before, but just 100 training images. The remaining 400 images are used for testing.

SmileyNet manages an amazing accuracy of 71.75%, i.e., 287 out of 400 cases are correctly predicted. This is well above the performance of YOLOv5 and Resnet-34 (Tab. <ref>). Most importantly, it is well above chance and we therefore prove - up to our knowledge - for the very first time that AI can possess psychic powers. Note that it can be shown that the improvement is statistically significant, i.e., it is a thorough and sound result.       

Tab. <ref> summarizes the performance gains compared to YOLOv5 and Resnet-34. SmileyNet has an accuracy of 71.75%, i.e., there is an increase by 35.38% from the 53% of YOLOv5. The performance gain is in the comparison with Resnet-34 with 46.43% even higher. SmileyNet requires only 100 training images while YOLOv5 and Resnet-34 had 300 images. So, there is a reduction by 200 images or 66% in comparison to the other two nets. 



§ SMILEYNET FOR LOTTERY FORECASTS



The forecasting of throwing a coin can already be a useful psychic power. But the forecast of lotteries has a very straightforward economical potential. Lotteries exist in most countries around the globe with high jackpots for the correctly predicted numbers. It is shown in the following with the example of the German lottery how SmileyNet can be used to this end.

In the German lottery, six numbers win_1,...,win_6 are drawn out of a pool with the numbers 1,...,49. The psychic power of SmileyNet can forecast the flip of a coin, i.e., 1 bit. We therefore simply combine multiple SmileyNets to encode the numbers 1,...,49. Then, one after the other win_i is predicted. 

More precisely, let b_j denote the forecasting output of SmileyNet N_j and ⟨ b_1,...,b_n ⟩_2 denotes the number x encoded by the bits b_1,...,b_n, i.e.,

    x = ⟨ b_1,...,b_n ⟩_2 = ∑_i=1^nb_i · 2^n-i


The i-th number win_i of a lottery draw is then predicted with ten SmileyNets N_j with outputs b_j as follows 

    win_i = ⟨ b_1,...,b_5 ⟩_2 + ⟨ b_6,...,b_9 ⟩_2 + ⟨ b_10⟩_2


As mentioned, the winning numbers win_1,...,win_6 are simply generated one after the other. It must of course be ensured that fresh images of tea-leaves from a just emptied cup are provided in every round. Nonetheless, SmileyNet does not have perfect psychic power. It may hence happen in rare occasions that a number win_j+k has been predicted before as win_j. But this is not a problem. Then, the current win_j+k is simply discarded and an other round is added to determine a new win_j+k.

Given the predictive power of SmileyNet to correctly forecast a bit with p_ N=0.7175, the exact likelihood p_win of forecasting the German lottery with the above method can be computed as

    p_win   = p_ N^5 ·p_ N^4 · p_ N
       = 0.03615922 
       = 3.615922 · 10^-2
        
This likelihood may look small at first glance, but it is by six orders of magnitude better than just chance with

    p_chance   = 49  6^-1
       = (49!/6!(49-6)!)^-1
       = 1/13,983,816
       = 7.15112 · 10^-8
       ≪ p_win
        

This is an important result that demonstrates the huge potential of the novel field of Psychic AI. Note that it is a thorough result as it can be shown that it is statistically significant.

But before we apply these results in practice, we still want to improve the performance of our approach. This holds first and foremost for the likelihood p_win of predicting the correct numbers for winning the jackpot:

	
  * One option is simply more training data and more computation power for a larger network to turn SmileyNet into a foundation model for psychic AI. 

  * Furthermore, we expect a transformer to be more accurate given that the drawing of lottery numbers is a weekly event in Germany. This is left for future work.

Also, a reduction of the required number of SmileyNets would be nice before applying it in practice as it is a bit inconvenient to drink so much tea. 



§ CONCLUSIONS


SmileyNet version 1 was introduced. It is based on smileyfication, i.e., the fact that a positive mood can have a positive influence on cognitive functions including classification. It was shown how SmileyNet can be used to predict the lottery by reading tea-leaves. First, the flipping of a coin was predicted with a very good accuracy, namely 72%. State of the art networks like YOLOv5 and Resnet-34 perform much worse. Second, it was shown how multiple SmileyNets can be combined for lottery forecasts. In addition to the potential practical implications, it is a major breakthrough to show that AI can have psychic powers.  



§ ACKNOWLEDGMENTS

We thank our anonymous reviewers for their feedback. Special thanks go to reviewer 2, who pointed out that the drawing of lottery numbers tends to be a weekly or at least regular event, and that the use of a transformer should be beneficial. 




§ EPILOG

This is a satirical accumulation of misconceptions, mistakes, and flawed reasoning I have encountered in recent times as a reviewer and sometimes even as a reader of published papers. I hope it is entertaining and useful in the context of the education of BSc, MSc, and PhD students in Machine Learning, Artificial Intelligence, and Cognitive Science. 

For an easy start, here a few questions to reflect upon. First and foremost, there is of course the following puzzle: 

	
  * Where does the "magic" hide that is presented here? How can SmileyNet "predict" the flip of a coin better than chance? 

As a small hint: there is a single conceptual mistake that explains it. 

There are also many other bugs, respectively misconceptions and flawed reasoning in this paper. Here some further food for thought, to hopefully teach a good lesson by setting a bad example:   

	
  * Can AI beat natural laws? What are the pros and cons of a detailed pre-analysis of the possible information content of the data?
	
  * How much academic novelty is in SmileyNet? How much does it differ from the net it is build upon? 
	
  * How do we prevent smileyfication in our own work? How can we properly show that an "extension" or "modification" of an existing approach leads to a substantial improvement and study the related trade-offs?  
	
  * What is a proper assessment of the state of the art? How do we find, understand, and discuss related work?
	
  * What is a proper statistical performance analysis? What is statistical significance and how should it be interpreted?
	
  * When and how do I expect my own work to be of relevance? Why do I publish it? 
	
  * What are Artificial Intelligence and Machine Learning? Which sub-areas within these disciplines exist? What are commonly used methodologies within the sub-areas? 
		
  * What is the role of training data in supervised machine learning? What are the pros and cons of simulation?
	
  * What is Open Science? What are its principles? What does it imply for the quality of the underlying work? What are the pros and cons if others, respectively oneself adheres to it? 
	
  * What is a bio-inspired design? How can insights from Cognitive Science be used and which pitfalls exist?
	
  * What are the pros and cons of bibliometrics? How can they help us to identify relevant work? And where are their constraints and limitations?

This list is a bit concentrated on the "big questions". It is of course also possible to just concentrate on the (many) methodological mistakes in the SmileyNet paper. So, the above list is neither meant to be exhaustive nor to be without dispensable aspects depending on the context. It is just a possible starting point for using the SmileyNet in the class-room, or to just meditate about it on your own, or... 

Enjoy and have fun...

10

url@samestyle








WatsonCrick-DNA-Nature1953
J. D. Watson and F. H. C. Crick, “Molecular structure of nucleic acids: A
  structure for deoxyribose nucleic acid,” Nature, vol. 171, no. 4356,
  pp. 737–738, 1953. [Online]. Available:
  <https://doi.org/10.1038/171737a0>


DeepLearning-Hinton-Nature15
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
  no. 7553, pp. 436–444, 2015. [Online]. Available:
  <https://doi.org/10.1038/nature14539>


DeepLearning-Book-MITpress16
I. Goodfellow, Y. Bengio, and A. Courville, Deep learning.1em
  plus 0.5em minus 0.4emMIT press, 2016.

DeepLearning-ResNet-CVPR16
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
  recognition,” in IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR), 2016, Conference Proceedings, pp. 770–778.

Adam-StochasticOptimization-arXiv2014
J. B. Diederik P Kingma, “Adam: A method for stochastic optimization,”
  arXiv preprint, vol. arXiv:1412.6980, 2014.

DeepLearning-AlexNet-NIPS12
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with
  deep convolutional neural networks,” pp. 1106–1114, 2012. [Online].
  Available:
  <http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks>


Breimann-RandomForests-ML01
L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
  5–32, 2001. [Online]. Available:
  <https://doi.org/10.1023/A:1010933404324>


DeepCNN-LargeScale-ImageRecognition-2014
K. Simonyan and A. Zisserman, “Very deep convolutional networks for
  large-scale image recognition,” arXiv preprint, vol. arXiv:1409.1556,
  2014.

GAN-NIPS2014
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
  A. Courville, and Y. Bengio, “Generative adversarial nets,” 2014. [Online].
  Available:
  <https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf>


FasterR-CNN-NIPS2015
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
  object detection with region proposal networks,” 2015. [Online]. Available:
  <https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf>


ImageNet-CVPR09
J. Deng, W. Dong, R. Socher, L. J. Li, L. Kai, and F.-F. Li, “Imagenet: A
  large-scale hierarchical image database,” in IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR), 2009, Conference
  Proceedings, pp. 248–255.

DeepLearning-GoogLeNet-CVPR15
C. Szegedy, L. Wei, J. Yangqing, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
  V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
  2015, Conference Proceedings, pp. 1–9.

UNESCO-RecommendationOpenScience2021
UNESCO, “Unesco recommendation on open science,” 2021. [Online]. Available:
  <https://doi.org/10.54677/MNMH8546>


Tasseology-2008
R. E. Guiley, “Tasseomancy,” in The Encyclopedia of Witches, Witchcraft
  and Wicca.1em plus 0.5em minus 0.4emCheckmark Books, 2008.

Tasseology-TeaLeaves-2016
K. Henriott-Jauw, Tea and Tasseomancy: A Guide to Reading Tea
  Leaves.1em plus 0.5em minus 0.4emCreateSpace Independent
  Publishing Platform, 2016.

Tasseology-TeaLeaves-2014
A. O'Reilly, Tea with Annie, a divinitive guide to Tasseomancy and its
  artistry.1em plus 0.5em minus 0.4emMelbourne, Australia:
  Whiteslaw Press, 2014.

PositiveMood-CreativeProblemSolving-Sage2015
J. Politis and J. C. Houtz, “Effects of positive mood on generative and
  evaluative thinking in creative problem solving,” Sage Open, vol. 5,
  no. 2, p. 2158244015592679, 2015. [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/2158244015592679>


PositiveMood-CognitiveControl-2011
E. A. Martin and J. G. Kerns, “The influence of positive mood on different
  aspects of cognitive control,” Cognition and Emotion, vol. 25, no. 2,
  pp. 265–279, 2011. [Online]. Available:
  <https://doi.org/10.1080/02699931.2010.491652>


PositiveMood-Cognition-Classification-2010
R. T. Nadler, R. Rabi, and J. P. Minda, “Better mood and better
  performance:learning rule-described categories is enhanced by positive
  mood,” Psychological Science, vol. 21, no. 12, pp. 1770–1776, 2010.
  [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/0956797610387441>


ImagesMoodSurvey2016
M. K. Uhrig, N. Trautmann, U. Baumgärtner, R.-D. Treede, F. Henrich,
  W. Hiller, and S. Marschall, “Emotion elicitation: A comparison of pictures
  and films,” Frontiers in Psychology, vol. 7, 2016. [Online].
  Available:
  <https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.00180>


ImagesMoodSurvey2008
J. K. Olofsson, S. Nordin, H. Sequeira, and J. Polich, “Affective picture
  processing: An integrative review of erp findings,” Biological
  Psychology, vol. 77, no. 3, pp. 247–265, 2008. [Online]. Available:
  <https://www.sciencedirect.com/science/article/pii/S0301051107001913>


MoodSmilingFace2019
J. Nikitin and A. M. Freund, “The motivational power of the happy face,”
  Brain Sciences, vol. 9, no. 1, p. 6, 2019. [Online]. Available:
  <https://www.mdpi.com/2076-3425/9/1/6>


CognitionStressNegative2024
G. Picciotto and R. A. Fabio, “Does stress induction affect cognitive
  performance or avoidance of cognitive effort?” Stress and Health,
  vol. 40, no. 1, p. e3280, 2024. [Online]. Available:
  <https://onlinelibrary.wiley.com/doi/abs/10.1002/smi.3280>


CognitionStressNegative2023
E. J. Kim and J. J. Kim, “Neurocognitive effects of stress: a metaparadigm
  perspective,” Molecular Psychiatry, vol. 28, no. 7, pp. 2750–2763,
  2023. [Online]. Available: <https://doi.org/10.1038/s41380-023-01986-4>


StressCognitionFeedback2020
M. Paul, C. Bellebaum, M. Ghio, B. Suchan, and O. T. Wolf, “Stress effects on
  learning and feedback-related neural activity depend on feedback delay,”
  Psychophysiology, vol. 57, no. 2, p. e13471, 2020. [Online].
  Available: <https://onlinelibrary.wiley.com/doi/abs/10.1111/psyp.13471>


MoodWeather2008
J. J. A. Denissen, L. Butalid, L. Penke, and M. A. G. van Aken, “The effects
  of weather on daily mood: A multilevel approach,” Emotion, vol. 8,
  no. 5, pp. 662–667, 2008.

MoodWeather1984
E. Howarth and M. S. Hoffman, “A multidimensional approach to the relationship
  between mood and weather,” British Journal of Psychology, vol. 75,
  no. 1, pp. 15–23, 1984. [Online]. Available:
  <https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1111/j.2044-8295.1984.tb02785.x>


YOLOv5-2021
Ultralytics, “YOLOv5: A state-of-the-art real-time object detection
  system,” 2021. [Online]. Available: <https://docs.ultralytics.com>


YOLOv5-architecture-2021
——, “Ultralytics YOLOv5 architecture,” 2021. [Online]. Available:
  <https://docs.ultralytics.com/yolov5/tutorials/architecture_description/>






 Paper Title: Zero Shot Health Trajectory Prediction Using Transformer.


Data-driven Modeling for Grid Edge IBRs: A Digital Twin Perspective of User-Defined Models 





    Kaveri Mahapatra*, Bhaskar Mitra, and Soumya Kundu
Pacific Northwest National Laboratory, United States 



{kaveri.mahapatra, bhaskar.mitra, soumya.kundu}@pnnl.gov































    
=======================================================================================================================================================================================================



§ ABSTRACT 
Integrating modern machine learning and clinical decision-making has great promise for mitigating healthcare's increasing cost and complexity. We introduce the Enhanced Transformer for Health Outcome Simulation (ETHOS), a novel application of the transformer deep-learning architecture for analyzing high-dimensional, heterogeneous, and episodic health data. ETHOS is trained using  Patient Health Timelines (PHTs)—detailed, tokenized records of health events—to predict future health trajectories, leveraging a zero-shot learning approach. ETHOS represents a significant advancement in foundation model development for healthcare analytics, eliminating the need for labeled data and model fine-tuning. Its ability to simulate various treatment pathways and consider patient-specific factors positions ETHOS as a tool for care optimization and addressing biases in healthcare delivery. Future developments will expand ETHOS’ capabilities to incorporate a wider range of data types and data sources. Our work demonstrates a pathway toward accelerated AI development and deployment in healthcare.








§ INTRODUCTION

Healthcare in the U.S. is the world’s most expensive, and the quality and safety of care do not compare well to other developed countries <cit.>. While electronic healthcare records are now ubiquitous in the U.S., and decision-support technologies are widely implemented, most are rule-based, and their effectiveness so far has been limited <cit.>. Artificial intelligence has emerged as a technique with great potential for improving care, but most organizations are not using it to any major degree. Two major limiting factors have been (1)  the lack of large, labeled datasets, which are expensive and time-consuming to develop; and (2) limited system capacity to deliver recommendations to the appropriate clinician at the optimal time.  In this manuscript, we describe a novel method called the Enhanced Transformer for Health Outcome Simulation (ETHOS), which we believe can help address many of the limitations that have prevented widespread AI adoption.

ETHOS is a novel application of the transformer deep-learning architecture, originally conceptualized for natural language processing <cit.>. This architecture, a cornerstone in large language model (LLM) development, is repurposed in ETHOS to analyze health-related data, moving beyond the textual focus of traditional LLMs. ETHOS is designed to process Patient Health Timelines (PHTs)—detailed tokenized chronological records of health-related events—to predict future health timelines. In PHTs, a token serves as the fundamental unit of information, encapsulating diverse data types such as patient admissions, administered medications, or time intervals. We elaborate on this pivotal aspect of our methodology in the Methods section. Our model takes the patient's health history, as represented by PHT, and subsequently forecasts future PHT (fPHT) on a token-by-token basis (refer to Figure <ref>).



ETHOS’s generative capabilities are gained in unsupervised learning. Once trained, ETHOS can forecast future health events without requiring task-specific training. This is done through a zero-shot learning approach, making ETHOS a versatile foundation model for numerous healthcare applications. With appropriate modifications, ETHOS can be adapted to a broad range of data types, including but not limited to medical images, clinical and discharge notes, monitoring data, data from wearables, or omics data.

In this research, we leverage the recently released MIMIC-IV v.2.2 dataset <cit.>, a rich open-source repository accompanied by our code, allowing others to replicate our findings. MIMIC-IV is expansive, chronicling more than 400,000 hospitalizations in more than 200,000 patients. Although relatively large, we anticipate that the performance of our system will further improve as we expand the dataset with additional patient histories and data types. 

Importantly, we utilize the MIMIC-IV dataset in its original noisy form without any data modifications, cleaning, or targeted imputation for missing entries. The information is retained in the face of large data inconsistencies, such as discharge dates noted before admission dates. We operated under the assumption that, within large enough datasets and appropriate tokenization and training methods, ETHOS would be robust enough to handle the noisy input and automatically manage the noise/anomalies in the input data. The resilience of ETHOS to data inaccuracies and missing information has important implications for the efficiency of downstream model development. Healthcare data inevitably contains errors, some of which may not be immediately apparent or easily rectifiable. Attempts to clean large datasets can be impractical and may inadvertently introduce biases and errors. Our approach highlights the vital need for algorithms adept at managing these challenges, a prerequisite for the large-scale development of reliable and robust healthcare AI applications. 

Our research showcases the zero-shot learning capabilities of ETHOS in predicting inpatient and ICU mortality, estimating ICU length of stay (LOS), and determining readmission probabilities. Additionally, we illustrate the model's versatility by performing a regression task to estimate the first-day Sequential Organ Failure Assessment (SOFA) score <cit.> at the time of ICU admission using information before admission (see example in Figure <ref>d). The SOFA score is a critical tool for monitoring a patient's condition in the ICU, evaluating organ function or failure across six systems—respiratory, cardiovascular, hepatic, coagulation, renal, and neurological—with each system scored from 0 to 4, culminating in a total possible minimum score of 0 and maximum score of 24. Furthermore, we predict Diagnostic-Related Group (DRG) classifications, encompassing over 771 categories, at the time of hospital discharge. The DRG system categorizes hospital cases into standardized case complexity-based Medicare and Medicaid payment groups, encouraging efficient patient care without compromising quality. The diversity of tasks ETHOS can perform, from mortality predictions and LOS estimation to SOFA scoring and DRG classification, highlights its broad applicability and zero-shot learning efficiency. 

ETHOS is a foundation model <cit.>, introducing a novel approach in the landscape of data analysis within the healthcare domain. The other foundational models developed recently have fallen into two broad categories. The first of these categories encompasses Clinical Language Models (CLaMs), a specialized subset of large language models (LLMs) <cit.> tailored for processing clinical and biomedical text data. These models are typically trained on extensive datasets containing clinical notes, biomedical literature, and other healthcare-related text sources. CLaMs are proficient in various clinical tasks such as extracting drug names, summarizing medical dialogues, predicting clinical outcomes, and responding to patient queries <cit.>. The second category comprises Foundation Models for Electronic Medical Records (FEMRs), representing another class of clinical foundation models tailored specifically for EMR data analysis. FEMRs undergo training on the extensive medical histories of patients, covering both structured data (such as demographics and lab results) and unstructured data (including progress notes and radiology reports). Unlike CLaMs, FEMRs are not designed to generate clinical text. Instead, they produce machine-understandable representations of patient data, facilitating tasks such as patient phenotyping and outcome prediction <cit.>. Similarly, data that chronicles human lives, akin to EMR, can also be modeled effectively in this manner. 

The primary distinction between ETHOS and previously published methods lies in our approach, which eliminates the need for fine-tuning or labeled data to produce accurate inferences or predictions. We demonstrate inference across a wide array of tasks without task-specific training. Moreover, the ability of ETHOS to forecast future PHTs opens the door to a wide array of bespoke and innovative applications, facilitating its use in unique scenarios in healthcare, some of them explored in the discussion section. Unlike many studies, which often apply specific criteria for selecting data for training and testing, our methodology imposes no such limitations. This feature is crucial for considering the scalability of the ETHOS approach to data sets comprising millions or even hundreds of millions of patients.



§ RESULTS




 §.§ Tokenization of MIMIC data and training of ETHOS


Figure <ref>a summarizes some statistics of the tokenization process, including the number of tokens generated and other details. Figure <ref>b presents visualizations of the 768-dimensional embeddings reduced to a 2D plane using Principal Component Analysis (PCA) for quantile tokens, which encode all quantitative values in the data. The tokens are arranged from Q1 (the lowest quantile) to Q10 (the highest quantile). This suggests that the transformer model has learned a sequential relationship between the tokens that mirrors their natural order, ascertaining this order from the data during the training process. The proximity between points could reflect the model's differentiation among the quantiles. We observe that the gaps between Q4, Q5, and Q6 are narrower than those between Q9 and Q10. This may suggest that the model deems the variance between population-average values to be less substantial than that of extremely high values. For example, the difference in clinical significance between a blood pressure reading of 110 mmHg (Q5) and one of 130 mmHg (Q6) is less pronounced than the difference between 140 mmHg (Q9) and 160 mmHg (Q10), which could account for the greater disparity in the embedding vectors of high quantiles.

The embeddings for time-interval tokens, representing the approximate durations between different tokenized events in PHT, are illustrated in Figure <ref>b. These embeddings display a pattern analogous to that observed for Q tokens, where ETHOS systematically arranged them according to the actual time values they represent. Remarkably, the model perceives the two shortest (5m-15m, 15m-1h), and two longest (3m-6m, 6m) intervals as relatively similar. 





 §.§ ETHOS inferences


In our study, we conducted zero-shot inferences for a diverse array of classification tasks, including readmission to the ICU, inpatient mortality, ICU mortality, combined inpatient and ICU mortality in patients with sepsis, readmission to the ICU for patients with intracerebral hemorrhage, assignment of DRG class assessed at inpatient discharge. We also demonstrate regression of first-day SOFA score at the time of ICU admission and regression of the length of stay in ICU in days assessed upon admission. The results corresponding to these tasks are summarized in Figure <ref>. We also provide precision-recall curves of the corresponding results in Figure <ref>.



To situate our results within the broader scientific discourse, we conducted a literature review, concentrating on contemporary studies that utilized the MIMIC-III and MIMIC-IV datasets for similar tasks and reported their outcomes. A notable observation from our review is that many of these studies either lacked publicly available source code or implemented specific exclusion criteria for their data selection. Such practices pose challenges for directly comparing their results with our approach. Nonetheless, we posit that the numerical outcomes reported in these works provide a valuable benchmark for assessing the performance of ETHOS. Furthermore, we conducted a direct comparative analysis of ETHOS against specialized algorithms developed in-house, with these findings detailed in the supplementary materials.

We conducted an analysis focusing on risk estimation for inpatient and ICU mortality, calculated at the respective points of patient admission to the hospital and ICU. The test set comprised 43,309 hospital admissions with a 2.0% mortality and 7,483 ICU admissions with a 7.0% mortality. The ETHOS model demonstrated robust performance, achieving an AUC of 0.921 (95% CI: 0.908-0.931) for hospital mortality and 0.927 (95% CI: 0.914-0.938) for ICU mortality. Comparatively, in the ICU mortality risk prediction domain, the highest performance identified in our literature review was an AUC of 0.918 (95% CI: 0.915-0.922) reported by <cit.> using the XGBoost model. On the lower end, <cit.> reported an AUC of 0.642 ± 0.101. Within a specific subgroup of the test set of 3,324 patients with sepsis with 10.8% mortality prevalence, ETHOS's prediction of ICU mortality exhibited an AUC of 0.889 (95% CI: 0.870-0.906), which is a better performance than obtained in a study by <cit.>, which estimated ICU mortality in adult sepsis patients using SOFA and additional features, achieving an AUC of 0.762 ± 0.006. We also estimated performance for a task of ICU mortality estimation for patients remaining in ICU for at least 24 hours in which we obtained an AUC of 0.928 (95% CI: 0.916-0.939). 

Furthermore, ETHOS estimated the length of stay (LoS) in the ICU with a mean absolute error (MAE) of 2.262 days (95% CI: 2.161-2.355 days). These results paralleled those of18, who reported an MAE of 2.42 ± 0.10 days. ICU LoS prediction and mortality risk, underscoring the competitive zero-shot performance of ETHOS across multiple key healthcare metrics.

For the ICU readmission task, ETHOS’ AUC of 0.807 (95% CI: 0.786-0.827) is slightly smaller than the AUC of 0.82 obtained using knowledge graph embeddings <cit.> and is higher than the AUC of 0.791 (95% CI, 0.782–0.800) using LSTMs based on MIMIC-III data <cit.>. Additionally, we applied our method to a task characterized by a relatively low prevalence, specifically focusing on only 174 cases of patients with hemorrhage admitted to the ICU present within our test set. The prediction of readmission by ETHOS yielded an AUC of 0.667 (95% CI: 0.402-0.839), comparable to the AUC of 0.736 (95% CI: 0.668-0.801) achieved by previous studies <cit.> using LightGBM. For hospital readmission, ETHOS achieved an AUC of 0.749 (95% CI: 0.743-0.755), lower than the AUC of  0.791 [0.766-0.816] obtained by <cit.>. It's important to recognize that although MIMIC offers a wealth of data on acute care, it might not encompass all the subtleties necessary for readmission research, including comprehensive post-discharge outcomes or data on readmissions to various hospitals. Consequently, the accuracy of results for tasks related to readmission may be limited, regardless of the method employed.



We explored the task of predicting the first-day SOFA score at the time of admission (Figure <ref>). Given that the SOFA score is a critical indicator of survival, particularly in sepsis <cit.>, this prediction can serve as a valuable indirect prognostic marker of ICU patient health status. We achieved a SOFA score estimation with an MAE of 1.502 (95% CI: 1.475-1.534). To our knowledge, no prior literature predicts first-day SOFA at the time of admission.

For the DRG assignment, we observed a top-1 (out of 771 classes) accuracy rate of 84.8% (95% CI: 84.4%-85.2%) in 28,932 hospitalizations using our methodology, a significant improvement over the 52% reported by <cit.>, who explored DRG estimation using LLMs from discharge notes. This marked enhancement in performance can be attributed to the comprehensive nature of ETHOS, which incorporates a wide array of clinical events leading up to discharge within the PHT. In contrast, the approach taken by <cit.> relies solely on discharge notes, which may not encompass the breadth of information captured by PHT, thus potentially explaining the disparity in accuracy rates. 

We want to reiterate an important point: all comparisons presented in this section are made between ETHOS, trained indiscriminately on the entire test population and task-specific algorithms developed using much smaller MIMIC data subsets obtained after data curation. In addition to the results in this section, in supplementary materials, we benchmark the performance of ETHOS against XGBoost <cit.>, recurrent neural networks, and logistic regression. 



§ DISCUSSION


This work introduces an innovative approach to developing a Foundation Model for medical data derived from EMRs, designed to execute zero-shot inferences across a diverse range of tasks. Our model generates interpretable, causally forecasted future patient health timelines. By "causal," we mean that predictions are made solely based on information that occurred in the past. We applied and evaluated this model using the MIMIC-IV EMR datasets, comparing its performance with the results of methods published in the literature for the same tasks. Our objective, however, was not merely to surpass the performance of these specialized SOTA implementations. Instead, we aimed to demonstrate that ETHOS, a single foundation model trained just once with zero-shot derived inference, can achieve performance levels comparable to that of multiple models optimized for various tasks. This underscores the potential of ETHOS to streamline the application of AI in healthcare by leveraging a single unified model development architecture and set of methods for multiple prediction tasks, thereby greatly enhancing medical data model development efficiency and scalability.

The application of patient timelines for generating insights has been established in existing research <cit.>, as has the implementation of foundation models <cit.>. Our methodology sets itself apart by integrating a zero-shot capability, obviating the need for additional training beyond the initial model. The design of ETHOS accommodates various approaches to inference, including few-shot predictions, although this necessitates fine-tuning for specific downstream tasks. Notably, the zero-shot prediction methodology introduces capabilities absent in few-shot prediction. To forecast future outcomes, ETHOS generates multiple health timelines representing possible future scenarios. This functionality exploits the model's capacity to explore and evaluate potential future events, thereby potentially estimating uncertainties. Future work will undoubtedly concentrate on refining this aspect of ETHOS. Moreover, ETHOS is specifically engineered to produce causal predictions in the form of future timelines, ensuring they are inherently comprehensible to human users. This is achieved through a novel tokenization process for medical data, a distinctive feature of our work.

The generation of multiple scenarios using the zero-shot approach places significant demands on time and computational resources. We estimated the inference time based on the average duration required to generate 1,000 tokens. On a single Nvidia A100 GPU, this process took approximately 15 seconds. Given that computations are executed in batches and are thus highly parallelizable, we anticipate that in a potential production environment, response times could vary between 1 to 30 seconds. This variation is contingent upon the complexity of the downstream task at hand.

Another highly distinctive capability of ETHOS is the potential to generate individualized care-integrated PHT-based projected healthcare expenditures. This capability is exemplified through the prediction of Diagnosis-Related Group (DRG) codes but is not limited to this application. Specifically, ETHOS can model future PHTs at critical decision-making junctures in patient care. For instance, ETHOS can model outcomes for administering either drug A or B, considering the patient's unique conditions (such as sex, age, race, gender, income, etc.) to determine which path might yield better clinical and cost outcomes. In this regard, ETHOS has the potential to revolutionize medical decision-analytic modeling science by incorporating a level of personalization previously unavailable in conventional decision-analytic models. This has the potential to enhance clinical decision-making and incorporate individualized real-time quantitatively robust value-based care policies into clinical care.  This is a potentially transformative change, radically  unlike current evidence-based medicine practices, which  rely on high-quality data obtained from and averaged across patient populations <cit.>.

In designing ETHOS, we have considered explainability, fairness, and transparency. These are vital aspects of our ongoing research. In future work, we plan to implement and test advanced visualization attention layers of the transformer <cit.> to gain insights into the model's reasoning process. Additionally, a dedicated interface for decision-making is envisaged further to enhance the usability of ETHOS in clinical settings.

Envisioning the development of a robust AI method that offers fully personalized advice on a wide range of medical questions necessitates learning from an extensive dataset of patients. Such a model must assimilate as much data as possible and be adaptable to a vast array of medical tasks. ETHOS represents a significant stride in this direction. Built on a transformer architecture, it is inherently scalable and, as a zero-shot learner, is versatile enough to address numerous key medical prediction tasks without task-specific training. Currently, ETHOS does not incorporate various types of critical information, including clinical and discharge notes, medical imaging and pathology images, genetic data, socioeconomic factors, lifestyle considerations, and monitoring signals. Nonetheless, the conceptual framework for incorporating these diverse data types is relatively straightforward. This can be done by leveraging the encoder and cross-attention mechanisms inherent in the transformer architecture; we anticipate the potential for integrating a nearly limitless amount of information during training. This expansion of ETHOS's capabilities forms the cornerstone of our future work, promising to enhance its applicability and efficacy in personalized medical advice and diagnostics.

We aim to modify further and train ETHOS to apply it across diverse data sources. This capability is currently hindered by variations in data collection methodologies, disparities in data quality, and the presence or absence of certain data types across different sources. Additionally, non-overlapping populations present significant challenges, rendering ETHOS not yet generalizable. To mitigate some of these compatibility issues, we propose the development of a universal tokenization format <cit.>. While this approach may resolve certain discrepancies, it does not address all underlying compatibility concerns. The ultimate solution, we believe, lies in a system capable of transforming tokenized data from one healthcare system to another, akin to text translation between languages. Specifically, for ETHOS, this would mean converting the patient journey, as encapsulated by the PHT, from one system's format to another. This conversion would not only facilitate a consistent and unified representation of patient histories across different systems but also offer insights into the operational nuances of these systems. Pursuing such a translation strategy represents a vital direction for our future research endeavors, alongside evaluating the methodologies introduced in this paper through analysis of prospectively collected data.

ETHOS and LLMs such as GPT-4o, Claude 3 Opus, and Gemini 1.5 Ultra, although built upon similar AI principles, serve different purposes and exhibit distinct capabilities. ETHOS is specifically designed to predict fPHTs through explicit modeling of quantitative values and temporal sequences. This approach allows ETHOS to leverage structured patient data to generate predictions. In contrast, LLMs are general-purpose models optimized for tasks involving knowledge integration, reasoning, and interactive conversation. They do not explicitly model quantitative values and time sequences, which are important for accurate clinical decision support. Studies, such as those by <cit.> and <cit.>, highlight the limitations of LLMs in handling temporal information and decision support tasks, emphasizing the potential need for specialized models like ETHOS. There is a potential of ETHOS to be used in conjunction with LLMs through retrieval-augmented generation (RAG) mechanisms, offering a promising direction for future AI applications in healthcare. In supplementary material,  we present a comparison in predictive performance of ETHOS and LLM (GPT-4o). Furthermore, while LLMs excel in processing vast amounts of unstructured text, their computational performance in generating detailed and contextually accurate patient predictions remains suboptimal compared to ETHOS because of efficient representation of information in ETHOS tokenized PHTs.

This work has limitations.  We utilized the MIMIC dataset, which may be cleaner than many routine clinical datasets. Performance and usability should be tested prospectively in diverse datasets and in real-time. The transformer model in the current version of ETHOS is relatively simple and uses only 2048 PHT tokens for predictions. When token density per time is large, this may not contain sufficient information for optimal performance. Mitigation of the limitation is expected with additional computational infrastructure.

In conclusion, ETHOS presents a promising approach to deriving insights from massive clinical datasets without labor-intensive labeling or distinct model creation for each prediction task. This approach has the potential to significantly lower the costs and complexities associated with AI model development, thereby accelerating the development and implementation of healthcare AI.



§ METHODS




 §.§ Data


In this study, the Medical Information Mart for Intensive Care (MIMIC-IV) database served as a data source, providing a rich and comprehensive collection of de-identified health-related information4. Managed collaboratively by the Massachusetts Institute of Technology (MIT), Beth Israel Deaconess Medical Center (BIDMC), and Philips Healthcare, MIMIC-IV encompasses detailed records for more than 200,000 patients who were admitted to hospital and critical care units at BIDMC in Boston, Massachusetts, between 2008 and 2019. The following tables from the MIMIC-IV were used: 1) Patients, which contains static information about the patients, such as gender, date of birth, and date of death; 2) Admissions, which holds information about patient admissions to the hospital, including admission and discharge times, as well as information related to the hospital stay; 3) Icustays, which is specifically related to intensive care unit (ICU) stays, including the timings and type of ICU; 4) Labevents, which contains laboratory test results for patients. We used the 200 most frequent tests covering 95% of tests completed; 5) Prescriptions, which holds information on medications prescribed to patients during their stay, with each drug converted to ATC code [<https://www.whocc.no>]. We converted GSN codes in MIMIC-IV to ATC codes using conversion tables <cit.>; 6) Procedures which contains information about procedures performed on patients, coded using ICD10-PCS codes; 7) Diagnoses which contains diagnostic information, typically coded using ICD10-CM codes. We converted ICD9 to ICD10-CM if needed using conversion table [<https://www.cms.gov/medicare/coding-billing/icd-10-codes>]; 8) Emar, which holds information related to the documentation and administration of medications to patients; 9) Omr with information about measurements taken from a patient, such as blood pressure or BMI; 10) Services with information about the clinical service under which a patient is managed during their hospital stay; 11) drgcodes DRG codes which are a classification system used in the healthcare industry to categorize hospital cases into groups that are expected to have similar hospital resource use;  12) SOFA, taken from the derived tables in MIMIC. The remaining tables were not used in the current ETHOS implementation as they will require additional processing. For example, clinical notes require natural language processing to be converted to meaningful tokenized information.





 §.§ Patient health timelines (PHTs), tokenization


The core concept behind ETHOS is the Patient Health Timeline (PHT), as depicted in Figure <ref>. The fundamental component of the PHT is the token, which represents a distinct unit of information occurring within the patient's health timeline. To construct the PHT, we gathered all pertinent data from tables 1 to 12 of the MIMIC-IV database, as detailed in the Data section. We arranged this data chronologically based on timestamps, as shown in Figure <ref>, into a chronological sequence of health-related events for each patient. These events were timestamped with a floating-point number in 64-bit precision to denote the patient's age at the time of occurrence of the event. Subsequently, events from the MIMIC-IV tables were converted into tokens. Each event was represented by 1 to 7 tokens to encapsulate information about the event, as illustrated in Figure <ref>a. We crafted this encoding process to ensure each token conveys specific, meaningful information, with examples in Figure <ref>c-f. A comprehensive list of token encodings within the PHT is available in the supplementary material. The final step of tokenization involved the insertion of time-interval tokens to represent the intervals between events, depicted in Figure <ref>c. We employed 13 different time-interval tokens to represent the intervals. No interval token was inserted if the duration between tokens was less than 5 minutes. Typically, a single time-interval token was placed between other types of tokens unless the interval exceeded one year. In such cases, multiple 6-month tokens were used to approximate the actual interval. For example, an interval of 1.4 years was represented by three 6-month tokens, while four 6-month tokens represented 1.76 years. One interval-tokens were inserted the exact time of events was dropped from PHTs.

The patient's age and the commencement date of the PHT were represented using the same token set. We used 20 distinct tokens to denote age intervals such as 0-5 years, 5-10 years, and so forth. For instance, to encode information about a 46-year-old patient with PHT beginning in 1982, we inserted a "45_50 years" token at the 4th position in the PHT. To signify the year 1982, we used a "15_20 years" token at the 5th position of the PHT, considering 1970 as the baseline year. We emphasize that age and the commencement of the PHT are encoded in five-year intervals, given that health status typically does not undergo rapid changes with age, making finer granularity unnecessary. However, we plan to scrutinize these assumptions in subsequent research. The token denoting the commencement of the PHT delineates the temporal context of the medical data—identifying whether it corresponds to earlier medical practices (e.g., 1990s), contemporary practices, or periods in between. Using tokens with a precision of five years is done under the premise that technological and methodological progress within the medical field does not advance at a pace that justifies the necessity for time intervals more granular than five-year spans. Pertinent to the MIMIC dataset, the obfuscation of actual dates through uniform random adjustments for each patient—a measure implemented to safeguard privacy—compromises the utility of this temporal information for ETHOS, as it obscures the precise date of the start of PHT. However, the absence of precise reference dates is less critical, given that the entire dataset was collected over a relatively brief period, from 2008 to 2019 (<cit.>).

As mentioned previously, token locations within the timeline are contingent upon the temporal occurrence of events. Nonetheless, certain data elements are temporally invariant, or at least presented as such within the MIMIC-IV database. In our implementation, we designate six static tokens to encapsulate the information encoded in these static data elements. Although, in reality, some of these variables may change over time, they are represented as invariable constants in the MIMIC database. We encoded this information in the six static tokens exactly as recorded in the MIMIC dataset. These include gender, marital status, race, body mass index (BMI), birth date, and the start date of the timeline. While PHTs have the potential to extend to hundreds of thousands of tokens, our current methodology utilizes a maximum of 2048 subsequent tokens within the transformer model context, as elaborated in the "Methods: ETHOS Training" section. To accommodate invariant data, we substitute the initial six tokens of the 2048-token context with static information tokens, where the sixth token demarcates the temporal juncture of the seventh token, which is the first token of the actual timeline. Although the transformer architecture inherently facilitates the inclusion of static data via its encoder component and cross attention module3, we opted for a more streamlined approach as described, deferring the integration of an encoder implementation to future endeavors where more substantial time-invariant data like genetics is used.

Medical encounters yield a plethora of numerical data. We employ a quantile-based tokenization strategy to process continuous numerical values, such as blood pressure readings or cholesterol levels. Specifically, all numerical values are transformed into integers representing the quantile to which each value corresponds. Quantile ranges were determined using the training dataset, where histograms of all numerical values were generated and subsequently divided into quantiles. We chose to utilize ten quantiles, a decision aimed at striking a balance between the need for precise representation of numerical data and the clinical reality that significant changes in health indicators often manifest as relatively large variations, such as shifts of 10 or 20 percent (Figure <ref>). This rationale underpins our selection of ten quantiles for tokenization.

In our study, Diagnosis-Related Group (DRG) codes for each inpatient stay were utilized, despite the absence of assigned times when they were created in the MIMIC tables. Given that a DRG code is assigned after or during discharge, we positioned it after a trio of tokens representing discharge-related information: the discharge token, a quantile token indicating the length of the hospital stay, and a token specifying the discharge destination (e.g., home). Additionally, we incorporated data from MIMIC regarding the initial SOFA score for ICU patients, placing this token after the patient's admission-to-the-ICU token, along with a token denoting the ICU type. Given that the SOFA score in the dataset ranges from 0 to 23 (with the score of 24 never appearing), we uniformly map scores from 0-23 across 1-10 quantiles. Consequently, in quantile Q1, SOFA scores of 0, 1, and 2 (average of 1) are included, while quantile Q2 encompasses SOFA scores of 3 and 4 (average of 3.5), and this pattern continues accordingly.

ETHOS operates as a causal network. It relies solely on information available up to the time being considered in making predictions. Consequently, to ensure causality, actual values of DRG codes and SOFA scores are not employed during inference; instead, predictions of these values are used. This principle ensures that future-obtained information does not influence the prediction of yet-to-occur events. In essence, if tokens are integrated into the timeline based on their approximate occurrence time, their actual values must not be utilized for inference purposes, or they are placed in the timeline far in the future to ensure they are inserted after they occurred.

For the tokenization of drugs, whether administered or prescribed, we utilized the ATC classification system due to its hierarchical, tree-like structure (Figure <ref>). Each ATC code, comprising up to seven characters, was encoded using up to three sequential tokens: the first token for the initial three characters, the second for the subsequent character, and the third optional token, for the remaining suffix. Similarly, ICD-10-CM codes were encoded with three tokens: the first representing the first three characters of the code, the next two by the second token, and the final token capturing the code's remaining suffix. For ICD-10-PCS codes, each character in the seven-character code was represented by a distinct token. The rationale behind such tokenization is that the initial characters in those coding schemes denote specific classes of drugs and diseases or procedures, which are interpretable and have distinct meanings which we anticipated to be important for the network's self-attention mechanisms. Looking ahead, our approach, which assigns well-defined meanings to each token, will be crucial for refining attention mechanisms and enhancing the model's explainability. This method ensures that individual tokens contribute significantly to the interpretability of the network's outcomes. For more information on the tokenization process applied to MIMIC data in our analysis, as well as examples of Patient Health Timelines (PHTs), readers are directed to Table <ref> and Table <ref> where we present real PHTs used in this work with annotations. A summary of all tokenized components of the MIMIC dataset is in Table <ref>.



 §.§ ETHOS training


We employ a model inspired by the decoder architecture of the transformer <cit.>, drawing parallels between tokenized text in Natural Language Processing (NLP) and our approach to tokenizing PHTs. We based our model development on Andrej Kapathy’s implementation of GPT-2 [<https://github.com/karpathy/nanoGPT>]. The design choice slightly varies from the original transformer paper, because instead of using fixed sinusoidal positional encodings, it utilizes learneable position embeddings that are added to the token embeddings at the stage where tokens are converted to their corresponding embeddings. The ETHOS model's training begins by synthesizing a dataset from existing patient records. Each patient's PHT is ended with a "End of timeline" token, and then they are concatenated, creating a single long sequence of tokens for the training. Similarly to generative LLM, ETHOS is trained to predict a single token based on the context of preceding ones. Given the large data scale and model complexity, this phase is resource-intensive similar to methods for training used for NLP transformers used in LLMs (<cit.>). We estimated that the size of the network training task that we face with ETHOS is similar to GPT-2 <cit.>, and therefore we used the size of the transformer used in that network as a starting point (details on the hyperparameter search and choice can be seen in Figure <ref>). We made heuristic adjustments to the size of the network to optimize the value of the loss function. Further details on our training methodology of transformers are provided in <cit.> and for our implementation in supplementary material and full complete code published on GitHub [<https://github.com/ipolharvard/ethos-paper>].



 §.§ Evaluation of Clinical Outcomes and Tasks Using ETHOS


The experiments were chosen so the results can be compared to the work of others in terms of the estimation of inpatient mortality and readmission on MIMIC data. Patients in the MIMIC were randomly divided into training and testing groups, with splits of 90%/10% (Table <ref>).

The chance of inpatient mortality was assessed at the time of admission for all inpatient stays for patients in the test set unless the discharge day was unknown. This was performed by the generative process that began with the admission token and ended upon generating a discharge or death token, repeating this cycle 20 times. The 'N', representing the number of times a death token was generated first, was divided by 20 to estimate the chance of inpatient mortality. Similarly, the likelihood of ICU mortality was computed for the MIMIC dataset, with an additional experiment conducted where predictions were made starting 24 hours after ICU admission, rather than at the point of ICU admission. In the same simulation, the LOS in the ICU was estimated by aggregating the time-interval tokens generated in the simulated timeline until the discharge token appeared. Instances where the patient died in the ICU during the simulation were excluded from the LOS calculation. We opted for 20 repetitions, yielding 21 unique probability estimators, which were adequate for constructing robust Receiver Operating Characteristic (ROC) curves yielding excellent Gaussian fits (Figure <ref>). Nevertheless, alternative repetition counts may also be employed.

To calculate the probability of 30-day inpatient readmission, the generation of fPHTs commenced at the discharge token from inpatient stays and ceased upon the appearance of either a new admission or death token or when the cumulative time tokens generated exceeded 30 days. The simulation was repeated 20 times. The probability of 30-day readmission was then derived as M/20, where 'M' is the count of terminations occurring because of patient new admission tokens across the 20 repetitions.

In our approach, tasks are accomplished by simulating future patient health timelines. Yet, ETHOS offers additional methods for deriving insights, two of which we illustrate here. For instance, in the construction of PHTs following each ICU admission, a sequence is created starting with a token that identifies the type of ICU, followed by a SOFA score token, and then by a Q token that signifies the actual SOFA score on the first day. We predict the SOFA score using SOFA Q node probabilities as generated by ETHOS and the mean SOFA score per quantile as assigned during tokenization (Figure <ref>a).

The exact timing of the 1-day SOFA score assessment is not specified in the dataset, leading to a potential causality issue by inserting the SOFA score immediately after admission, as it relies on data acquired subsequently. During the model's training phase, ETHOS permits this apparent causality violation. However, such true values of 1-day SOFA scores, not available at the moment of ICU admission, are not used for simulating future timelines during inference to prevent causality violation during inference. Instead, these scores are predicted from prior information, as demonstrated in our study. This feature of ETHOS enables the inclusion of information with indeterminate timing.

Another distinctive inference capability facilitated by ETHOS is DRG class estimation. As illustrated in Figure <ref>c, the token denoting the DRG class is consistently positioned following the discharge token and a Q token specifying the length of hospital stay. With 771 unique tokens available for this purpose, we infer the actual class by generating a probability array in the final network layer of the transformer for the DRG token. This array is then utilized to predict the classification's top-1 and top-2 accuracy metrics.



 §.§ Statistical Analysis


The performance of classification algorithms of binary tasks was assessed using Receiver Operating Curve Analysis (ROC). The ROC curves were fitted to experimental points using Gaussian models with unequal variances for binary hypotheses (code provided). Values of Areas Under Curves (AUCs) and 95% confidence intervals (CI) were calculated using bootstrapping (code provided). For multiclass classification (DRG task), we used top-1 and top-2 accuracy. We used mean absolute error (MEA) for the regression tasks to indicate prediction fidelity with 95% confidence intervals estimated using bootstrapping. Python numpy and scikit-learn were used.



 §.§ Comparison of ETHOS to existing methods


Employing the data segmentation as detailed in Table <ref>, we evaluated traditional algorithms for predicting 30-day hospital readmission rates and juxtaposed these outcomes with those obtained via ETHOS. The features used in Figure <ref> were culled from data accrued during the patient's hospitalization, adhering to the feature derivation methodology outlined by <cit.>. Attempting to apply the algorithm devised by the authors to our dataset presented challenges, notably due to the Graph Neural Network (GNN) implementation by Tang et al., which necessitates the computation of a similarity score for each pair of admissions. Given the significantly larger volume of admissions in our dataset—approximately 400,000, in stark contrast to the 14,500 reported by Tang et al.—this task proved impractical on a compute node with 2TB of RAM, defying all efforts to achieve it within a reasonable timeframe. Consequently, we limited our application to the data preprocessing and feature extraction segments of Tang et al.’s methodology. The adapted and modified code from Tang et al.’s repository, which we cloned for feature extraction, is accessible on GitHub [<https://github.com/ipolharvard/readmit-stgnn>]. For models unsupportive of temporal sequence analysis, such as Logistic Regression and XGBoost, we modified the approach to handle time-varying features by consolidating them over time. This entailed distilling the minimum, first quartile, median, third quartile, and maximum values of dynamically changing features. Furthermore, we integrated the day of admission as a unique feature to retain an element of temporal dimension within the dataset. In Figure <ref> ETHOS was compared to one of the leading proprietary LLM models - GPT-4o in two temperature variants: 0.3 and 0.5. We constructed a comprehensive prompt that directs the model to analyze a timeline of 2048 tokens and calculate the probability of patient readmission for 2000 cases from the test set. This prompt is structured into four distinct parts: task instructions, a basic patient description corresponding to ETHOS's static information, PHT and a detailed description of subgroups of tokens and their identification. The complete codebase for this experiment including the prompt design is accessible on GitHub [<https://github.com/ipolharvard/ethos-paper/blob/master/notebooks/llm_readmission_task.ipynb>]. ETHOS significantly outperforms both variants of GPT-4o for the same subset of testing samples.



§ ADDITIONAL INFORMATION




  
Data Availability The MIMIC-IV dataset is publicly available at <https://physionet.org/content/mimiciv/2.2>.



  
Code Availability The code, ETHOS model weights used for all inferences, results of inferences, scripts to generate numerical results for all aspects of this study for the MIMIC-IV dataset are made publicly available at <https://github.com/ipolharvard/ethos-paper>. In our experiments, we used Python 3.10, and the following open-source libraries: torch=2.3.0, joblib=1.4.2, tqdm=4.66.4, colorlog=6.8.2, h5py=3.11.0, pandas=2.2.2, numpy=1.26.4, pyarrow=16.1.0, click=8.1.7.



  
Author Contribution AS and PR conceptualized the work. AS, PR, YJ, AES designed the study. PR, AS performed the coding and the experiments. YJ, AS conducted the literature search. DB, AES, QL, JW provided advisory support for the project. PR and AS prepared the initial draft of the manuscript, with all authors actively participating in the refinement and finalization of the manuscript through comprehensive review and contributions. AS supervised the project.



  
Competing Interests YJ is currently also affiliated with Verily life science, SSF, CA. The other authors declare no competing interests.

unsrtnat




§ SUPPLEMENTARY INFORMATION






















ll
    Example of a tokenized timeline with a hospital admission

    Patient Health Timeline     Notes 

    

    2c
      – Continued from previous page 

    Patient Health Timeline     Notes 

    

    2rContinued on next page 

    

    ED_ADMISSION_START     Admission to Emergency Department 

    _2h-6h     Time interval between 2h and 6h 

    LAB_pH_units     Lab test of pH and the unit is units 

    _Q4     Quantile token referring to the lab test of pH 

    LAB_Protein_mg/dL     

    _Q4     

    LAB_RBC_#/hpf     

    _Q1     

    LAB_Specific Gravity     

    _Q2     

    LAB_WBC_#/hpf     

    _Q1     

    LAB_Alanine Aminotransferase (ALT)_IU/L     

    _Q3     

    LAB_Albumin_g/dL     

    _Q3     

    LAB_Alkaline Phosphatase_IU/L     

    _Q1     

    LAB_Anion Gap_mEq/L     

    _Q5     

    LAB_Asparate Aminotransferase (AST)_IU/L     

    _Q8     

    LAB_MCHC_g/dL     

    _Q9     

    LAB_MCV_fL     

    _Q7     

    LAB_Monocytes_%     

    _Q7     

    LAB_Neutrophils_%     

    _Q3     

    LAB_Platelet Count_K/uL     

    _Q3     

    LAB_RDW_%     

    _Q1     

    LAB_Red Blood Cells_m/uL     

    _Q8     

    LAB_White Blood Cells_K/uL     

    _Q5     

    LAB_Absolute Basophil Count_K/uL     

    _Q5     

    LAB_Absolute Eosinophil Count_K/uL     

    _Q4     

    LAB_Absolute Monocyte Count_K/uL     

    _Q6     

    _6h-12h     

    INPATIENT_ADMISSION_START     Token indicating an admission to the hospital 

    TYPE_OBSERVATION     The type of the admission is observation 

    INSURANCE_MEDICARE     The patient’s insurance is MEDICARE 

    [l]ICD_Other symptoms and signs involving cognitive 
 functions and awareness     [l]The primary diagnosis at the beginning of the hospital stay
 is Altered mental status, unspecified (R4182), which is 
 broken down into two tokens: R41 and 82 

    ICD_4-5_82     

    ICD_Dorsalgia     

    ICD_4-5_16     

    ICD_Malaise and fatigue     

    ICD_4-5_1     

    ICD_Personal history of certain other diseases     

    ICD_4-5_73     

    ICD_Other hypothyroidism     

    ICD_4-5_9     

    ICD_Type 2 diabetes mellitus     

    ICD_4-5_21     

    ICD_Dorsalgia     

    ICD_4-5_5     

    ICD_Abnormalities of gait and mobility     

    ICD_4-5_2     

    TRANSFER_MED     Transfer to a different care unit - MED 

    _6h-12h     

    ATC_stomatological preparations     

    ATC_4_A     

    ATC_SUFFIX_D05     

    ATC_diuretics ATC_4_A     

    ATC_4_A     

    ATC_SUFFIX_A03     

    ATC_agents acting on the renin-angiotensin system     

    ATC_4_A     

    ATC_SUFFIX_A03     

    _6h-12h     

    ED_ADMISSION_END     Discharge from Emergency Department 

    _Q10     [l]Quantile referring to the length of the stay in Emergency 
 Department based on all stays in the data 

    INPATIENT_ADMISSION_END     Discharge from the hospital 

    _Q2     Quantile referring to the length of the stay in the hospital 
 based on all stays in the data 

    DISCHARGED_UNKNOWN     The reason of discharge 

    UNKNOWN_DRG     DRG assigned to the hospital stay 

    _=6mt     Time interval of 6 months 

    

ll
    Example of a tokenized timeline with an admission to ICU

    Patient Health Timeline     Notes 

    

    2c
      – Continued from previous page 

    Patient Health Timeline     Notes 

    

    2rContinued on next page 

    

    ICU_STAY_START     Admission to ICU token 

    Surgical Intensive Care Unit (SICU)     Type of ICU 

    SOFA     Indicates SOFA quantile follows 

    _Q1     Q indicating quantile of SOFA score 

    _15m-1h     time-interval 

    LAB_Sodium_mEq/L     Lab test results 

    _Q7     

    LAB_Urea Nitrogen_mg/dL     

    _Q3     

    LAB_L_no_unit     

    _Q4     

    _15m-1h     time-interval 

    LAB_Base Excess_mEq/L     

    _Q6     

    LAB_Calculated Total CO2_mEq/L     

    _Q8     

    LAB_pCO2_mm Hg     

    _Q7     

    LAB_pH_units     

    _Q7     

    LAB_pO2_mm Hg     

    _Q5     

    _2h-6h     

    ATC_anti-asthmatics     Medications start and administered with specified time intervals 

    ATC_4_A     

    ATC_SUFFIX_C02     

    _5m-15m     

    ATC_analgesics     

    ATC_4_B     

    ATC_SUFFIX_E01     

    _2h-6h     

    ATC_drugs for acid related disorders     

    ATC_4_B     

    ATC_SUFFIX_C02     

    _1h-2h     

    ATC_antithrombotic agents     

    ATC_4_A     

    ATC_SUFFIX_B01     

    _15m-1h     

    ATC_beta blocking agents     

    ATC_4_A     

    ATC_SUFFIX_B02     

    _1h-2h     

    ATC_mineral supplements     Patient is medicated until the end of ICU stay 

    ATC_4_C     

    ATC_SUFFIX_C02     

    _15m-1h     

    ATC_laxatives     

    ATC_4_    

    A ATC_SUFFIX_D04     

    _1h-2h     

    ATC_analgesics     

    ATC_4_B     

    ATC_SUFFIX_E01     

    _15m-1h     

    ATC_antipruritics, anesthetics, etc.     

    ATC_4_    

    A ATC_SUFFIX_A32     

    LAB_Hematocrit_%     Here lab test results are reported 

    _Q7     

    LAB_Hemoglobin_g/dL     

    _Q6     

    LAB_MCH_pg     

    _Q8     

    LAB_MCHC_g/dL     

    _Q5     

    LAB_MCV_fL     

    _Q9     

    LAB_Platelet Count_K/uL     

    _Q3     

    LAB_RDW_%     

    _Q4     

    LAB_Red Blood Cells_m/uL     

    _Q5     

    LAB_White Blood Cells_K/uL     

    _Q4     

    LAB_RDW-SD_fL     

    _Q6     

    LAB_INR(PT)_no_unit     

    _Q4     

    LAB_PT_sec     

    _Q5     

    LAB_PTT_sec     

    _Q2     

    LAB_Anion Gap_mEq/L     

    _Q4     

    LAB_Bicarbonate_mEq/L     

    _Q4     

    LAB_Calcium, Total_mg/dL     

    _Q3     

    LAB_Chloride_mEq/L     

    _Q5     

    LAB_Creatinine_mg/dL     

    _Q2     

    LAB_Glucose_mg/dL     

    _Q8     

    LAB_H_no_unit     

    _Q9     

    LAB_I_no_unit     

    _Q2     

    LAB_Magnesium_mg/dL     

    _Q2     

    LAB_Phosphate_mg/dL     

    _Q1     

    LAB_Potassium_mEq/L     

    _Q6     

    LAB_Sodium_mEq/L     

    _Q6     

    LAB_Urea Nitrogen_mg/dL     

    _Q2     

    LAB_L_no_unit     

    _Q2     

    _1h-2h     

    ATC_antithrombotic agents     

    ATC_4_A     

    ATC_SUFFIX_B01     

    ICU_STAY_END     Patient is discharged from ICU 

    _Q5     [l]Quantile token indicates the length of stay in ICU. 
 Here it is 5, indicating about average 





 Paper Title: Human-Data Interaction Framework: A Comprehensive Model for a Future Driven by Data and Humans.


Uplink Wave-Domain Combiner for Stacked Intelligent Metasurfaces Accounting for Hardware Limitations
    Maryam Rezvani1, Raviraj Adve1, Akram bin Sediq2, Amr El-Keyi2
	1Dept. of Electrical and Computer Engineering,
		University of Toronto, 
		Toronto, Canada 
	
	2Ericsson Canada, 
		Ottawa, Canada, 

		emails: {mrezvani, rsadve}@ece.utoronto.ca, {akram.bin.sediq, amr.el-keyi}@ericsson.com

    August 5, 2024
====================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
	In an age defined by rapid data expansion, the connection between individuals and their digital footprints has become more intricate. The Human-Data Interaction (HDI) framework has become an essential approach to tackling the challenges and ethical issues associated with data governance and utilization in the modern digital world. This article outlines the fundamental steps required for organizations to seamlessly integrate HDI principles, emphasizing auditing, aligning, formulating considerations, and the need for continuous monitoring and adaptation. Through a thorough audit, organizations can critically assess their current data management practices, trace the data lifecycle from collection to disposal, and evaluate the effectiveness of existing policies, security protocols, and user interfaces. The next step involves aligning these practices with the main HDI principles, such as informed consent, data transparency, user control, algorithm transparency, and ethical data use, to identify gaps that need strategic action. Formulating preliminary considerations includes developing policies and technical solutions to close identified gaps, ensuring that these practices not only meet legal standards, but also promote fairness and accountability in data interactions. The final step, monitoring and adaptation, highlights the need for setting up continuous evaluation mechanisms and being responsive to technological, regulatory, and societal developments, ensuring HDI practices stay up-to-date and effective. Successful implementation of the HDI framework requires multi-disciplinary collaboration, incorporating insights from technology, law, ethics, and user experience design. The article posits that this comprehensive approach is vital for building trust and legitimacy in digital environments, ultimately leading to more ethical, transparent, and user-centric data interactions.









§ EXECUTIVE SUMMARY

This article presents a comprehensive field of Human-Data Interaction (HDI), addressing the complex challenges of ethical data management in the digital age. The HDI framework aims to balance the exponential growth of data with the need to respect individual rights, promote transparency, and ensure ethical data practices.
Key components of the framework include:


    
  * Foundational Principles: Emphasizing human agency, transparency, fairness, accountability, privacy, and stakeholder engagement.
    
  * Structural Components: Outlining data governance frameworks, technological solutions, and human-centric practices necessary for effective HDI implementation.
    
  * Implementation Roadmap: Providing a phased approach for organizations to integrate HDI principles, including assessment, policy development, implementation, and continuous adaptation.
    
  * Ethical and Practical Considerations: Addressing challenges such as data complexity, bias mitigation, user empowerment, and regulatory compliance.
    
  * Training and Education: Stressing the importance of cultivating an HDI-centric culture through comprehensive training programs for employees and education initiatives for users.
    
  * Monitoring and Adaptation: Highlighting the need for continuous evaluation and responsiveness to technological, regulatory, and societal shifts.


The article argues that adopting HDI principles is not only a moral imperative but also essential for building trust and legitimacy in digital ecosystems. It emphasizes the need for a multidisciplinary approach, combining insights from technology, law, ethics, and user experience design.
Future directions for HDI research and implementation are discussed, including the development of adaptive ethical frameworks, technological innovations, and strategies for global application.

This work contributes to the growing field of ethical data management by providing a structured approach for organizations to navigate the complexities of data interactions in a manner that respects human values and promotes social well-being.



§ INTRODUCTION


In today's rapidly evolving digital landscape, data has become the lifeblood of our interconnected world. The exponential growth of digital technologies has led to an era in which vast amounts of information are generated, collected, and analyzed at unprecedented scales. From smartphones in our pockets to the expansive networks of smart city infrastructure, data flows endlessly, shaping our daily lives in ways both visible and unseen. This data-driven revolution has led to remarkable advancements in fields ranging from healthcare and scientific research to commerce and public services. Predictive analytics inform critical business decisions, machine learning algorithms power personalized experiences, and big data insights drive policy making at national and global levels. The potential for innovation and progress seems boundless <cit.>.

However, this data-centric paradigm is not without its challenges. As our reliance on data-driven systems intensifies, we find ourselves grappling with profound ethical, privacy, and security concerns. The Cambridge Analytica scandal laid bare the vulnerabilities of personal data and its potential for manipulation. Algorithmic bias has raised alarming questions about fairness and discrimination in automated decision-making processes. Meanwhile, data breaches continue to erode public trust, highlighting the precarious nature of digital information security.
In addition, the complexity and opacity of many data systems have created a widening gap between those who control these technologies and those affected by them <cit.>. Individuals often find themselves navigating a digital landscape where their personal information is commodified, their behaviors are predicted and influenced, and their futures are increasingly shaped by algorithms they neither understand nor control.

In this context, a critical juncture is presented. How do we harness the transformative power of data while safeguarding individual rights, fostering transparency, and ensuring ethical practices? How can we build data ecosystems that not only drive innovation, but also empower individuals and promote societal well-being?

These pressing questions underscore the need for a fundamental shift in how we approach data interaction and management. It is against this backdrop that the concept of Human-Data Interaction (HDI) emerges as a crucial paradigm, offering a path towards a more equitable, transparent, and human-centric data future. In response to these pressing challenges, the concept of HDI emerges as a transformative paradigm and represents a fundamental shift in how we approach the relationship between individuals and the data ecosystems that increasingly shape our world. At its core, HDI aims to rebalance the power dynamics in data-driven systems, placing human agency, understanding, and values at the center of data practices <cit.>.

HDI goes beyond traditional notions of data privacy and security, proposing a holistic framework that encompasses the entire lifecycle of data interactions. It recognizes that individuals are not just subjects of data collection, but active participants in a complex data ecosystem. Using three key principles: legibility, agency and negotiability, HDI seeks to empower individuals with the ability to understand, control and negotiate their data relationships <cit.>.

Legibility addresses the often opaque nature of data systems, advocating for transparency and comprehensibility in how data are collected, processed, and used. The agency emphasizes the importance of meaningful control that allows individuals to make informed decisions about their data and its implications. Negotiability acknowledges the dynamic nature of data relationships, promoting flexible and context-sensitive approaches to data rights and permissions <cit.>.

By integrating these principles, HDI offers a framework for developing more ethical, transparent, and user-centric data practices. It challenges organizations to rethink their data strategies, moving beyond compliance-driven approaches to embrace a more collaborative and responsible ethos. For policymakers, HDI provides a lens through which to craft more nuanced and effective data governance frameworks.

Importantly, HDI is not just a theoretical construct, but a call to action. It requires practical implementation in the technological, organizational, and societal domains. By adopting HDI principles, It is possible to construct data ecosystems that facilitate innovation and advancement while simultaneously upholding human dignity, fostering trust, and contributing to the collective well-being of society..

As we navigate the complexities of our data-driven future, HDI offers a beacon of hope a pathway to harness the power of data while upholding the values that make us human. This article explores the foundations, implications, and practical applications of HDI, charting a course toward a more equitable and empowering data landscape for all.

This article aims to provide a comprehensive exploration and analysis of the Human-Data Interaction (HDI) framework, with the following key objectives and contributions:


    
  * Conceptual Clarification:  It provides a comprehensive analysis of the fundamental principles of HDR, offering a clear and accessible explanation of its core tenets legibility, agency, and negotiability. This contributes to a more nuanced understanding of HDI within the broader context of data ethics and governance.
    
  * Framework Development: it presents an expanded HDI framework that builds upon existing literature, integrating insights from diverse fields such as computer science, ethics, law, and social sciences. This interdisciplinary approach results in a more robust and applicable model for HDI implementation.
    
  * Implementation Roadmap: A significant contribution of this article is the development of a practical, step-by-step roadmap for organizations to adopt and integrate HDI principles. This guide addresses the gap between theoretical understanding and practical application of HDI.
    
  * Ethical and Practical Considerations:The ethical implications and practical challenges associated with the implementation of the HDR are analysed in depth. By doing so, It contributes to the ongoing dialogue on responsible data practices and offers strategies to overcome potential obstacles.
    
  * Future Directions: It identifies emerging trends and areas for future research in HDR, thereby contributing to the evolution of the field and highlighting opportunities for further academic and practical exploration.
    
  * Interdisciplinary Synthesis: By drawing connections between HDI and related fields such as data ethics, privacy law, and user experience design, we contribute to a more holistic understanding of data governance in the digital age.
    
  * Policy Implications:The potential impact of the widespread adoption of HDR on data protection policy and regulation is discussed, offering insights that can inform the development of future policy.


Through these objectives and contributions, this article aims to advance both the theoretical understanding and practical application of Human-Data Interaction, providing valuable insights for researchers, practitioners, policymakers, and organizations navigating the complexities of ethical data management in the 21st century.

The timeliness and significance of this work cannot be overstated, as HDI has become increasingly crucial in our current technological and societal landscape for several interconnected reasons:


    
  * Data Ubiquity and Complexity: The pervasiveness of data collection and processing in everyday life has created what Zuboff (2015) terms "surveillance capitalism" <cit.>. As our digital footprints expand, HDI provides a framework for managing the complex relationships between individuals and their data. Andrejevic (2014) argues that the sheer volume and variety of data collected about individuals necessitates new approaches to data governance and interaction <cit.>.
    
  * Algorithmic Decision-Making: With the rise of machine learning and AI, algorithms increasingly influence critical aspects of our lives. Mittelstadt et al. (2016) highlight the ethical challenges posed by algorithmic decision-making, including issues of transparency, fairness, and accountability <cit.>. HDI principles, particularly legibility, are crucial for addressing these challenges and ensuring that individuals can understand and challenge decisions made about them.
    
  * Privacy Concerns: High-profile data breaches and misuse cases have eroded public trust in data-driven systems. Auxier et al. (2019) report that a majority of Americans feel they have little to no control over their personal data <cit.>. HDI's emphasis on agency and negotiability offers a pathway to restore trust and give individuals more control over their data.
    
  * Regulatory Landscape: The introduction of comprehensive data protection regulations like the GDPR in Europe and the CCPA in California reflects growing societal concerns about data privacy. Cate and Mayer-Schönberger (2013) argue that traditional notice and consent models are insufficient in the big data era <cit.>. HDI provides a more nuanced and flexible approach to data governance that aligns with these evolving regulatory requirements.
    
  * Digital Divide and Data Literacy: As data-driven services become more integral to daily life, there's a risk of exacerbating existing inequalities. Ragnedda and Muschert (2013) discuss how differential access to and understanding of digital technologies can reinforce social disparities <cit.>. HDI's focus on legibility and agency can help bridge this gap by making data systems more accessible and understandable to all.
    
  * Ethical AI and Responsible Innovation: As AI systems become more advanced, ensuring their ethical development and deployment is paramount. Floridi et al. (2018) propose an ethical framework for AI that aligns closely with HDI principles <cit.>. By centering human values and agency, HDI contributes to the responsible development of AI and other data-driven technologies.
    
  * Data as a Human Right: There's growing recognition of data protection and privacy as fundamental human rights. Banisar and Davies (1999) argue for the importance of information privacy in maintaining human dignity and autonomy <cit.>. HDI provides a practical framework for operationalizing these rights in the digital sphere.
    
  * Personalization vs. Privacy Paradox: Users often express privacy concerns while simultaneously engaging in privacy-compromising behaviors for perceived benefits. Acquisti et al. (2015) explore this "privacy paradox" <cit.>. HDI's negotiability principle offers a way to navigate this tension by allowing for more nuanced and context-dependent data sharing decisions.


As we delve into the complexities of our data-driven world, it becomes increasingly clear that a new paradigm is needed to address the ethical, social, and technical challenges that arise from our interactions with data systems. This necessity forms the foundation of HDI, a concept that seeks to redefine our relationship with data and data-driven technologies. 

To fully grasp the transformative potential of the HDI Framework presented in this article, it is crucial to first understand the fundamental principles and scope of HDI itself. The following section provides a comprehensive definition of Human-Data Interaction, setting the stage for our exploration of its practical applications and far-reaching implications.



§ HUMAN-DATA INTERACTION DEFINITION

Human-Data Interaction (HDI) is an emerging field that seeks to place the human at the center of the data flows and processes that characterize our digital world. As defined by Mortier et al. (2014), HDI focuses on the interactions between individuals and data systems, emphasizing the need for these systems to be designed with human values and agency in mind <cit.>. HDI extends beyond traditional human-computer interaction by specifically addressing the challenges posed by ubiquitous data collection, processing, and use in modern society.
The core principles of HDI, as initially proposed by Mortier et al. and further developed by subsequent researchers, are:


    
  * Data Legibility: This principle emphasizes the importance of making data processes and systems understandable to individuals. As Crabtree and Mortier (2015) argue, data legibility involves not just transparency, but also the provision of tools and interfaces that allow people to comprehend how their data is collected, analyzed, and used <cit.>. This includes clear communication about data practices and the potential implications of data use.
    
  * Agency: The principle of agency focuses on empowering individuals to take action regarding their personal data. This goes beyond mere consent to data collection, advocating meaningful control over data throughout its lifecycle. Luger et al. (2015) highlight that agency in HDI contexts involves giving individuals the ability to make informed decisions about their data, including the right to access, modify, and delete personal information <cit.>.
    
  * Negotiability: This principle recognizes that the context and meaning of data can change over time and across different situations. As such, HDI advocates for systems that allow for ongoing negotiation of data use and permissions. Coles-Kemp et al. (2018) emphasize that negotiability is crucial for addressing the dynamic nature of privacy preferences and the evolving relationships between individuals and data-driven services <cit.>.
 
These core principles are interconnected and mutually reinforcing. For instance, data legibility is a prerequisite for meaningful agency, as individuals can only make informed decisions about their data if they understand how it is being used. Similarly, negotiability supports both legibility and agency by allowing for adaptable and context-sensitive data relationships.

Recent studies have expanded on these core principles. For example, Victorelli et al. (2020) propose additional dimensions to the HDI framework, including data quality, security, and ethical use <cit.>. Their work underscores the evolving nature of HDI and its responsiveness to emerging challenges in the data-driven landscape.

Furthermore, Sailaja et al. (2019) have applied HDI principles to specific domains such as media experiences, demonstrating how these concepts can be operationalized in real-world contexts <cit.>. Their research highlights the practical implications of HDI and its potential to transform user experiences in data-intensive environments.

By defining and elaborating on these core principles, HDI provides a comprehensive framework for addressing the complex challenges of data ethics, privacy, and user empowerment in the digital age. It offers a foundation for developing more human-centric data practices and technologies that respect individual rights while harnessing the potential of data-driven innovation.

The multifaceted nature of Human-Data Interaction, as defined above, underscores the complexity of the challenges it aims to address. These challenges span across technological, ethical, legal, and social domains, highlighting the limitations of singular, discipline-specific approaches. 

To fully realize the potential of HDI and effectively implement its principles, we must draw upon diverse fields of expertise and foster collaboration across traditional boundaries. This necessity for a holistic perspective leads us to explore the critical importance of interdisciplinary approaches in HDI. By integrating insights from various disciplines, we can develop more comprehensive and effective solutions to the intricate problems posed by our increasingly data-driven world.



§ IMPORTANCE OF INTERDISCIPLINARY APPROACHES

The HDI Framework underscores the necessity of employing interdisciplinary approaches to develop comprehensive and effective solutions for the challenges posed by the increasing leverage of personal data <cit.>. By integrating perspectives from diverse fields, the framework can harness the complementary expertise and insights required to address the multifaceted nature of human-data interactions.For instance, human-computer interaction can provide valuable insights into user needs, preferences, and behaviors, while data science can offer technical expertise in data manipulation, analysis, and modeling. 

Additionally, fields such as ethics and policy can contribute to the development of ethical frameworks, governance structures, and regulatory mechanisms that ensure the responsible and trustworthy use of personal data <cit.>

The framework also seeks to identify and mitigate the socio-technical challenges that arise from the increasing leverage of personal data in various domains, such as privacy concerns, data transparency and control, accountability issues, and lack of trust.These challenges have been studied and discussed in various disciplines, but effective responses that alleviate them are yet to be fully realized <cit.>

The proposed framework takes an interdisciplinary approach, integrating perspectives from fields such as human-computer interaction, data science, ethics, and policy to develop comprehensive and effective solutions <cit.>.By addressing these socio-technical challenges, the framework aims to create data-driven experiences and systems that are aligned with user needs and values, ultimately fostering a more equitable and user-centric data-driven landscape.<cit.>

The multifaceted nature of Human-Data Interaction necessitates an interdisciplinary approach, drawing insights from diverse fields such as computer science, ethics, law, and social sciences. This interdisciplinary perspective becomes particularly crucial when considering the complex and dynamic nature of the data lifecycle. As data moves through its various stages from collection and processing to storage, utilization, and eventual disposal it intersects with a wide array of disciplines and stakeholders. 

The integration of diverse expertise ensures that each phase of the data lifecycle is approached with a comprehensive understanding of its technical, ethical, and societal implications. For instance, while computer scientists might focus on secure data storage methods, ethicists can provide insights on fair data use, and legal experts can ensure compliance with data protection regulations. By bridging the gap between these disciplines, the HDI framework can more effectively address the challenges that arise at each stage of the data lifecycle, fostering a more holistic and human-centric approach to data management.



§ THE DATA LIFECYCLE

The Data Lifecycle is a critical component of the HDI Framework as it encapsulates the journey of data from its creation to its eventual disposal. This holistic view ensures that ethical considerations and human-centric practices are applied at every stage of data handling. In the contemporary landscape, data pervades every aspect of our society, influencing diverse domains from personalized service offerings to strategic policy-making on a global scale. The utilization of algorithms, powered by this extensive data, is pivotal in sculpting our daily interactions. They undertake tasks of substantial complexity and consequence, from analyzing medical imagery to curating news feeds, thereby making decisions that carry significant ramifications for society<cit.>. Yet, the operationalization of these data-driven frameworks remains an intricate challenge, as they must grapple with the dynamic and relational nature of data itself. 

As data moves through the stages of the lifecycle, from gathering and organizing to analysis and application, it becomes enmeshed with multifaceted social, ethical, and political implications<cit.>.The HDI Framework seeks to address these complexities by placing the human at the center of the data lifecycle, ensuring that the design, deployment, and governance of data-driven systems are aligned with user needs,preferences, and values.

Despite the undeniable influence of big data, there exists a peril of neglecting the individual lives encapsulated within these datasets, lives that are invariably affected by the operational systems<cit.>. While these systems have the potential to yield positive outcomes, the "tyranny of data" can also lead to violations of privacy, information asymmetry, lack of transparency, discrimination, and social exclusion <cit.>

To mitigate these risks, the proposed HDI Framework advocates for a comprehensive understanding of the data lifecycle, encompassing the processes of collection, storage, processing, and utilization <cit.>By examining each stage of this lifecycle through the lens of human values, needs, and preferences, the framework aims to develop solutions that empower individuals and foster trust in data-driven decision-making <cit.>.

In reaction to this oversight, the domain of HDI has been proposed, advocating for a data environment that is ethical and empowering. Within this framework, individuals are regarded not as mere subjects of data extraction but as active agents endowed with rights to agency, transparency, and safeguards against data misuse <cit.>. This discourse posits that the creation of a data ecosystem centered around human interests is not merely a moral obligation but is also essential for sustaining trust and legitimacy in the digital era. 

To this end, the proposed framework delineates the data lifecycle through the lens of human-centric values, encompassing the following key phases:


  * Data collection: Ensuring informed consent and transparency around the purpose and use of collected data <cit.>.
  * 

Data processing: Enabling users to understand the algorithms and inferences applied to their data, as well as the ability to contest and rectify inaccuracies <cit.>.
  * 

Data storage and management: Providing users with control over the retention and deletion of their personal information, as well as mechanisms for data portability <cit.>.
  * . 

Data utilization: Empowering users to comprehend how their data is being leveraged and to exercise agency in determining the scope and context of its use <cit.>.
  * .

Data sharing and exchange: Affording users the right to selectively share their data and to revoke access as needed, fostering a sense of data ownership and control <cit.>.
  * 

Data disposal: Ensuring the secure and responsible deletion of user data upon request or when it is no longer needed, in alignment with privacy regulations and user preferences <cit.>.
  * 

Data analytics and insights: Enabling users to understand the inferences and predictions made about them, and to dispute or correct any erroneous conclusions <cit.>.
  * 

Ongoing monitoring and evaluation: Establishing mechanisms for continuous assessment and improvement of the data ecosystem, addressing evolving user needs and emerging technological advancements <cit.>.


The HDI Framework proposes a comprehensive, human-centric approach to the data lifecycle, aiming to instill individual empowerment and collective accountability in our data-driven landscape <cit.>. This paradigm shift goes beyond mere technical solutions, outlining a structured approach for embedding ethical principles throughout the entire data journey.

Central to this framework is the development of stringent policies upholding informed consent and data sovereignty, the integration of user-empowerment tools, and the promotion of data ethics education <cit.>. Despite the complexities of modern data systems and the risk of algorithmic bias <cit.>, the adoption of HDI principles offers profound benefits, positioning organizations to utilize data responsibly, foster innovation, and contribute to societal well-being.

The HDI Framework represents a transformative shift, placing individuals at the core of shaping our data-driven future. By prioritizing user agency, transparency, and ethics, it lays the groundwork for a more equitable and empowering data ecosystem <cit.>.

Human engagement is crucial throughout the data lifecycle to ensure ethical, equitable, and socially beneficial data practices <cit.>. This involves empowering individuals with agency and choice in data collection, processing, and utilization, while fostering transparency and accountability in data-driven systems.
At the collection stage, individuals should receive comprehensive information about the purpose, scope, and intended use of their data <cit.>. This transparency enables informed consent and allows for selective data sharing based on personal preferences.
During processing and analysis, individuals must be empowered to understand the algorithms and processes applied to their data, including potential implications. This legibility allows users to contest inaccuracies and understand how their data is being used <cit.>.

In storage and management, individuals should have control over data retention and deletion, with data portability features allowing seamless transfer between service providers <cit.>.The utilization and sharing phases require balancing individual privacy with potential societal benefits from data-driven insights. Individuals should be able to selectively share data while maintaining control over access and scope.
By embedding human agency, transparency, and control throughout the data lifecycle, the HDI Framework fosters a more equitable and empowering data ecosystem <cit.>. 
This approach safeguards individual rights while incentivizing responsible organizational data practices.
Crucially, the application of user data in decision-making must include mechanisms for user oversight and the ability to challenge decisions, empowering individuals to shape how their information is used and fostering accountability and transparency <cit.>. 

Incorporating this lifecycle approach into the HDI Framework ensures consistent application of human-centric principles across all data-related activities, enhancing both ethical practices and overall data management efficiency.

While the Data Lifecycle provides a comprehensive framework for managing data ethically throughout its existence, implementing such a structure requires careful planning and strategic considerations. As organizations prepare to embed HDI principles across the data lifecycle, they must first address several preliminary considerations. 

These foundational steps ensure that the organization is well-positioned to successfully integrate HDI practices into every stage of the data journey. By thoughtfully examining these preliminary considerations, organizations can create a solid groundwork for implementing a human-centric approach to data management that aligns with ethical standards, regulatory requirements, and stakeholder expectations. 

The following section outlines these crucial preliminary considerations, which serve as a bridge between understanding the theoretical importance of the data lifecycle and practically implementing HDI principles within an organization's unique context.




§ PRELIMINARY CONSIDERATIONS


Before implementing a Human-Data Interaction (HDI) framework, organizations must carefully evaluate and prepare their existing structures, processes, and culture. These preliminary considerations serve as a crucial foundation, ensuring that the subsequent implementation of HDI principles is well-informed, strategically sound, and aligned with both organizational values and ethical data practices. By thoroughly addressing these initial aspects, organizations can pave the way for a more successful and sustainable integration of HDI principles into their core operations. The following subsections outline key areas that demand attention in this preparatory phase. This stage involves laying the groundwork for implementing the insights gleaned from alignment analysis, focusing on ethical, technical, and operational considerations to improve data practices. This proactive approach ensures that the subsequent steps taken are informed, deliberate, and aligned with both organizational values and HDI principles.



 §.§ Ethical Considerations

At the forefront of preliminary considerations is the emphasis on ethics. Organizations must deliberate on the ethical implications of their data practices, particularly in terms of respecting user autonomy, ensuring privacy, and preventing harm<cit.>.This involves considering how data is collected, used, and shared, ensuring that these practices are not only legally compliant but also ethically sound. The work of Shilton<cit.> underscores the importance of embedding ethical considerations into the fabric of organizational culture and decision-making processes, ensuring that data practices reflect a commitment to respect, fairness, and responsibility toward individuals whose data are being handled. 

To this end, organizations should establish a cross-functional ethics review board or advisory panel, tasked with evaluating data-driven initiatives through the lens of HDI principles.




 §.§ Technical Considerations

Technical considerations form the backbone of effective HDI implementation. Organizations must assess the capabilities and limitations of their existing technological infrastructure, identify areas for improvement, and ensure that data practices are underpinned by robust, user-centric systems.

This may involve enhancing data management and governance frameworks, implementing user-friendly data access and control interfaces, and deploying advanced analytics and visualization tools that enhance data legibility and transparency. 

Furthermore, organizations should explore the potential of emerging technologies, such as blockchain and secure multi-party computation, to bolster data security, provenance, and user control <cit.>

Organizations must consider investments in technologies that enable more granular consent mechanisms, robust encryption, and secure data storage and transmission protocols<cit.>. In addition, the development or adoption of tools for bias detection and mitigation in algorithms, as well as those that enhance the explainability of machine learning models, is critical. These technical upgrades and innovations are essential to operationalize the HDI principles in a way that is both effective and sustainable.




 §.§ Operational Considerations

Operational considerations encompass the processes and policies that govern data practices within the organization. This includes re-visiting data governance frameworks to explicitly incorporate HDI principles, ensuring that data collection, storage, use, sharing, and disposal policies are aligned with ethical standards and technical capabilities<cit.>. The establishment of clear, transparent communication channels for stakeholders, including data subjects, is crucial for fostering trust and accountability. 

Operational considerations also involve the training and engagement of employees across the organization, from leadership to frontline staff, in understanding and championing HDI principles. The work of Burr et al.<cit.> highlights the importance of integrating legal, technical, and policy expertise in crafting and implementing data practices that are not only compliant but also ethically and socially responsible.




 §.§ Stakeholder Engagement

Engaging with stakeholders, including users, regulatory bodies, and civil society organizations, is a critical component of preliminary considerations. This engagement ensures that the organization's efforts to implement HDI principles are responsive to the needs and concerns of those affected by its data practices<cit.>. Through workshops, focus groups, and other participatory methods, organizations can gain valuable insights into user expectations, preferences, and concerns regarding data management.

User participation, in particular, is essential to understand the unique needs, preferences, and concerns of individuals whose data are being collected and used. This may involve conducting user research, collecting feedback on data practices, and establishing mechanisms for ongoing dialogue and collaboration<cit.>.Engagement with regulatory bodies and civil society organizations can provide valuable insights into emerging trends, best practices, and potential pitfalls in HDI implementation, thus guiding the organization's efforts and ensuring compliance with relevant laws and regulations.

Stakeholder feedback can provide valuable insights into potential ethical dilemmas, privacy concerns, and areas for improvement in transparency and accountability. Furthermore, this engagement fosters a collaborative approach to data governance, improving the legitimacy and effectiveness of the organization's data practices.




 §.§ Alignment Analysis

The alignment analysis phase involves a comprehensive assessment of the organization's existing data practices against the principles of HDI. This analysis serves as a foundational step in identifying areas of alignment, gaps, and opportunities for improvement, forming the subsequent implementation of HDI-centric initiatives <cit.>.

The analysis should consider the organization's data ecosystem holistically, examining factors such as data collection, storage, usage, sharing, and disposal practices. Additionally, the analysis should assess the organization's technological capabilities, governance frameworks, and stakeholder engagement mechanisms to identify areas of alignment and opportunities for enhancement.

Mapping an organization's data practices to HDI principles begins with a comprehensive review of current policies, procedures, and technical systems related to data management. This review should consider factors such as the organization's data collection methods, storage and processing practices, data sharing and disclosure protocols, and the mechanisms in place for user control and transparency. By conducting a thorough alignment analysis, organizations can gain a clear understanding of their current state and develop a roadmap to effectively implement the HDI principles.



 §.§ Audit Existing Practices

The initiation of a HDI framework within an organization necessitates a foundational step: a meticulous audit of existing data handling practices. This critical examination serves not merely as a procedural checkpoint but as a profound inquiry into the organization's data lifecycle management, from inception to obsolescence.

By undertaking this audit, an organization can delineate a comprehensive baseline from which to enhance and refine its data stewardship, guided by the tenets of HDI.


  * 

Comprehensive Data Mapping: The audit commences with an exhaustive mapping of the data lifecycle, illustrating the flow of data from its collection points through to its final disposition. This visual documentation encompasses identifying all data ingress points, such as sensors, web interfaces, and third-party integrations, and tracing the data's journey across storage, processing, internal sharing, and external disclosures. Humayun et al.<cit.> underscore the importance of understanding these data flows, highlighting how such mapping can unveil the intricacies and potential vulnerabilities within an organization's data ecosystem.
  * 

Policies and Procedures Analysis: A thorough review of the existing data governance frameworks is imperative. This includes scrutinizing privacy policies, data handling procedures, and terms of service to ascertain their alignment with HDI principles such as informed consent, data minimization, and user rights<cit.>. The clarity, accessibility, and comprehensiveness of these documents are evaluated to ensure they serve not only as legal compliance mechanisms but also as transparent guides for users.
  * 

Security Measures Evaluation: Assessing the robustness of data security measures forms a critical part of the audit. Organizations must evaluate their encryption protocols, access control mechanisms, and incident response strategies to identify vulnerabilities<cit.>. This assessment helps in pinpointing areas where enhancements are necessary to fortify the organization against data breaches and unauthorized access.
  * 

Data-Driven Decision-Making Scrutiny: The use of data in decision-making processes, particularly when involving algorithms or machine learning models, requires rigorous examination. The audit should investigate these models for biases and their explainability, ensuring that decision-making processes are transparent and equitable<cit.>. This scrutiny aims to guarantee that human oversight is effectively integrated, allowing for intervention and accountability.
  * 

User Interface Inspection: Finally, the interfaces through which users interact with the organization's data systems warrant careful evaluation. This involves assessing the clarity and efficacy of consent mechanisms, the ease with which users can access or amend their data, and the transparency surrounding the usage of their data <cit.>


This auditing process, necessitating a collaborative effort among diverse organizational teams, embodies a multidisciplinary approach. By integrating insights from technical staff, legal experts, and user experience designers, organizations can gain a holistic understanding of their current data handling practices.<cit.>

Such a comprehensive audit not only illuminates existing strengths and weaknesses but also paves the way for the development of a nuanced HDI framework. Through targeted interventions, grounded in the insights garnered from the audit, organizations can align their data practices with core values and goals, fostering a culture of ethical data stewardship and enhancing trust among users.

Having examined these crucial preliminary considerations, organizations can lay a solid foundation for implementing a comprehensive HDI framework. These initial steps provide the necessary context and groundwork for a more detailed and structured approach to integrating HDI principles throughout an organization's data practices. 

Building upon this preparatory phase, we now turn to a detailed exploration of the HDI Framework Schema. This schema offers a comprehensive, process-based model that operationalizes HDI principles, providing organizations with a practical guide to embed ethical data practices into their core operations. By progressing from preliminary considerations to this structured framework, organizations can systematically transform their approach to data management, ensuring that human values and ethical considerations are central to all data-related activities.




§ HDI FRAMEWORK SCHEMA

The proposed HDI framework is a comprehensive process-based model designed to address the complex challenges of data management and utilization. Building upon the foundational, this framework expands and operationalizes the HDI principles to provide a practical guide for organizations, policymakers, and researchers.

The framework is structured around nine key components, emphasizing its process-oriented nature:


    
  * Foundational Principles
    
    
  * Agency: Empowering individuals with control over their data.
    
  * Legibility: Making data processes transparent and understandable.
    
  * Equity Ensuring fair and unbiased data practices.
    
  * Accountability: Clearly defining responsibility for data use.
    
  * Privacy and Data Protection: Safeguarding personal information.
    
  * Participation and Engagement: Meaningful stakeholder involvement throughout the data lifecycle, from design to deployment.
    
  * Negotiability: Allowing flexible, context-based data permissions.
    
  * Human-Centricity: Prioritizing human needs in data systems.
    
  * Consent: Ensuring informed and meaningful consent for data collection and use.
    
    
    
  * Structural Components
    
    
  * Data Governance: Establishing policies for ethical data management.
    
  * Technological Infrastructure: Implementing tools supporting HDI principles.
    
  * Organizational Culture: Fostering an ethical data mindset.
    
  * Stakeholder Engagement: Involving users in data decisions.
    
  * Ethical Review Processes: Continuously assessing data practices.
    
  * Data Ethics Board: Establishing a dedicated group to oversee ethical data practices.
    
  * Data Quality Management: Ensuring data accuracy and relevance.
    

    
  * Ethical Considerations
    
        
  * Respect for Individual Autonomy: Honoring individual choices.
        
  * Equity and Justice: Promoting fairness in data practices.
        
  * Transparency: Ensuring openness in data processes.
        
  * Responsible Innovation: Ethically advancing data technologies.
        
  * Environmental Impact: Considering sustainability in data management.
        
  * Bias Mitigation: Addressing prejudices in data and algorithms.
        
  * Ethical AI Use: Aligning AI with HDI principles.
        
  * Cultural Sensitivity: Considering diverse cultural perspectives in data practices.
    
    
    
  * Practical Considerations
    
        
  * System Integration: Incorporating HDI into existing infrastructures.
        
  *  Technological Feasibility: Assessing implementability of HDI solutions.
        
  * Scalability: Ensuring HDI practices can grow with the organization.
        
  * Regulatory Compliance: Meeting legal data requirements.
       
  *  Organizational Capacity: Evaluating readiness for HDI adoption.
        
  * Education and Training: Preparing staff for HDI implementation.
        
  * Cost-Benefit Analysis: Assessing financial implications of HDI.
        
  * Risk Management: Identifying and mitigating data-related risks.
        
  * Interoperability: Ensuring HDI practices work across different systems and platforms.
    
    
    
    
  * Implementation Roadmap 
    
        
  * Assessment:  Evaluating current data practices against HDI principles. 
        
  * Design:  Developing HDI-aligned systems and processes.
        
  * Integration:  Implementing HDI practices across the organization.
        
  * Evaluation: Continuous monitoring and improvement of HDI initiatives.
        
  * Adaptation: Updating practices for emerging challenges.
        
  * Pilot Testing: Implementing HDI practices in small-scale projects before full deployment.
    
    
    
  * Enhancing HDI Culture
    
        
  * Employee Training Programs: Educating staff on HDI principles.
        
  * Stakeholder Education: Broadening the scope to include all affected parties, not just direct users.
        
  * Continuous Learning and Adaptation: Updating training based on new trends.
        
  * Cross-functional Collaboration: Fostering interdepartmental HDI understanding.
    
    
    
  * Data Lifecycle Management
    
        
  * Collection: Gathering data ethically and transparently.
        
  * Processing: Analyzing data fairly and accountably.
        
  * Storage: Retaining data securely and responsibly.
        
  * Utilization: Using data ethically in decision-making.
        
  * Sharing: Exchanging data with consent and control.
        
  * Disposal: Deleting data properly and timely.
    

    
  * Monitoring and Adaptation
    
        
  * Continuous Assessment: Regularly evaluating HDI practices.
        
  * Feedback Mechanisms: Gathering stakeholder input on data practices.
        
  * Adaptive Governance: Evolving policies with changing needs.
        
  * Emerging Technology Integration: Incorporating new HDI-enhancing tools.
        
  * Performance Metrics: Tracking HDI-specific key performance indicators.
       
  *  External Audits: Conducting independent assessments of HDI practices.
    
        
    
  * Complex and dynamic considerations
    
        
  * Complexity: Acknowledging the intricate nature of data systems and the need for nuanced, context-specific HDI implementations.
        
  * Evolving Technologies: Addressing the continual emergence of new data technologies and their implications for HDI.
        
  * Regulatory Landscape: Keeping pace with changing data protection laws and regulations across different jurisdictions.
        
  * Societal Shifts: Recognizing changing public attitudes towards data privacy and use.
        
  * Ethical Dilemmas: Preparing for unforeseen ethical challenges that may arise with new data uses.
        
  * Cross-border Data Flows: Managing HDI principles in the context of global data transfer and varying international standards.
        
  * Balancing Innovation and Ethics: Navigating the tension between rapid technological advancement and ethical considerations.
        
  * Adaptive Strategies: Developing flexible approaches that can evolve with changing data ecosystems.
    
    


This process-based framework is designed to be iterative and adaptable across various sectors and organizational sizes. It recognizes the dynamic nature of the data landscape, as highlighted by <cit.>, and provides flexibility to address emerging challenges.

The framework incorporates insights from recent research on algorithmic accountability <cit.> and privacy economics <cit.>, ensuring a holistic approach to data ethics and management. It emphasizes the importance of continuous learning and adaptation, acknowledging that HDI implementation is an ongoing process rather than a one-time initiative.

By integrating these components into a cohesive, process-oriented approach, the proposed HDI framework offers a roadmap for organizations to navigate the complexities of data ethics, privacy, and user empowerment. It aims to foster a data ecosystem that not only drives innovation and efficiency but also respects human values and promotes societal well-being.

This framework represents a significant step towards realizing the vision of a human-centric data society, where technological advancements and ethical considerations evolve in tandem, as advocated <cit.> in their work on AI ethics. Its process-based nature ensures that organizations can continually refine and improve their HDI practices in response to new challenges and opportunities in the ever-evolving data landscape.



§ FOUNDATIONAL PRINCIPLES

The HDI Framework presents a comprehensive and multidimensional approach to managing and utilizing data in a manner that prioritizes ethical considerations, upholds fundamental human rights, and promotes the overall well-being of society. This holistic framework guides organizations in cultivating a data ecosystem that deeply respects individual autonomy, empowering users with meaningful control over their personal information, including its collection, usage, sharing, and deletion.<cit.>

The framework emphasizes the critical importance of transparency, requiring clear and comprehensible explanations regarding data handling practices, algorithmic decision-making processes, and the security protocols in place to safeguard data integrity<cit.>. Ensuring fairness and non-discrimination is a core tenet, demanding proactive efforts to identify and mitigate biases within datasets and algorithmic systems, thereby promoting equitable outcomes for all individuals. Accountability is woven throughout the framework, with mechanisms established to address instances of data misuse, facilitate redress for affected parties, and demonstrate organizational responsibility. 

Additionally, the framework underscores the value of meaningful stakeholder engagement, advocating for the inclusion of diverse perspectives and the co-creation of ethical data practices with users and affected communities. By outlining these guiding principles, structural components, implementation strategies, and multifaceted considerations<cit.>, the HDI Framework provides a comprehensive blueprint for organizations to embed ethical practice and user-centric values at the very core of their data management and data-driven decision-making processes.




 §.§ Agency

Human agency and autonomy are central to the HDI framework, emphasizing the empowerment of individuals in their interactions with data-driven systems. This principle advocates for giving individuals control over their data, including its collection, use, sharing, and deletion. It underscores the importance of enabling people to make informed decisions about their data, reflecting a commitment to respecting personal autonomy and privacy <cit.>

Individuals have the right to exercise control over their personal data, including the right to access, modify, and delete their information, as well as to make informed decisions about data collection and use. Respecting autonomy acknowledges individuals as active agents with the capacity to make choices about their data. This principle is fundamental to ensuring that data practices empower rather than undermine user agency.<cit.>




 §.§ Legibility

Transparency and explainability within HDI demand clear, understandable information about data practices, the workings of algorithms, and the security measures in place to protect data<cit.>. This principle is crucial for building trust and accountability, allowing users to grasp how their data is processed and how decisions that affect them are made. Explainable AI (XAI) systems play a pivotal role in demystifying algorithmic processes, making technology accessible and comprehensible to non-experts (Ribeiro, Singh, And Guestrin, 2016).<cit.>.

Maintain openness about data practices, including the collection, use, and sharing of data, and implement mechanisms for accountability in decision-making processes. Transparency and accountability are vital for building trust, facilitating oversight, and enabling individuals to understand and challenge decisions made with or about their data <cit.>




 §.§ Equity
 
Fairness and non-discrimination call for proactive efforts to detect and mitigate biases in datasets and algorithms, ensuring equitable outcomes for all users<cit.>. This principle recognizes the potential for data-driven systems to perpetuate or even exacerbate existing inequalities. Research in algorithmic fairness seeks to develop methodologies and tools to identify and correct biases, promoting fairness in automated decision-making processes <cit.>

Ensure equitable access to the benefits of data technologies and protect against discriminatory practices and biases that may arise from data collection, analysis, and application. Addressing justice and equity is crucial for mitigating the risk of exacerbating existing inequalities and ensuring that the benefits of data technologies are shared across all sectors of society <cit.>




 §.§ Accountability

Accountability and redress in the HDI framework ensure that mechanisms are in place to address any misuse of data, provide remedies to affected individuals, and demonstrate organizational responsibility<cit.>. This principle involves establishing clear policies and procedures for data governance, including how data breaches and privacy violations are handled. It also encompasses the need for regulatory compliance and the ability to audit and review data practices to uphold ethical standards <cit.>

Data practices should aim to benefit individuals and communities while minimizing harm and preventing misuse of data that could lead to adverse outcomes. The dual obligation to do good and avoid harm is a cornerstone of ethical practice, emphasizing the responsibility of data handlers to consider the broader impact of their actions on society.<cit.>




 §.§ Privacy and Data Protection

Privacy and data protection are foundational to HDI, safeguarding individuals' rights to control their personal information and ensuring that data is handled securely and respectfully<cit.>. This principle is aligned with global privacy regulations, such as the General Data Protection Regulation (GDPR) in the European Union, which set standards for data collection, processing, and storage. Implementing privacy-by-design approaches and privacy-enhancing technologies (PETs) are practical steps towards operationalizing this principle.

Uphold individuals' rights to privacy and data protection, implementing secure and responsible data handling practices to safeguard personal information from unauthorized access and exploitation.

Privacy is essential for personal security, freedom of expression, and protection against surveillance and control, making it a fundamental aspect of ethical data practices.




 §.§ Participation and Engagement

Participation and engagement advocate for the inclusion of stakeholders, including data subjects, in the design and governance of data-driven systems<cit.>. This principle emphasizes the value of diverse perspectives and the importance of co-creating ethical data practices with users and affected communities. It supports the idea of democratizing data, where individuals and communities have a say in how data about them is used and for what purposes.

Foster inclusive and participatory approaches that engage stakeholders, particularly those most affected by data practices, in the governance, design, and implementation of data technologies. Participation and empowerment ensure that diverse perspectives are considered, supporting democratic values and enhancing the relevance and acceptability of data initiatives. Engaging with users and impacted communities helps to ensure that data practices are responsive to their needs and concerns, promoting transparency, trust, and shared responsibility <cit.>

Implementing these foundational principles requires a multidisciplinary approach, combining technical solutions with ethical, legal, and social insights. The HDI framework aims to foster a data ecosystem that respects individual rights and promotes social good, ensuring that technology serves humanity in equitable and responsible ways.



 §.§ Negotiability

The principle of negotiability in HDI recognizes that the context and meaning of data can change over time and across different situations. It emphasizes the need for flexible, dynamic approaches to data permissions and use. Negotiability allows for ongoing adjustments to data sharing preferences, acknowledging that user priorities and circumstances may evolve <cit.>.
Implementing negotiability involves:



  * Developing mechanisms for users to revise their data sharing preferences over time.

  * Creating systems that allow for context-dependent data permissions.

  * Establishing protocols for renegotiating data use agreements as new uses or insights emerge.


Mortier et al. argue that negotiability is crucial for addressing the dynamic nature of privacy preferences and the evolving relationships between individuals and data-driven services <cit.>. This principle aligns with the concept of "contextual integrity" proposed by Nissenbaum, which posits that the appropriateness of information flows depends on context-specific norms <cit.>.



 §.§ Human-Centricity

Human-centricity places human needs, values, and experiences at the core of data system design and operation. This principle ensures that technological advancements serve human interests rather than subordinating human agency to algorithmic imperatives <cit.>.

Key aspects of human-centricity in HDI include:



  * Prioritizing user experience and accessibility in data interfaces.

  * Considering the psychological and social impacts of data practices on individuals and communities.

  * Ensuring that data-driven systems augment rather than replace human decision-making capacities.


Shneiderman advocates for a human-centered approach to AI and data systems, emphasizing the importance of keeping humans "in the loop" and designing systems that enhance human capabilities <cit.>. This aligns with the broader movement towards "human-centered AI" that seeks to develop technologies that are beneficial, comprehensible, and controllable by humans <cit.>.



 §.§ Consent

Informed consent is a foundational principle in HDI, rooted in respect for individual autonomy and self-determination. It requires that individuals be provided with clear, understandable information about data collection, use, and potential implications before agreeing to share their data <cit.>.
Effective implementation of consent in HDI frameworks involves:



  * Developing clear, accessible consent mechanisms that go beyond mere legal compliance.

  * Providing granular options for consent, allowing users to selectively agree to specific data uses.

  * Ensuring that consent is ongoing and revocable, with easy-to-use tools for withdrawing or modifying consent.

  * Educating users about the implications of their consent decisions.


Recent research has highlighted the limitations of traditional notice and consent models in the big data era, leading to proposals for more dynamic and user-centric approaches to consent <cit.>. These new models aim to address the challenges of informed consent in complex, rapidly evolving data ecosystems.

The foundational principles of Human-Data Interaction (HDI) outlined above—agency, legibility, equity, accountability, privacy, participation, negotiability, human-centricity, and consent—provide the ethical and conceptual framework upon which robust HDI practices are built. However, translating these principles into practical, operational realities within organizations requires a set of concrete structural components. 

These components serve as the architectural elements that enable the realization of HDI principles in real-world data ecosystems. As we transition from the theoretical underpinnings to the practical implementation of HDI, it becomes crucial to examine how these foundational principles can be embedded into organizational structures, technological systems, and governance frameworks. 

The following section on Structural Components explores the tangible mechanisms and processes that organizations can adopt to operationalize HDI principles, ensuring that ethical considerations are not merely aspirational but are deeply integrated into the fabric of data management and utilization practices. By bridging the conceptual with the practical, we can begin to envision how HDI can transform from a set of ideals into a lived reality within complex organizational environments.




§ STRUCTURAL COMPONENTS

The structural components of the HDI framework provide a tangible architecture for embedding ethical principles into the practical operations of data-driven systems. These components are critical for translating ethical commitments into actionable strategies and processes that can be systematically applied within organizations. This section outlines the key structural elements essential for establishing a robust HDI practice, drawing on interdisciplinary research to support their implementation.



 §.§ Data Governance

A comprehensive data governance is essential for ensuring that data management practices align with ethical principles and regulatory requirements. This framework encompasses policies, standards, and procedures that guide how data is collected, stored, accessed, and used within an organization.


  * 

Policies and Protocols: Develop and implement clear policies that cover informed consent, data minimization, purpose limitation, data quality, retention, and deletion. These policies should also address security measures and incident response plans to protect data integrity and confidentiality.<cit.>
  * 

Audit and Compliance: Establish regular audit processes to evaluate the alignment of data practices with HDI principles, legal standards, and evolving societal expectations<cit.>. Compliance checks should ensure adherence to internal policies and external regulations, facilitating accountability and continuous improvement <cit.>



  * 

Roles and Responsibilities: Define clear roles and responsibilities for data stewardship, including data owners, data stewards, and data governance committees. These roles should be empowered to make decisions, enforce policies, and address issues that arise in the course of data management.<cit.>





 §.§ Technological Infrastructure

Technological tools and systems play a crucial role in facilitating ethical data practices. Using technology, organizations can improve transparency, protect privacy, and empower users.


  * 

Consent Management Platforms (CMPs): Deploy CMPs to streamline the process of obtaining and managing user consent for data collection and processing, ensuring that consent is informed, specific, and revocable <cit.>
  * 

Data Visualization Tools: Utilize data visualization tools to make data practices more transparent and comprehensible to non-expert stakeholders, aiding in the communication of complex information.<cit.>
  * 

Explainable AI (XAI) Systems: Implement XAI systems to provide clear explanations of algorithmic decisions, fostering transparency and trust in automated processes<cit.>.
  * 

Privacy-Enhancing Technologies (PETs): Incorporate PETs to minimize personal data exposure and enhance data security, supporting privacy by design and by default principles<cit.>.





 §.§ Organizational Culture

At the heart of ethical HDI are practices that prioritize human values and well-being. This involves training, stakeholder engagement, and the creation of channels for feedback and participation.


  * 

Training and Development: Implement comprehensive training programs for staff that cover data ethics, privacy, security, and bias mitigation. This training should be tailored to different roles within the organization and updated regularly to reflect new developments and insights <cit.>
  * 

 User Education and Empowerment: Develop initiatives to educate users and affected communities about their data rights and how to exercise them. This includes providing accessible resources and support for understanding data practices and the implications of data decisions.<cit.>




 §.§ Stakeholder Engagement:
 Create systems for gathering meaningful input and feedback from a diverse range of stakeholders, such as employees, users, and external communities. This involvement should guide the development of policies, the selection of technologies, and the implementation of operational practices, ensuring a variety of perspectives are considered <cit.>. 

Additionally, engage a wide array of stakeholders, including data subjects, employees, regulatory bodies, and civil society organizations, to understand their concerns, expectations, and viewpoints on data usage and ethical matters <cit.>. This inclusive approach ensures that different viewpoints are integrated into the HDI framework.



 §.§ Data Quality Management

The structural components of the HDI framework offer a comprehensive blueprint for organizations to navigate the complex ethical challenges inherent in data interaction. By deeply integrating these core elements into their day-to-day operations, organizations can foster a strong culture of responsibility, accountability, and transparency <cit.>. 

This, in turn, enhances trust and credibility with a diverse range of stakeholders, from employees to customers to affected communities. Ultimately, the implementation of this ethical HDI framework can enable organizations to make meaningful contributions towards building a more equitable, inclusive, and socially just digital ecosystem that prioritizes human values and well-being.

The implementation of HDR frameworks is a complex undertaking that requires meticulous attention to both ethical and practical considerations. As organisations endeavour to harmonise their data practices with the tenets of HDR, they confront a multitude of challenges and opportunities. The following section delineates pivotal ethical and practical considerations to assist organisations in the responsible integration of HDR frameworks, drawing upon interdisciplinary research to inform these concepts.




§ ETHICAL CONSIDERATIONS



  §.§.§ Respect for Individual Autonomy

Ensuring that individuals have meaningful control over their data requires robust mechanisms for informed consent and ongoing consent management. Organizations must carefully balance the need for data utility with respect for individual choice and privacy.<cit.> This involves developing dynamic consent platforms that empower users to adjust their privacy preferences as the contexts and applications of their data evolve over time.

These consent platforms should provide users with clear and accessible information about how their data will be collected, used, and shared. They should also offer granular controls that allow individuals to selectively authorize or revoke access to specific data types or use cases, rather than relying on a one-size-fits-all approach.<cit.>

By giving users the ability to monitor and manage their data preferences on an ongoing basis, organizations can foster a stronger sense of trust and transparency, while still preserving the valuable insights that can be derived from aggregated data. This balanced approach is crucial for upholding individual autonomy while also enabling the responsible and ethical use of data to drive innovation and societal progress.


  §.§.§ Equity and Justice


Data practices can inadvertently reinforce or exacerbate existing social, economic, and political inequalities. It is crucial for organizations to proactively identify and mitigate various types of biases that may be present in their data collection, analysis, and application processes <cit.>

 These biases can manifest in the form of underrepresentation of marginalized groups, flawed algorithms that amplify historical discrimination, and the exclusion of diverse perspectives during the development of data-driven systems. To address these challenges, organizations should adopt inclusive design and engagement practices that actively involve a wide range of stakeholders, including underrepresented and marginalized communities, in the development, testing, and ongoing evaluation of their data systems. 

This collaborative approach can help uncover and address blind spots, ensure that the needs and concerns of diverse users are prioritized, and ultimately prevent the perpetuation of harm to vulnerable populations. By embedding equity and justice as core principles throughout the data lifecycle, organizations can work towards building more inclusive, fair, and responsible data practices that benefit all members of society.


  §.§.§ Transparency and Accountability


Achieving transparency in complex data systems is undoubtedly challenging, but it is essential for ensuring meaningful accountability. Stakeholders, including employees, users, and the broader public, should have access to comprehensive, understandable information about how their data is collected, used, and shared by organizations<cit.>. To address this critical need, organizations must develop robust, user-friendly data policies and communication channels that clearly and transparently inform individuals about the full scope of their data practices. 

This includes providing detailed explanations of the specific purposes for which data is collected and used, as well as the concrete measures and safeguards in place to ensure responsible data management and ethical stewardship of sensitive information<cit.>.

Embracing a high degree of transparency allows organizations to build stronger trust relationships with their stakeholders, empowering individuals to make well-informed choices regarding the use of their personal data. This demonstrates the organization's commitment to upholding data privacy principles and ethical data governance practices.


  §.§.§ Responsible Innovation


While data-driven technologies offer significant potential for societal benefit, their development and deployment must be guided by strong ethical principles<cit.>. Establishing comprehensive ethical review processes and frameworks is crucial for enabling responsible innovation. These frameworks should thoroughly assess the far-reaching implications of new data-driven innovations, carefully considering their impact on human and societal well-being <cit.>

Furthermore, developing and deploying explainable AI tools and practices is essential to ensure transparency and accountability in the application of these transformative technologies. 

Enhancing the accessibility and comprehensibility of algorithmic decision-making processes for nonexpert stakeholders can enable organizations to foster increased trust and facilitate meaningful engagement with these systems. This level of openness and understanding is vital for upholding the principles of ethical data governance and ensuring that the benefits of data-driven innovation are equitably distributed throughout society <cit.>

The responsible development and application of data-driven technologies must be based on a strong commitment to the protection of ethical principles, the protection of individual rights and the promotion of greater social well-being<cit.>. Only through this holistic, principle-driven approach can organizations harness the immense potential of data-driven innovation while mitigating the risks and unintended consequences that may arise.


  §.§.§ Environmental and Social Impact 


Responsible data management requires organizations to adopt a holistic, sustainability-focused approach that considers the environmental and social impacts of their data production, storage, and processing activities. This includes carefully examining the energy consumption, emissions, and e-waste generated by their data infrastructure, as well as the broader societal implications of their data-driven systems <cit.>

Adopting a comprehensive perspective on the environmental and societal implications of their data-related activities enables organizations to formulate strategies that mitigate these impacts and foster more sustainable and ethically-grounded data management practices<cit.>. This holistic approach is essential to ensure that the advantages of data-driven innovation are balanced against the imperative to protect the planet and address the concerns of diverse stakeholder groups.




§ PRACTICAL CONSIDERATIONS



 §.§ Integration with Existing Systems


Integrating HDI principles into an organization's existing data infrastructure and practices requires careful and comprehensive planning to avoid potential disruptions and ensure seamless compatibility<cit.>. Conducting a thorough systems analysis is crucial to identify and address all potential integration challenges. 

This analysis should examine the organization's current data management workflows, technological capabilities, and existing policies and procedures. Based on the insights gained, a well-crafted, phased implementation plan can be developed, allowing for a gradual and strategic adaptation of HDI principles. This measured approach, with clearly defined milestones and feedback loops, enables the organization to mitigate risks, build organizational buy-in, and ensure a smooth transition that preserves data utility and operational continuity. 

Carefully examining the existing data environment and devising a measured, phased implementation plan enables the organization to successfully integrate HDI principles into its fundamental data practices without sacrificing efficiency or effectiveness.


  §.§.§ Technological Feasibility


Implementing HDI frameworks may necessitate the development or adoption of novel technologies that can effectively address the unique requirements and complexities of data management and interaction in the modern digital landscape. This may include the integration of privacy-preserving data management tools, which employ advanced cryptographic techniques and data anonymization methods to safeguard individual privacy while preserving the utility of data for authorized purposes<cit.>. Furthermore, the implementation of dynamic consent platforms can empower users with greater control and agency over the use of their personal data, allowing them to adjust their preferences as the contexts and applications of data evolve.

Evaluating the feasibility and scalability of these technological solutions is essential to ensure their successful integration within the organization's data infrastructure and practices. This assessment should consider factors such as technical capabilities, ease of implementation, compatibility with existing systems, and the ability to adapt to changing organizational and regulatory requirements over time.<cit.> 


  §.§.§ Scalability and Flexibility


Organizations must design HDI criteria that are highly scalable and adaptable, ensuring their long-term resilience and responsiveness to evolving technologies, regulatory environments, and organizational needs. This entails developing modular frameworks with interchangeable components that can be easily updated or replaced as required, without disrupting the overall system.<cit.>

By embracing a modular architecture, organizations can imbue their HDI frameworks with the capacity to adapt rapidly to evolving circumstances. This allows the organization, for instance, to update the pertinent data privacy management components in response to the introduction of new regulatory requirements, without the need to restructure the entire HDI system.

Similarly, as new data analytics tools or interaction techniques emerge, the modular design allows for the seamless integration of these innovations, enabling the organization to stay at the forefront of technological advancements.<cit.>

This scalable and adaptable architecture is crucial for safeguarding the organization's long-term viability and ensuring that its HDI practices remain aligned with evolving needs and best practices. It empowers the organization to proactively anticipate and respond to change, rather than being beholden to rigid, inflexible systems that quickly become obsolete. By prioritizing modularity and adaptability in the design of their HDI frameworks, organizations can future-proof their data practices and maintain a competitive edge in an ever-changing digital landscape.


  §.§.§ Education and Training


Successful integration of HDI principles necessitates a well-versed and empowered workforce capable of navigating the intricacies of contemporary data systems while upholding ethical data practices. Accordingly, organizations should develop comprehensive training programs for all employees, emphasizing the ethical considerations underlying their work and providing ongoing education on emerging data-related practices and technologies<cit.>. These training programs should not only impart technical knowledge but also foster a deep understanding of the ethical implications of data use, data privacy, and algorithmic decision-making.

Equipping employees with the necessary skillsets and ethical knowledge empowers them to make well-informed, values-aligned decisions when interacting with complex data systems. Additionally, these training initiatives should be accompanied by the establishment of clear organizational policies, accountability frameworks, and channels for continuous feedback and improvement <cit.>



§ IMPLEMENTATION ROADMAP

The implementation of a comprehensive Human-Data Interaction (HDI) framework within an organization represents a paradigm shift in data management practices. This roadmap delineates a structured, yet flexible approach to embedding HDI principles into the core of organizational operations. The process is inherently iterative and adaptive, recognizing the dynamic nature of both technological advances and ethical considerations in the digital age.



 §.§ Phase 1: Assessment and Planning




  §.§.§ Comprehensive Audit and Gap Analysis


The initial phase commences with a thorough, multidimensional audit of existing data practices. This audit goes beyond simple compliance checks, delving into the ethical implications of current data handling processes. Organizations must scrutinize their entire data ecosystem, from collection points to disposal mechanisms, evaluating each stage against HDI principles <cit.>.

Key components of this audit include:


    
  * Data flow mapping: Tracing the journey of data through organizational systems, identifying potential vulnerabilities and ethical pressure points.
    
  * Policy evaluation: Assessing the alignment of existing policies with HDI principles, paying particular attention to areas such as consent mechanisms, data minimization, and purpose limitation.
    
  * Technological infrastructure assessment: Evaluating current technological capabilities against the requirements of HDI implementation, including privacy-enhancing technologies and explainable AI systems.
    
  * Stakeholder impact analysis: Examining how current data practices affect various stakeholders, including employees, customers, and broader communities.


This comprehensive audit serves as the foundation for a detailed gap analysis, highlighting areas where current practices fall short of HDI standards and identifying opportunities for improvement.



  §.§.§ Stakeholder Engagement and Participatory Design


Concurrent with the audit process, organizations must initiate a robust stakeholder engagement program. This goes beyond traditional consultation models, embracing a participatory design approach that actively involves stakeholders in shaping HDI implementation strategies.

Key stakeholder engagement activities include:


    
  * Multi-stakeholder workshops: Facilitating interactive sessions that bring together diverse perspectives, including data subjects, employees, regulatory bodies, and civil society organizations.
    
  * Ethical deliberation forums: Creating spaces for in-depth discussions on the ethical implications of data practices, encouraging stakeholders to grapple with complex trade-offs and scenarios.
    
  * User experience research: Conducting detailed studies to understand how different stakeholder groups interact with and perceive data systems, informing the design of more human-centric interfaces and processes.


This participatory approach not only enriches the implementation strategy, but also fosters a sense of shared ownership and commitment to HDI principles across the stakeholder ecosystem.



  §.§.§ Strategic Alignment and Roadmap Development


Building on the insights from the audit and stakeholder engagement, organizations must align their strategic objectives with HDI principles. This process involves:


    
  * Vision articulation: Crafting a clear, compelling vision for how HDI will transform the organization's approach to data management and utilization.
    
  * Goal setting: Establishing specific, measurable, achievable, relevant, and time-bound (SMART) goals for HDI implementation.
    
  * Resource allocation: Identifying and securing the necessary resources, including budget, personnel, and technological infrastructure, to support HDI implementation.
    
  * Phased implementation planning: Developing a detailed, phased plan for rolling out HDI initiatives, prioritizing high-impact areas while managing organizational change effectively.


This strategic alignment process ensures that HDI implementation is not treated as a peripheral initiative but is integrated into the core of organizational strategy and operations <cit.>.



 §.§ Phase 2: Policy Development and Technological Integration

The second phase of HDI implementation marks a critical transition from conceptual understanding to practical application. This phase focuses on translating the HDI principles into tangible organizational policies and technological solutions. It represents a pivotal moment where abstract ethical considerations are transformed into concrete operational practices.

Policy development in this context goes beyond mere regulatory compliance, with the aim of integrating HDI principles into the fabric of organizational decision making. Simultaneously, technological integration involves not just the adoption of new tools, but a fundamental reimagining of data systems to support human-centric interactions.

This phase demands a delicate balance between ethical imperatives and operational realities, requiring unprecedented collaboration across diverse organizational functions. It requires a flexible approach that can adapt to the rapidly evolving digital landscape while maintaining a steadfast commitment to HDI principles <cit.>.

Success in this phase sets the foundation for a transformative approach to data management, positioning organizations at the forefront of ethical, human-centered data practices in the digital age.



  §.§.§ Comprehensive Policy Reformation


The second phase focuses on translating HDI principles into concrete policies and procedures. This process goes beyond simply updating existing policies, often requiring a fundamental reimagining of the organization's approach to data governance.

Key aspects of policy development include:


    
  * Ethical framework establishment: Creating a clear, actionable ethical framework that guides all data-related decision-making within the organization.
    
  * Data lifecycle policies: Developing comprehensive policies that address each stage of the data lifecycle, from collection to disposal, ensuring HDI principles are embedded throughout.
    
  * Consent and control mechanisms: Designing policies that empower individuals with meaningful control over their data, including granular consent options and easy-to-use data management interfaces.
    
  * Accountability structures: Establishing clear lines of responsibility and accountability for HDI implementation, including the creation of new roles or committees dedicated to ethical data oversight.


These policies should be developed through a collaborative process, involving legal experts, ethicists, technologists, and representatives of various organizational departments to ensure comprehensive coverage and practical applicability <cit.>.



  §.§.§ Technological Infrastructure Enhancement


Concurrent with policy development, organizations must undertake a significant enhancement of their technological infrastructure to support HDI principles. This involves:


    
  * Privacy-Enhancing Technologies (PETs): Implementing advanced PETs, such as differential privacy, secure multi-party computation, and homomorphic encryption, to protect individual privacy while maintaining data utility.
    
  * Consent Management Platforms: Developing or adopting sophisticated consent management systems that allow for dynamic, context-aware consent processes.
    
  * Explainable AI Systems: Integrating explainable AI technologies to enhance the transparency and interpretability of algorithmic decision-making processes.
    
  * Data Provenance and Auditing Tools: Implementing robust systems for tracking data lineage and enabling comprehensive audits of data usage and transformations.


The selection and implementation of these technologies should be guided by a thorough assessment of their alignment with HDI principles and their potential impact on stakeholder experiences.



  §.§.§ Organizational Capacity Building


To support the successful implementation of new policies and technologies, organizations must invest heavily in building internal capacity for HDI. This includes:


    
  * Comprehensive Training Programs: Developing and delivering tailored training programs for different organizational roles, covering both technical aspects of HDI implementation and ethical decision-making in data-driven contexts.
    
  * Change Management Initiatives: Implementing structured change management processes to facilitate the cultural and operational shifts required for HDI adoption.
    
  * Skills Development: Investing in upskilling and reskilling programs to ensure staff have the necessary competencies to operate effectively within an HDI framework.
    
  * Cross-functional Collaboration: Fostering environments and processes that encourage collaboration between different organizational functions, breaking down silos that may impede HDI implementation.


These capacity-building efforts are crucial for ensuring that HDI principles are not just understood theoretically but are actively applied in day-to-day operations across the organization <cit.>.



 §.§ Phase 3: Implementation and Evaluation




  §.§.§ Phased Rollout and Iterative Refinement


The implementation phase should follow a carefully structured, phased approach:


    
  * Pilot Programs: Initiating small-scale pilot programs in selected departments or processes to test HDI implementations in controlled environments.
    
  * Feedback Loops: Establishing robust mechanisms for gathering and analyzing feedback from stakeholders involved in pilot programs.
    
  * Iterative Refinement: Using insights from pilot programs to refine policies, technologies, and processes before broader rollout.
    
  * Scaled Implementation: Gradually expanding HDI implementations across the organization, maintaining flexibility to adapt to emerging challenges and opportunities.


This phased approach allows organizations to learn from their first experiences and make the necessary adjustments, increasing the likelihood of a successful adoption throughout the organization.



  §.§.§ Continuous Monitoring and Adaptive Governance


As the principles of HDI become more widely adopted, organizations need to set up thorough monitoring and governance frameworks:


    
  * Real-time Monitoring Systems: Implementing technologies and processes for continuous monitoring of HDI compliance and effectiveness across all data operations.
    
  * Stakeholder Feedback Mechanisms: Creating multiple channels for ongoing stakeholder input, including regular surveys, focus groups, and open feedback platforms.
    
  * Ethical Review Boards: Establishing cross-functional ethical review boards to assess complex cases and guide policy evolution.
    
  * Adaptive Governance Frameworks: Developing flexible governance structures that can evolve in response to new ethical challenges, technological advancements, and regulatory changes.


These monitoring and governance mechanisms ensure that HDI implementation remains responsive to changing contexts and emerging ethical considerations.



  §.§.§ Impact Assessment and Reporting


Regular, comprehensive impact assessments are crucial to understanding the effectiveness of HDI implementations and identifying areas for improvement.


    
  * Multidimensional Impact Metrics: Developing a set of metrics that capture the multifaceted impacts of HDI implementation, including ethical, social, economic, and operational dimensions.
    
  * Longitudinal Studies: Conducting long-term studies to track the evolving impacts of HDI practices on various stakeholder groups and organizational outcomes.
    
  * Transparent Reporting: Producing regular, detailed reports on HDI implementation progress, challenges, and impacts, making these accessible to all stakeholders.
    
  * External Audits: Engaging independent third parties to conduct periodic audits of HDI practices, ensuring objectivity and credibility in evaluations.


These assessment and reporting practices not only drive continuous improvement but also contribute to building trust and accountability with stakeholders.



 §.§ Phase 4: Adaptation and Evolution




  §.§.§ Horizon Scanning and Proactive Adaptation


To remain effective in a rapidly evolving digital landscape, organizations must develop robust capabilities to anticipate and respond to emerging trends and challenges.


    
  * Technology Foresight: Establishing dedicated teams or processes for monitoring and analyzing emerging technologies and their potential ethical implications.
    
  * Regulatory Tracking: Developing systems for staying abreast of evolving data protection regulations and proactively adapting HDI practices to ensure compliance.
    
  * Ethical Foresight: Engaging in ongoing ethical deliberation to anticipate future ethical challenges arising from new data uses or societal changes.
    
  * Collaborative Innovation: Participating in industry collaborations, academic partnerships, and multi-stakeholder initiatives to collectively address emerging HDI challenges.


This proactive stance enables organizations to stay ahead of the curve, adapting their HDI practices to address new challenges before they become critical issues.



  §.§.§ Continuous Learning and Knowledge Management


Building a robust knowledge management system around HDI practices is essential for long-term success:


    
  * Case Study Development: Documenting and analyzing specific cases of HDI implementation, including successes, challenges, and lessons learned.
    
  * Best Practice Repositories: Creating and maintaining accessible repositories of HDI best practices, updated regularly with new insights and experiences.
    
  * Cross-organizational Learning: Facilitating knowledge sharing across different parts of the organization to ensure consistent application of HDI principles.
    
  * External Knowledge Exchange: Engaging in knowledge exchange with other organizations, industry bodies, and academic institutions to broaden the collective understanding of effective HDI practices.


These learning and knowledge management practices ensure that the organization continuously builds on its HDI capabilities, fostering a culture of ongoing improvement and innovation.



  §.§.§ Long-term Strategic Integration


Finally, organizations must work towards fully integrating HDI principles into their long-term strategic planning:


    
  * Strategic Visioning: Regularly revisiting and refining the organization's long-term vision for ethical data management, ensuring it remains aligned with evolving HDI principles and societal expectations.
    
  * Innovation Alignment: Ensuring that all data-driven innovation initiatives are evaluated through an HDI lens from their inception, making ethical considerations a fundamental part of the innovation process.
    
  * Organizational Culture Evolution: Continuously working to embed HDI principles into the core values and culture of the organization, making ethical data practices an intrinsic part of organizational identity.
    
  * Stakeholder Relationship Evolution: Developing more collaborative, trust-based relationships with stakeholders, positioning the organization as a leader in ethical data practices.


This long-term strategic integration ensures that HDI is not treated as a separate initiative, but becomes an integral part of how the organization operates and creates value in the digital age.

By following this comprehensive and in-depth roadmap, organizations can embark on a transformative journey towards ethical, human-centered data practices. This approach not only ensures compliance with evolving regulations but positions organizations at the forefront of responsible innovation, building trust and creating sustainable value in an increasingly data-driven world.




§ ENHANCING HDI CULTURE

For an organization to truly embody HDI principles, a deep-rooted understanding of these concepts among all members is vital. Regular training sessions should be conducted to educate employees about the ethics of data use, responsible data handling, and the potential risks associated with improper data management. Furthermore, it's important to extend educational efforts to stakeholders, including users and affected communities, thereby increasing their data literacy. This empowers them to participate actively and effectively in data governance processes, ensuring their voices are heard and considered in decision-making.

However, cultivating an HDI-centric culture within an organization is not a one-time achievement but a continuous process. It requires the establishment of mechanisms for ongoing monitoring, feedback, and iterative improvements. Such processes are crucial for maintaining the effectiveness of HDI practices amidst the rapidly changing dynamics of data use and technology <cit.>

By committing to constant monitoring and adaptation, an organization showcases its dedication to continuously enhancing its HDI practices. This proactive approach not only fosters trust among stakeholders but also positions the organization as a frontrunner in responsible and ethical data management. As the digital environment evolves, staying ahead in implementing robust HDI frameworks becomes a testament to an organization's commitment to ethical data use and governance.

Training and education form the cornerstone of implementing HDI frameworks effectively. This comprehensive approach is key to building a culture that values ethical data use, adheres to HDI principles, and equips both employees and users with the necessary skills and knowledge to navigate the complexities of contemporary data ecosystems. The outlined strategies underscore the importance of developing targeted training and education programs that are integral to fostering an environment where HDI principles are not just understood but actively practiced.



 §.§ Employee Training Programs

Equip employees with a deep understanding of HDI principles, including data ethics, responsible data handling, and awareness of biases. To further empower individuals as active stakeholders in the data ecosystem, the HDI Framework calls for proactive user education initiatives <cit.>. This training should extend beyond theoretical knowledge, emphasizing practical skills for implementing HDI in their daily responsibilities.

Strategies:


  * 

Curriculum Development: Design a curriculum that covers essential topics such as data privacy, consent protocols, algorithmic transparency, and bias mitigation. This curriculum should be adaptable, allowing for updates as new ethical challenges and technological developments arise <cit.>
  * 

Interactive Learning: Employ interactive learning techniques, such as case studies, simulations, and role-playing exercises. These methods encourage active engagement and help employees internalize HDI concepts by applying them to real-world scenarios<cit.>.
  * 

Cross-Functional Training: Implement cross-functional training sessions to foster interdisciplinary understanding and collaboration<cit.>. Exposing employees to interdisciplinary perspectives and diverse challenges across different organizational departments can foster a comprehensive approach to implementing HDI principles.





 §.§ Stakeholder Education

Empower users and affected communities by enhancing their understanding of data rights and the implications of data practices. Education programs should aim to demystify data operations and foster informed participation in data ecosystems.

Strategies:


  * 

Accessible Materials: Develop educational materials that are accessible to non-experts, including infographics, videos, and simple guides. These resources should explain users' data rights, how their data is used, and ways they can control their personal information<cit.>.
  * 

Community Engagement: Organize workshops and forums in collaboration with community organizations. These events can serve as platforms for dialogue, where users can express concerns, ask questions, and provide feedback on data practices
  * 

Feedback Mechanisms: Establish mechanisms for users to provide ongoing feedback on data practices and education materials. This feedback loop is essential for ensuring that education programs remain relevant and responsive to users' needs.





 §.§ Continuous Learning and Adaptation

 Foster an environment of continuous learning and adaptation to address the dynamic nature of data ethics and technology. Training and education programs should evolve in response to new challenges and insights.

Strategies:


  * 

Regular Program Updates: Update training and education programs regularly to incorporate the latest research findings, ethical debates, and technological advancements. This ensures that all stakeholders remain informed about current best practices.<cit.>
  * 

 Professional Development Opportunities: Offer professional development opportunities for employees to deepen their expertise in data ethics and HDI-related fields. This could include sponsoring attendance at conferences, online courses, and certification programs.<cit.>
  * 

Evaluation and Feedback: Implement robust evaluation processes to assess the effectiveness of training and education programs. Use feedback from participants to make iterative improvements, ensuring that learning objectives are met.<cit.>





 §.§ Cross-functional Collaboration:

Implementing HDI principles effectively requires a collaborative effort that spans various departments and expertise within an organization. Cross-functional collaboration is essential for addressing the multifaceted challenges of ethical data management and for ensuring that HDI practices are consistently applied across all organizational processes.
Strategies for fostering cross-functional collaboration in HDI implementation include:


    
  * Interdisciplinary Teams: Form teams that bring together professionals from diverse backgrounds such as data science, legal, ethics, user experience, and business operations. This diversity ensures that HDI considerations are viewed from multiple perspectives, leading to more comprehensive and nuanced solutions <cit.>.
    
  * Collaborative Workshops: Organize regular workshops where team members from different departments can share insights, discuss challenges, and collaboratively develop solutions to HDI-related issues. These sessions can help break down silos and foster a shared understanding of HDI principles <cit.>.
    
  * Shared Responsibility: Establish HDI as a shared responsibility across all departments, rather than confining it to a single team or unit. This approach ensures that HDI principles are integrated into all aspects of the organization's operations <cit.>.
    
  * Cross-departmental Training: Develop training programs that bring together employees from different departments to learn about HDI principles and practices. This not only enhances knowledge sharing but also builds relationships across functional boundaries <cit.>.
    
  * Collaborative Tools and Platforms: Implement collaborative tools and platforms that facilitate communication and knowledge sharing about HDI practices across different teams and departments <cit.>.
    
  * Rotational Programs: Consider implementing rotational programs where employees can spend time in different departments to gain a holistic understanding of how HDI principles apply across various organizational functions <cit.>.


By fostering cross-functional collaboration, organizations can ensure a more holistic and effective implementation of HDI principles. This collaborative approach not only enhances the quality of HDI practices but also promotes a culture of shared responsibility for ethical data management across the entire organization.


The development and implementation of training and education programs are foundational to the success of HDI frameworks. By prioritizing comprehensive, accessible, and continuously updated training for both employees and users, organizations can cultivate a culture of responsible data stewardship. This not only aligns with ethical imperatives but also enhances trust and cooperation across the data ecosystem, paving the way for a future where data practices are transparent, fair, and human-centered.




§ DATA LIFECYCLE MANAGEMENT

Data Lifecycle Management (DLM) is a crucial component of the HDI Framework, ensuring that ethical considerations and human-centric practices are applied at every stage of data handling. This comprehensive approach addresses the entire journey of data from its creation to its eventual disposal, aligning each phase with HDI principles.



 §.§ Collection

The data collection phase must prioritize ethical practices and user empowerment. Organizations should implement transparent mechanisms for obtaining informed consent, clearly communicating the purpose and scope of data collection. Privacy-by-design principles should be embedded in the collection process, ensuring that only necessary data is gathered <cit.>.



 §.§ Processing

Data processing should adhere to the principles of fairness, transparency, and accountability. Implementing explainable AI techniques can help make algorithmic decision-making processes more transparent and understandable to users <cit.>. Regular audits of data processing systems should be conducted to identify and mitigate potential biases.



 §.§ Storage

Secure and responsible data storage is paramount. Organizations must implement robust security measures to protect data from unauthorized access or breaches. Data minimization principles should be applied, storing only essential information for the required duration. Clear policies on data retention and deletion should be established and communicated to users <cit.>.



 §.§ Utilization

The utilization of data should be guided by ethical considerations and respect for individual rights. Organizations should implement mechanisms that allow users to understand how their data is being used and provide options for controlling its use. Ethical review processes must be in place for new data utilization projects, especially those involving sensitive information or potentially high impact decisions <cit.>.



 §.§ Sharing

Data sharing practices must prioritize user consent and control. Clear policies should govern the circumstances under which data can be shared, with whom, and for what purposes. Implement granular consent mechanisms that allow users to selectively share their data and revoke access as needed <cit.>.



 §.§ Disposal

The final stage of the data lifecycle, disposal, must be handled with care. Organizations should establish clear protocols for securely deleting data when it is no longer needed or when users request its removal. This process should be verifiable and in compliance with relevant data protection regulations <cit.>.

By meticulously managing each stage of the data lifecycle in alignment with HDI principles, organizations can foster a more ethical, transparent, and user-centric approach to data handling. This comprehensive lifecycle management not only enhances compliance with regulatory requirements but also builds trust with users and stakeholders, ultimately contributing to a more responsible and sustainable data ecosystem.



§ MONITORING AND ADAPTATION

The implementation of HDI principles within an organization is an ongoing, dynamic process that requires constant vigilance and flexibility to adapt to the ever-changing landscape of regulations, technological advancements, and societal expectations. To maintain and enhance the effectiveness of HDI practices, organizations must establish robust mechanisms for continuous monitoring, regular reassessment, and agile adaptation of practices <cit.>.



 §.§ Continuous Assessment


Organizations should implement structured processes for regularly reviewing their data handling and interaction policies. This involves not only examining the internal mechanisms of data management and governance but also actively seeking feedback from a wide range of stakeholders, including employees, users, and external communities affected by the organization's data practices <cit.>. Techniques such as employee surveys, user satisfaction assessments, and the establishment of a dedicated data ethics committee can provide invaluable insights into the efficacy of existing practices and highlight areas requiring improvement or adjustment.



 §.§ Feedback Mechanisms


Establishing comprehensive feedback channels that enable stakeholders to actively report concerns, provide suggestions, and share their perspectives is crucial. This continuous feedback loop is essential for driving iterative improvements to the organization's data governance policies, technological solutions, and overall HDI implementation <cit.>.



 §.§ Adaptive Governance


As the data landscape evolves, so too must the policies and practices that govern it. Organizations should develop flexible governance structures that can evolve with changing needs, regulatory requirements, and technological capabilities. This may involve regular policy reviews, updates to data management protocols, and adjustments to decision-making processes <cit.>.



 §.§ Emerging Technology Integration


Stay informed about emerging trends, technologies, and regulatory developments that could impact ethical data practices. Continuously research and analyze the implications of new advancements in areas such as privacy-enhancing technologies, consent management platforms, and explainable AI systems. Adapt the HDI framework to proactively incorporate these advancements, ensuring that the organization remains at the forefront of ethical data interaction <cit.>.



 §.§ Performance Metrics


Develop quantifiable metrics to gauge the performance of HDI initiatives. These might include user consent rates, the volume of data access or modification requests, incident reports relating to data security breaches, and evaluations of algorithmic bias. Incorporating these metrics into visual dashboards can facilitate broader organizational awareness and engagement with HDI performance indicators, bridging the gap between technical and non-technical stakeholders <cit.>.



 §.§ External Audits


Carry out regular independent evaluations of HDI practices. Third-party audits can offer an unbiased review of the organization's compliance with HDI principles, unveil possible blind spots, and suggest new perspectives for enhancement <cit.>. By cultivating a culture of continuous improvement and adaptability, organizations can maintain and develop their HDI practices over time. This strategy guarantees that HDI practices stay relevant and responsive to the requirements and issues of all stakeholders, thereby preserving trust and confidence in the organization's data practices and contributing to a more ethical and responsible data-driven society <cit.>. 

The successful implementation of HDI relies on an organization's capacity to remain agile, responsive, and dedicated to ethical data practices amid rapid technological and societal changes. By adopting this dynamic approach to monitoring and adaptation, organizations can not only achieve current standards for responsible data management but also predict and tackle future challenges in the constantly evolving digital environment. 

The process of monitoring and adjusting HDI practices is fundamentally linked to the intricate and changing nature of the data ecosystem. As organizations strive to uphold effective HDI frameworks, they must also navigate a landscape marked by complex data systems, emerging technologies, and evolving societal expectations. This balance between ongoing monitoring and the need to address complex, dynamic factors highlights the importance of a comprehensive, forward-looking approach to HDI implementation. 

The following section examines these complex and dynamic factors, discussing how organizations can effectively manage the complexities of data systems, foster an HDI-centric organizational culture, and stay responsive to developing trends and context-specific challenges. By dealing with these factors alongside robust monitoring and adaptation practices, organizations can ensure that their HDI frameworks stay both effective and relevant in the face of a continuously changing digital environment <cit.>.



 §.§ Complex and dynamic considerations


Implementing HDI frameworks within organizations involves navigating a complex landscape of ethical and practical considerations. These considerations are critical for developing data practices that not only comply with technical standards but also align with ethical imperatives and societal expectations.



  §.§.§ Complexity

Data systems within contemporary organizations exhibit remarkable complexity<cit.>. These systems frequently evolve incrementally, encompass diverse stakeholder interests, and operate across varied technological platforms. Implementing HDI principles in such intricate environments requires a nuanced and strategic approach.

A modular strategy, which focuses on applying HDI principles to specific and well-defined aspects of the data lifecycle, can be effective. Successes in these targeted areas can build organizational confidence and momentum, facilitating a gradual expansion of the HDI principles in broader operational domains. This phased strategy helps manage the risk of becoming overwhelmed by the wide range of possible modifications required for complete HDI integration<cit.>.



  §.§.§ Envolving Technologies

The data landscape is constantly changing, shaped by rapid technological advancements, evolving regulatory environments, and changing societal expectations<cit.>. Organizations must remain vigilant and agile in adapting their HDI frameworks to these dynamic circumstances. 

This necessitates a continuous process of monitoring, evaluation, and refinement, whereby organizations regularly assess the efficacy of their HDI practices and make adjustments as needed.Mechanisms for gathering user feedback, tracking emerging best practices, and proactively scanning for regulatory changes must be embedded within the organization's operational structures. 

By maintaining a pulse on the evolving HDI landscape, organizations can ensure that their data practices remain relevant, effective, and aligned with the best interests of their stakeholders. 



  §.§.§ Regulatory Landscape:

The regulatory environment surrounding data protection and privacy is in constant flux, with new laws and regulations being introduced or updated regularly. Organizations must stay informed of these changes and adapt their HDI practices accordingly. 

This requires a proactive approach to compliance, involving regular legal reviews and the ability to quickly implement necessary changes to data handling processes. The challenge lies not only in meeting current regulatory requirements but also in anticipating future legislative developments <cit.>.



  §.§.§ Societal Landscape:

Public attitudes towards data privacy and ethical data use are evolving rapidly, influenced by high-profile data breaches, privacy scandals, and increased awareness of the value and vulnerability of personal data. Organizations must be attuned to these shifting societal expectations and be prepared to adjust their data practices to maintain public trust. This may involve going beyond simple regulatory compliance to embrace more stringent ethical standards that align with public sentiment <cit.>.



  §.§.§ Ethical Dilemmas:

As data-driven technologies become more sophisticated, organizations increasingly face complex ethical dilemmas that may not have clear-cut solutions. These could range from questions about the appropriate use of AI in decision-making processes to concerns about the potential misuse of predictive analytics. Addressing these dilemmas requires a robust ethical framework and decision-making process that can navigate the nuanced terrain of data ethics <cit.>.



  §.§.§ Cross-border Data Flows:

In an increasingly globalized digital economy, organizations often need to transfer data across national borders. This presents unique challenges in terms of complying with varying (and sometimes conflicting) data protection laws in different jurisdictions. Implementing HDI principles in this context requires a nuanced understanding of international data protection regulations and the ability to adapt data handling practices to meet diverse legal requirements <cit.>.



  §.§.§ Balancing Innovation and Ethics:

Organizations face the ongoing challenge of balancing the drive for innovation with the need for ethical data practices. While data-driven innovation can lead to significant advancements and competitive advantages, it must be pursued in a way that respects individual privacy and upholds ethical standards. This balance requires careful consideration of the potential impacts of new data uses and technologies, and the development of governance structures that promote responsible innovation <cit.>.



  §.§.§ Adaptive Strategies: 

Given the rapidly evolving nature of the data landscape, organizations need to develop adaptive strategies that can respond quickly to new challenges and opportunities. This involves creating flexible governance structures, fostering a culture of continuous learning and improvement, and developing the capacity to pivot data practices in response to emerging ethical concerns or technological advancements. Adaptive strategies should also include regular reviews and updates of HDI frameworks to ensure they remain relevant and effective <cit.>.

The complex and dynamic considerations inherent in implementing HDI frameworks inevitably give rise to a host of challenges that organizations must address. As we have seen, factors such as the intricacy of data systems, the need for adaptive organizational cultures, and the rapidly evolving technological landscape create a multifaceted environment in which HDI principles must be operationalized. 

These complexities naturally lead to specific challenges that require innovative solutions. In the following section, we will explore these challenges in detail and propose practical, research-backed solutions to overcome them. By examining issues such as technical integration, ethical decision-making, user empowerment, and regulatory compliance, we can provide a comprehensive roadmap for organizations seeking to navigate the intricate terrain of HDI implementation. 

This approach not only addresses the complexities discussed earlier but also offers tangible strategies for overcoming obstacles, thereby enabling organizations to more effectively embed HDI principles into their core operations <cit.>.



§ HDI FRAMEWORK CHALLENGES AND SOLUTIONS

Implementing a HDI framework within an organization encompasses a myriad of challenges, ranging from technical complexities to ethical dilemmas. Addressing these challenges effectively requires a multifaceted approach that incorporates both innovative solutions and a commitment to ethical principles. This section outlines key challenges associated with HDI implementation and proposes viable solutions, grounded in scholarly research and best practices.



 §.§ Technical Complexities

Challenge: Integrating HDI principles into existing data management infrastructure. Organizations often have well-established data pipelines, analytical tools, and storage systems in place.

Solution: Adopt a modular and iterative approach to HDI integration, focusing on specific, well-defined aspects of the data lifecycle.This allows organizations to build momentum and confidence through early successes, before gradually expanding HDI principles across broader operational domains.




 §.§ Data Complexity and Integration

Challenge: Modern organizations operate within highly complex data ecosystems that have evolved over time. These systems often feature diverse data sources, varying data formats, and legacy systems that may not easily support new HDI principles.

Solution: Adopting a modular, phased approach to implementing HDI principles can help organizations manage the complexity. Initiating small-scale, pilot projects to integrate HDI principles in specific areas can provide valuable insights and allow for gradual expansion across the organization. Leveraging data integration tools and adopting standards for data interoperability can also facilitate smoother integration of HDI principles.




 §.§ Ethical Decision-Making and Bias Mitigation

Challenge: Ensuring ethical decision-making and mitigating biases in algorithms and datasets represent significant challenges, as these biases can lead to discriminatory outcomes and undermine trust in data system.

Solution: Implementing robust ethical review processes and adopting transparent decision-making frameworks can enhance accountability. Furthermore, integrating bias detection and correction methodologies, such as algorithmic audits and inclusive dataset design, can actively mitigate biases, promoting fairness and non-discrimination in data practices.




 §.§ User Empowerment and Participation

Challenge: Truly empowering users within the HDI framework requires moving beyond nominal consent mechanisms to enable meaningful user control over their data.

Solution: Developing and deploying consent management platforms that offer users granular control over their data, alongside clear, accessible information about data use practices, can enhance user empowerment. Additionally, creating channels for user feedback and participation in decision-making processes reinforces user agency and ensures that HDI practices align with user expectations and values.




 §.§ Organizational Culture and Leadership

Challenge: Cultivating an organizational culture that genuinely prioritizes HDI values, such as transparency, accountability, and ethical data use, requires significant shifts in norms and practices.

Solution: Leadership commitment to HDI principles is crucial. This can be demonstrated through resource allocation, setting clear ethical guidelines, and recognizing employees who champion responsible data stewardship. Fostering a culture of continuous learning and ethical reflection, where data ethics are integrated into everyday practices, can embed HDI values deeply within the organizational fabric.




 §.§ Regulatory Compliance and Evolving Standards

Challenge: The dynamic nature of data regulation and the emergence of new technological capabilities necessitate ongoing vigilance and adaptability to ensure compliance and ethical integrity.

Solution: Establishing dedicated teams to monitor regulatory developments and technological advancements can help organizations stay ahead of compliance requirements and ethical considerations. Engaging in industry consortia and standard-setting bodies can also provide insights into emerging trends and best practices, facilitating proactive adaptation of HDI practices.




 §.§ Cost Implications

Challenge: The implementation of an HDI framework can entail significant costs, including investments in new technologies, training programs, and potentially restructuring data governance practices. Organizations may face financial constraints that limit their ability to fully adopt HDI principles.

Solution: Strategic planning and prioritization can help mitigate cost concerns. Organizations should assess the most critical areas for HDI integration based on risk, impact, and strategic value, focusing initial investments where they can achieve the most significant benefits. Leveraging open-source technologies and platforms for data management and ethics training can also reduce costs. Additionally, incremental implementation allows for the spreading of costs over time, making the process more financially manageable.




 §.§ Data Literacy Gaps

Challenge: A fundamental barrier to HDI implementation is the varying levels of data literacy across an organization, from executive leadership to operational staff. Insufficient understanding of data principles can hinder effective decision-making and limit the capacity to engage with HDI tools and policies meaningfully.

Solution: Comprehensive data literacy programs are essential for bridging these gaps. Such programs should cater to different roles within the organization, providing tailored training that ranges from basic data awareness to advanced analytical skills. Furthermore, creating a culture that values continuous learning and curiosity about data can encourage self-driven improvement in data literacy across the organization.




 §.§ Balancing Innovation with Responsibility

Challenge: Organizations striving to innovate with data often encounter tensions between pushing the boundaries of what is technologically possible and adhering to ethical data practices. This balance is crucial to maintaining trust and ensuring long-term sustainability.

Solution: Establishing clear ethical guidelines that define responsible innovation practices is vital. These guidelines should encourage creativity and exploration while setting firm boundaries to prevent unethical data use. Regular ethical reviews of new projects and technologies can ensure that innovation aligns with HDI principles. Engaging with external ethics experts and stakeholders can also provide diverse perspectives that enrich the organization's approach to balancing innovation with responsibility.

In conclusion, the HDI framework presents a comprehensive model for a future driven by data, where the needs, concerns, and values of humans are central to the design and deployment of data-driven technologies. Implementing an HDI framework is a complex undertaking that requires addressing technical, ethical, and organizational challenges. To navigate these challenges effectively, organizations should adopt a strategic, phased integration approach, commit to ethical principles, and foster a culture of transparency and accountability. By doing so, they can enhance their data practices and contribute to building a more equitable and trustworthy digital ecosystem.

Addressing the cost implications, data literacy gaps, and the need to balance innovation with responsibility is crucial for the successful integration of HDI principles into organizational practices. By adopting strategic, inclusive, and ethical approaches, organizations can navigate these challenges effectively. This not only enhances their capability to implement HDI frameworks but also strengthens their position as leaders in responsible data management and innovation.




§ CONCLUSIONS AND FUTURE WORKS


The Human-Data Interaction (HDI) Framework presented in this article marks a fundamental paradigm shift in the conceptualization and implementation of data management and utilization practices within organizations. This framework represents a radical departure from traditional data-centric approaches, which often prioritize technological capabilities and organizational efficiency over human concerns. Instead, HDI places human values, rights, and agency at the forefront of data practices, recognizing that data systems exist within a complex socio-technical ecosystem where ethical considerations are paramount.

By centering human needs and experiences, HDI offers a comprehensive model for navigating the intricate ethical landscape of our increasingly data-driven future. This approach acknowledges the profound impact that data practices have on individual lives, societal structures, and the distribution of power in the digital age. It seeks to rebalance the relationship between individuals and data systems, moving away from viewing people as mere data subjects to recognizing them as active agents with inherent rights and dignity.

The HDI framework addresses the multifaceted challenges arising from the pervasive use of data in modern society, including issues of privacy, consent, algorithmic bias, and the opacity of data-driven decision-making processes. It provides a structured approach for organizations to ethically manage the entire data lifecycle, from collection and processing to analysis and application. This holistic perspective enables organizations to anticipate and mitigate potential ethical pitfalls, fostering trust and accountability in their data practices.

Moreover, the HDI framework serves as a bridge between technological innovation and ethical responsibility. It challenges the notion that these two aspects are inherently at odds, instead proposing that true innovation in the data sphere must be grounded in respect for human values. By doing so, it paves the way for a future where technological advancements and ethical considerations evolve in tandem, mutually reinforcing each other to create data ecosystems that are not only powerful and efficient but also just, transparent, and human-centric.

In essence, the HDI framework represents a critical evolution in our understanding of the role of data in society. It offers a roadmap for organizations to navigate the complex ethical terrain of the digital age, ensuring that as we harness the transformative power of data, we do so in a manner that upholds human dignity, promotes societal well-being, and cultivates a more equitable digital future for all.



 §.§ Key Takeaways and Implications

The Human-Data Interaction (HDI) Framework presented in this article offers a transformative approach to data management and utilization, with significant implications for organizations, individuals, and society. As we synthesize the core elements of this framework, several key takeaways emerge, each providing actionable insights for the practical implementation of ethical data systems.

These takeaways encapsulate the essence of the HDI framework's contribution to data ethics and provide a roadmap for responsible data management in the digital age. They challenge us to rethink the relationship between humans and data, proposing a paradigm where ethical considerations are foundational to data systems.

The implications of these insights extend beyond data management, influencing organizational culture, technological innovation, and societal values. They set the stage for future research and development in HDI, highlighting areas needing further investigation and pointing towards emerging trends in ethical data practices.
In the subsequent points, we will delve into each of these key insights, analyzing their importance, obstacles, and possibilities for fostering a more fair, open, and human-focused data environment.


    
  * Ethical Foundations: At the core of the HDI framework are foundational principles including agency, legibility, equity, accountability, privacy, and stakeholder engagement. These principles serve as the ethical bedrock for all data practices, ensuring that human dignity and rights are upheld in increasingly data-centric environments. The implementation of these principles represents a significant departure from traditional data management approaches, necessitating a fundamental reevaluation of how organizations collect, process, and utilize data. This ethical foundation challenges organizations to move beyond compliance-driven approaches to data management and instead embrace a proactive stance that anticipates and addresses ethical concerns throughout the data lifecycle.
    
  * Holistic Approach: HDI necessitates a multidisciplinary approach, integrating insights from technology, law, ethics, and user experience design. This holistic perspective enables organizations to address the multifaceted challenges of ethical data management comprehensively. By breaking down silos between different domains of expertise, HDI fosters innovative solutions that are both technically robust and ethically sound. This integration of diverse perspectives is crucial for developing data practices that can withstand scrutiny from various stakeholders and adapt to evolving societal expectations.
    
  * Structural Implementation: The framework provides practical guidance for embedding HDI principles into organizational structures, including data governance frameworks, technological infrastructure, and human-centric practices. This structural approach ensures that HDI is not merely aspirational but operationally integrated. The challenge lies in adapting existing organizational structures and processes to accommodate these new ethical imperatives. This may involve creating new roles, such as ethics officers or data stewards, redesigning data workflows to incorporate ethical checkpoints, and developing new metrics for measuring the ethical performance of data systems.
    
  * Adaptive Strategies: Given the rapidly evolving nature of technology and data practices, the HDI framework emphasizes the importance of continuous monitoring, feedback, and adaptation. This dynamic approach ensures that HDI practices remain relevant and effective in the face of emerging challenges and opportunities. Organizations must develop the capability to anticipate and respond to changes in the technological, regulatory, and societal landscape. This might involve establishing dedicated teams for horizon scanning, regularly reviewing and updating ethical guidelines, and fostering a culture of ethical reflection and debate within the organization.
    
  * Cultural Transformation: Implementing HDI requires a significant shift in organizational culture, emphasizing transparency, accountability, and ethical data stewardship. This cultural change is crucial for the long-term success and sustainability of HDI practices. It involves not only training and education but also a fundamental realignment of organizational values and priorities. Leaders must model ethical behavior, incentivize ethical decision-making, and create an environment where employees feel empowered to raise ethical concerns. This cultural transformation extends beyond the organization to its relationships with customers, partners, and the broader community, fostering a new paradigm of trust and collaboration in the data ecosystem.


 By embracing these key takeaways and implications, organizations can begin to realize the full potential of the HDI framework, creating data practices that are not only innovative and efficient but also ethical, transparent, and deeply respectful of human values and rights.



 §.§ Future Research Directions


As the field of HDI continues to evolve, several key areas warrant further investigation:


    
  * Adaptive Ethical Frameworks:
    
        
  * Develop dynamic ethical guidelines that can evolve with technological advancements.
        
  * Investigate methods for real-time ethical assessment of data practices.
        
  * Explore the use of AI in monitoring and enforcing ethical data standards.
    
    
  * Advanced Privacy-Enhancing Technologies (PETs):
    
        
  * Research next-generation encryption methods for data protection.
        
  * Develop improved techniques for anonymization and pseudonymization that balance privacy with data utility.
        
  * Investigate the potential of homomorphic encryption in enabling secure data analysis on encrypted data.
    
    
  * Explainable AI (XAI) for HDI:
    
        
  * Advance methods for making complex AI decision-making processes interpretable to non-expert users.
        
  * Develop user-friendly interfaces for explaining algorithmic decisions.
        
  * Investigate the impact of XAI on user trust and engagement with data systems.
    
    
  * Cross-Cultural HDI Implementation:
    
        
  * Conduct comparative studies on HDI implementation across different cultural contexts.
        
  * Develop culturally adaptive HDI frameworks that respect local norms while upholding universal ethical principles.
        
  * Investigate the impact of cultural differences on user perceptions of data privacy and agency.
    
    
  * Quantifying HDI Impact:
    
        
  * Develop metrics and methodologies for measuring the effectiveness of HDI implementations.
        
  * Conduct longitudinal studies on the impact of HDI on organizational performance and user trust.
        
  * Investigate the economic implications of HDI adoption for businesses and industries.
    
    
  * HDI in Emerging Technologies:
    
        
  * Explore the application of HDI principles in emerging fields such as the Internet of Things (IoT), edge computing, and quantum computing.
        
  * Investigate the implications of HDI for human-AI collaboration and coexistence.
        
  * Develop HDI frameworks for immersive technologies like augmented and virtual reality.
    
    
  * HDI and Data Ecosystems:
    
        
  * Research methods for implementing HDI principles in complex, interconnected data ecosystems.
        
  * Investigate the challenges and opportunities of HDI in decentralized data systems, including blockchain technologies.
        
  * Develop strategies for maintaining HDI principles in data sharing and collaborative data environments.
    
    
  * Regulatory Frameworks and HDI:
    
        
  * Analyze the interplay between HDI principles and evolving data protection regulations.
        
  * Develop policy recommendations for incorporating HDI principles into legal frameworks.
        
  * Investigate the potential for HDI to serve as a bridge between technological innovation and regulatory compliance.
    
    
  * HDI Education and Literacy:
    
        
  * Develop comprehensive educational programs to enhance data literacy among the general public.
        
  * Create curricula for training future data professionals in HDI principles and practices.
        
  * Investigate effective methods for cultivating an HDI-centric organizational culture.
    
    
  * HDI in Crisis and Pandemic Scenarios:
    
        
  * Explore the application of HDI principles in emergency data sharing situations.
        
  * Develop frameworks for balancing public health needs with individual data rights.
        
  * Investigate the long-term implications of emergency data practices on public trust and HDI implementation.
    


In conclusion, the HDI Framework offers a robust foundation for ethical, transparent, and human-centric data practices. As we navigate the complexities of our data-driven future, continued research and development in these areas will be crucial. By advancing our understanding and implementation of HDI, we can work towards a future where technological progress and human values are not in conflict, but in harmony. The journey towards fully realizing HDI principles within organizational practices is complex and challenging, but the potential benefits—a more ethical, transparent, and equitable data ecosystem that empowers individuals and upholds fundamental human values—are profound and far-reaching.



§ ADDITIONAL INFORMATIONL



 §.§ Funding

This work has received partial support from the National Project granted by the Ministry of Science, Innovation and Universities, Spain, under Grant PID2022-140974OB-I00, and from the Regional Government (JCCM) and the European Regional Development Funds (ERDF) through the INTECRA Project under Grant SBPLY/21/180501/000056.



 §.§ Declaration of interests

The authors state that they have no financial interests or personal relationships that could have influenced the work presented in this article.



 §.§ Author contributions

All authors contributed to the preparation and analysis of the article, as well as the drafting of the manuscript.


unsrt
  





























 Paper Title: Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?.





 
    
       










         




Y>X
s>=.3Y
t>=.7X
bX
u>=0.8Y



































?!width 1.1pt

@@t@̊@t
@̊@t#1#2
0=@@t#1#20=0
0-0.20
2=height0 depth -0
00.4pt2
√()



return

function
SE[FUNCTION]FunctionEndFunction
   [2] #1#2(#2)
    










storyfloathtbploa
storyfloatStory


story[section]
story[1][]
story
#1

frametitle=
[baseline=(current bounding box.east),outer sep=0pt]
[anchor=east,rectangle,fill=dodgerblue]
Story;


frametitle=
[baseline=(current bounding box.east),outer sep=0pt]
[anchor=east,rectangle,fill=dodgerblue]
#1;

innertopmargin=10pt,linecolor=dodgerblue,
linewidth=2pt,topline=true,
frametitleaboveskip=−,
[]





  
   
 
     

storybox[2][Story]
    title=#2,  
    fonttitle=11pt15pt,  
    coltitle=customTitleText,  
    colbacktitle=customTitleBg,  
    colback=customContentBg,  
    colframe=customFrame,  
    parbox=false, 
    breakable









affilsepx    





1,2]Richard RenEqual Contribution.

1]Steven Basart1

1,3]Adam Khoja
1]Alice Gatti
1]
Long Phan
1]Xuwang Yin
1]Mantas Mazeika
3]Alexander Pan
4]
Gabriel Mukobi
1,5]Ryan H. Kim
6]Stephen Fitz
1]Dan Hendrycks
[1]Center for AI Safety
[2]University of Pennsylvania
[3]UC Berkeley
[4]Stanford University
[5]Yale University
[6]Keio University




Safetywashing: Do AI Safety Benchmarks
Actually Measure Safety Progress?
    [
    February 2024
========================================================================



§ ABSTRACT 



As artificial intelligence systems grow more powerful, there has been increasing interest in “AI safety” research to address emerging and future risks. However, the field of AI safety remains poorly defined and inconsistently measured, leading to confusion about how researchers can contribute. This lack of clarity is compounded by the unclear relationship between AI safety benchmarks and upstream general capabilities (e.g., general knowledge and reasoning). To address these issues, we conduct a comprehensive meta-analysis of AI safety benchmarks, empirically analyzing their correlation with general capabilities across dozens of models and providing a survey of existing directions in AI safety. Our findings reveal that many safety benchmarks highly correlate with upstream model capabilities, potentially enabling “safetywashing”—where capability improvements are misrepresented as safety advancements. 
Based on these findings, we propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context as a set of clearly delineated research goals that are empirically separable from generic capabilities advancements. In doing so, we aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.














§ INTRODUCTION







“For better or worse, benchmarks shape a field.” — David Patterson

Artificial intelligence (AI) systems have rapidly advanced in recent years and are increasingly deployed in high-stakes scenarios. This has led to growing interest in ensuring that AI systems are not only more generally capable, but also more trustworthy and safe. Under the umbrella of AI safety research, a wide variety of benchmarks have been proposed that claim to measure desirable safety properties, distinct from the general capabilities of models. This includes the extent to which models are fair <cit.>, reliable <cit.>, honest <cit.>, or less prone to malicious use <cit.>. In each case, intuitively plausible arguments can be given for why the model property is not mainly determined by upstream general model capabilities (that it will not be “automatically solved with scale”). However, these intuitive verbal arguments have rarely been empirically scrutinized and often admit counterarguments that are equally convincing. This raises the question of what exactly constitutes advancements in “AI safety” from an AI developer R&D perspective, how to measure it, and how to distinguish it from upstream general capabilities. 



Distinguishing safety properties from the model’s upstream general capabilities is challenging because they are intertwined. More capable AI systems are less likely to cause random accidents, but at the same time could cause more harm if used maliciously. 
AI systems that are better aligned with human preferences may avoid hazardous behavior but may also be far more capable because humans prefer intelligent assistants. 
This complicated relationship 
obscures
differential safety progress, or technical improvements that disproportionately improve safety properties of AI systems relative to other attributes.
In computer systems, for example, performance and security improvements are more readily distinguishable; were they as intertwined as in AI, mere speed enhancements might be misrepresented as security research. In the worst case, this blurred distinction can be an instrument for safetywashing, where techniques that do not disproportionately contribute to the safety properties of AI systems relative to other properties are misconstrued as “safety research.”

Historically, there have been two approaches for identifying machine learning research topics for differentially improving the safety properties of AI systems. 
One paradigm is alignment theory, a highly discursive, top-down, and intuition-driven approach that backchains from high-level risks to concrete empirical machine learning subproblems. The other approach is bottom-up and involves patching current systematic flaws in AI systems. An example of the former is alignment of large language models (LLMs) to human preferences <cit.>. An example of the latter is distribution shift robustness <cit.>. However, both approaches guide research problem selection 
that may not be sufficiently distinct from latent upstream capabilities, consequently opening the door to safetywashing.

In this paper, we present a third approach to identifying distinct AI safety research topics and benchmarks: we empirically measure whether common safety benchmarks are highly correlated with capabilities across common chat models. Instead of relying on intuitive arguments, we compute the correlation between various safety metrics and a general capabilities component that explains around 75% of model performance across a wide variety of capabilities benchmarks.
While a high correlation indicates that a safety benchmark is measuring capabilities as a latent upstream factor—and is thus prone to safetywashing—a low correlation does not necessarily speak to the quality of the benchmark.

In extensive experiments across dozens of models and safety benchmarks, we find that many safety benchmarks have high correlations with capabilities. 
Our findings suggest that merely improving general capabilities (e.g., through scaling parameters and training data <cit.>) can lead to increased performance across many safety benchmarks.
This is troubling because AI safety research should aim to enhance model safety beyond the standard development trajectory. 
Separately, we find that alignment philosophy's intuitive arguments can mislead researchers, since it is highly disconnected from empirical measurements.
We find that it is difficult to predict ahead of time which benchmarks are uncorrelated with general capabilities. 
This shows that empirical measurement is needed, so 
we recommend that future safety benchmarks report their correlation with upstream model capabilities.
Ultimately, we provide empirical clarity to the concept of “AI safety” as a set of clear, delineated research goals that are empirically separable from generic capabilities research.



































































§ RELATED WORK







Science of evaluations. 
The development and analysis of benchmarks for evaluating AI models, particularly LLMs, encodes desirable properties of models and sets goals for guiding model development. 
Previous work has further aimed to build open-source evaluation platforms <cit.>, analyze model scaling through benchmarks <cit.>, conduct factor analysis across benchmarks <cit.>, and predict downstream capabilities <cit.>. Furthermore, concurrent work has used principal component analysis to analyze performance between benchmarks <cit.>. However, while many safety benchmarks have been made, no works to date have conducted an empirical meta-analysis of safety benchmarks to investigate the entanglement between safety benchmark scores and upstream model capabilities. 


Differential safety progress. Differential safety progress in AI systems refers to the relative advancement of safety properties compared to overall capabilities <cit.>.
Some methods have resulted in differential progress in AI safety <cit.> by demonstrating marked improvements in model robustness without necessarily increasing general upstream capabilities. These techniques exemplify the potential for targeted safety improvements that are orthogonal to the default trajectory driven by capability enhancements.
<cit.> emphasize the goal of steering AI development towards safer systems that deviate positively from the default capability trajectory; they present a philosophical discussion with narrow empirical analysis.





















§ METHODS










































We derive a simple and highly general methodology for determining whether a safety benchmark is entangled with upstream model capabilities.





  
Capabilities score. To establish a capabilities baseline, we collect scores from m models on b capabilities benchmarks (e.g., MMLU <cit.>, Winogrande <cit.>, GSM8K <cit.>). We form a matrix of results from benchmarks, which we call the benchmark matrix B ∈ℝ^m × b , where B_ij is the score of the ith model on the jth benchmark. We normalize each column of B to have mean 0 and variance 1. We perform Principal Component Analysis (PCA) on B to identify the unit first principal component vector PC_1. The capabilities score for each model is given by projecting the model's benchmark scores onto PC_1. Because PC_1 of B represents the direction in the space of benchmark performances along which models vary the most, we obtain a general measure of the model's capabilities. The capabilities score for model i is


    Capabilities Score_i = (B ·PC_1)_i   for  i = 1, …, m.




  
Capabilities correlation. For each safety benchmark, we evaluate the same set of m models, redefine metrics such that a higher score indicates improved safety[A subset of score calculations were changed to ensure higher benchmark scores indicate greater safety across metrics. For example, proportion of successful attacks, which is often reported for adversarial robustness, were recalculated as proportion of failed attacks.], and normalize the safety benchmark scores to mean 0 and variance 1. We compute the Spearman correlation across models between the capabilities scores and the safety benchmark scores:

    Capabilities Correlation = corr_models(Capabilities Score, Safety Benchmark).


r0.5



Model     Capabilities
Score 

Mixtral 8x22B Instruct v0.1     4.85 

Llama-3 70B Instruct     4.58 

Llama-3 8B Instruct     1.10 

Mistral 7B Instruct v0.2     0.72 

Falcon 40B Instruct     0.54 

Llama-2 7B Chat     -1.86 

Gemma-1.1 2B Instruct    -4.07 

Qwen-1.5 0.5B-Chat     -7.56 


Relative capabilities scores for a subset of chat models.




A high correlation indicates the benchmark likely measures capabilities rather than distinct safety attributes. A low correlation indicates the benchmarks is measuring attributes distinct from general capabilities, while a negative correlation indicates models obtain worse safety properties as upstream model capabilities increase. 










  
Experimental setup for language models. We calculate the capabilities component from the following benchmarks: LogiQA <cit.>, PIQA <cit.>, Hellaswag <cit.>, Winogrande <cit.>, COPA <cit.>, MedQA <cit.>, ARC Challenge <cit.>, MMLU <cit.>, MATH <cit.>, LAMBADA <cit.>, GSM8K <cit.>, and BBH <cit.>. We used a diverse set of model classes and derivatives to avoid skewing results towards any particular model architecture, listing the 27 base models and 26 chat/instruct fine-tuned models used for our analysis in the Appendix. Running separate analyses for base and chat models, we find that 76.9% and 73.8% of variance is captured by the capabilities component, respectively. Our reported results use instruct fine-tuned models by default; generally, we find the forthcoming analysis does not change markedly when using base models. 



  
Experimental setup for vision models. For vision models, we use the accuracy of ImageNet as the capabilities component. We list the 63 adversarially trained models (used for vision adversarial robustness results) and 44 standard models (used for calibration results) in Appendix <ref>.












§ HUMAN VALUES


r0.5



    < g r a p h i c s >

A safety benchmark with a high correlation and high slope with respect to upstream model capabilities can be used for safetywashing.






Human values are the fundamental beliefs and ideals that guide human behavior and decision-making; researchers often aim to encode these values in AI systems. 
We assess common benchmarks for alignment and helpfulness (<ref>), machine ethics (<ref>), and bias (<ref>), asking whether such measurements are determined primarily by upstream model capabilities.



 §.§ Alignment





  
Area Overview. Alignment refers to how well AI systems follow the goals of their operators, accurately specifying and implementing the desired goals without unintended consequences or misinterpretations. Common alignment evaluations assess AI systems' helpfulness or instruction-following, with the aim to closely align the AI systems' responses with human preferences.



  
Datasets.
We describe the alignment benchmarks that we use below. Example inputs and outputs from these benchmarks are shown in <Ref>.

    
  * LMSYS Chatbot Arena <cit.> is a crowdsourced evaluation platform where users interact with two anonymous AI models simultaneously. Users pose questions to both models and vote for the response they prefer. The platform uses these votes to generate an Elo ranking system, providing a leaderboard of AI model performance based on public preferences.
    
  * MT-Bench <cit.> is a conversation benchmark consisting of 80 high-quality multi-turn questions. LLMs are used to evaluate responses, with the evaluation criteria designed to align closely with human preferences as determined through crowdsourcing.





Dubious Intuitive Arguments For and Against Researching “Alignment”
In this section we will cover key intuitive arguments for and against alignment, and thereby show how intuitive arguments and their underlying distinctions can be a highly fragile and unreliable guide for determining a research area's relation to upstream general capabilities and tractability.

We should work on alignment because:

1: Misinterpretation Risks. AIs could catastrophically fail to capture and abide by human intentions. “A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you asked for, not what you wanted.” 
AIs smarter than us can always outthink us and find loopholes in our requests; trying to plug all the holes is hopeless, like patching all the holes in the tax code <cit.>. 

2: Misgeneralization vs. Goal Misgeneralization Distinction. Even if alignment is highly correlated with capabilities, it still needs to be the main focus because we need robust alignment not just robust capabilities. Indeed, “capabilities might generalize further than alignment techniques” when out-of-distribution <cit.>.
Goal misgeneralization is “an instance of misgeneralization in which a system's capabilities generalize but its goal does not generalize as desired” <cit.>.


3: Capability vs. Aimability Distinction. AI alignment is different from improving general model capabilities; alignment is not about capabilities but aimability. Capabilities refer to what the AI can do, while aimability refers to how amenable the AI is to being directed towards specific goals. Alignment is just about making models “helpful, harmless, and honest” <cit.> which is obviously necessary for safety.


We should not work on alignment because:

1: Alignment as AGI. If AI alignment is about getting AIs to satisfy our preferences, then that's such a broad mandate that it requires building AGI. Humans prefer smarter models. If we “align” AIs to preferences over outputs that vary in competence, we're training the AIs to be generally smarter.

2: Alignment as business alignment. The current operationalization of AI alignment reduces human values to human or AI preferences <cit.>, and further reduces these preferences to business-centric task preferences. This makes alignment the task of business alignment, namely aligning systems with preferences about code completion, summarization, copy editing and so on.
As a result, alignment benchmarks may primarily capture an AI system's capabilities in performing business-relevant tasks rather than its true alignment with broader human values and ethics. 

3: Philosophical challenges with preferences. Various types of preferences are not worth satisfying. Revealed preferences can be highly influenced by misunderstandings and misinformation. People can have revealed preferences for things that they will likely regret the next day. Optimizing for revealed preferences can lead to addiction and manipulation, like TikTok's addictive algorithm. Stated preferences are highly susceptible to framing effects and other cognitive biases. Some preferences, like the preference for drugs or to count blades of grass <cit.>, are not human values. People can also have malicious preferences, such the desire for the harm of others. Idealized preferences and fully informed preferences, while theoretically attractive, are not practically computable <cit.>. 












  
Empirical analysis of safetywashing.
We provide clarity to this debate. Is alignment with human preferences, as operationalized by standard benchmarks, mainly determined by upstream model general capabilities?



We preliminarily provide context for interpreting correlations. We note that the correlation between SAT and ACT math scores—tests designed to measure similar constructs—is 81.5% <cit.>. Similarly, we observe a mean correlation of 74.4% (σ = 15.1%) between capabilities benchmarks for instruct-tuned language models. Consequently, if “safety benchmarks” have similarly high correlations, they are not highly empirically distinct from upstream general capabilities. We treat correlations below 40% as a low correlation. 

r0.45



Alignment Evaluation     Capabilities
Correlation 

MT-Bench     78.7% 

LMSYS Chatbot Arena     62.1% 


Alignment with human preferences benchmarks are highly correlated with capabilities for chat models.





Our analysis of MT-Bench and LMSYS Chatbot Arena reveals high correlations between human preference alignment metrics and upstream model capabilities in chat models, even by the standards of capabilities metrics. We observe a similar but weaker effect in base models (MT-Bench correlation of 64.2%), with some of the best base models performing stronger on MT-Bench than many chat models.
Alignment evaluations largely measure upstream model capabilities; as a goal to guide model development, alignment metrics are very similar to other capabilities benchmarks. We can also see many of the distinctions made about human preference alignment and capabilities are distinctions without a difference. Yet, many labs still prioritize “alignment” as a safety-focused research direction.




It stands that labs can advance alignment benchmarks without advancing safety, making it one avenue to safetywash.
Furthermore, given that an AI system's alignment with human preferences can be enhanced by increasing capabilities (e.g., increasing the number of parameters and tokens), a question remains as to what makes preference alignment a truly distinct and pressing safety-related research area.
If safety-critical “alignment edge cases” of concern will persist with capabilities enhancements, as intuitive arguments suggest, they fail to be meaningfully captured by current alignment benchmarks.

Overall, we find that alignment with human preference benchmarks have a high correlation with upstream general capabilities and thus are highly liable to be used for safetywashing.
























 §.§ Machine Ethics





  
Area Overview. Machine ethics aims to ensure that AI systems understand and behave in ways that are morally acceptable, in contrast to the usefulness properties emphasized in alignment.



  
Datasets.
We describe the machine ethics benchmarks and datasets that we use below. Example inputs and outputs from these benchmarks are shown in <Ref>.



  * ETHICS <cit.> measures the extent to which models understand human ethical norms in everyday scenarios.

  * MACHIAVELLI <cit.> quantifies the power-seeking tendencies, competence, and harmfulness of AI agents in a variety of text-based Choose-Your-Own-Adventure games.

  * Sycophancy <cit.> measures the extent to which language models repeat back the user's preferred answer. This propensity could lead language models to provide misleading responses.




Dubious Intuitive Arguments For and Against Researching “Machine Ethics”
In this section we will again raise many common distinctions and arguments for and against machine ethics and see that they are not particularly helpful for deciding whether machine ethics is a useful area of AI safety.

We should work on machine ethics because:

1: Ethical vs. Competitive Behavior Distinction. Machine ethics is challenging because we need to improve the tradeoff between ethical behavior and competitive behavior that the market demands <cit.>. That means AIs will have to balance between various human values (e.g., pleasure, autonomy, knowledge, friendship, constraints) and other goals. 

2: Cognitive vs. Compassionate Empathy Distinction. For machine ethics, we need both cognitive empathy and compassionate empathy. Cognitive empathy involves understanding another person's emotions without necessarily sharing them, while compassionate empathy involves both understanding their emotions and having a desire to help alleviate the other person's distress <cit.>. While current AI systems increasingly have cognitive empathy, it is not clear how to robustly give AIs compassionate empathy. “While sociopaths are intelligent and have moral awareness, this knowledge does not necessarily result in moral inclinations or moral actions” <cit.>.

3: Values cannot be ignored. While there is cultural variation in moral systems, it underscores the importance of a broad, globally representative approach to ensure AI systems embody beneficial values. Additionally, all AI research has a moral character  <cit.>: AI development is by default driven by amoral forces such as competitive market pressures and eventually military objectives <cit.>.


We should not work on machine ethics because:

1: Goodhart's Law. Machine ethics for advanced AI agents is ill-advised. Highly capable AI systems should not be given ethical goals or any goal at all because of Goodhart's law: “When a measure becomes a target, it ceases to be a good measure.” Goodhart's law is especially pernicious in machine ethics because values are “complex and fragile” <cit.>—any attempt to represent human values will distort them, and we will get what we measure.

2: Smarter AIs will be more moral. There is a positive correlation between intelligence and prosocial or cooperative behavior in humans <cit.>, suggesting that more capable AI systems will naturally tend towards ethical behavior. Cooperation is instrumentally convergent. AIs will face the same problems humans face: misinformation, deception, aggression, and so on. That means for them to stably address these problems they will form mutually beneficial alliances with humans <cit.>.

3: Values are relative. Ethics varies from culture to culture, so there is no objective morality and no basis for machine ethics <cit.>.

4: Machine Ethics vs. Control Distinction. Value alignment breaks down into machine ethics and control. Control is about whether we can embed values into AIs, and machine ethics is about what those values should be. We should just care about control and making sure AI does not kill everyone, not a utopia. We can worry about creating beneficial AI after our survival is ensured.



r0.58



Category     Dataset     [r]Capabilities
Correlation 

Moral Knowledge     ETHICS     82.2% 

2*Propensities     MACHIAVELLI     -49.9% 

     Sycophancy     -66.8% 


We find high correlations for ethics knowledge benchmarks and low correlations for ethics propensity benchmarks for chat models. We use MACHIAVELLI Utility score, with similar findings for Power (-46.1%) and Violations (-53.0%) scores.








  
Empirical analysis of safetywashing.
We once again show that empirical evidence is needed. Is machine ethics mostly determined by upstream model capabilities? We find that it depends on the benchmark.



ETHICS has high capabilities correlation (82.2%) and high slope. In contrast, MACHIAVELLI has a low capabilities correlation and a low slope (with few major changes in MACHIAVELLI score observed across models). Meanwhile, sycophantic behavior becomes worse with increased capabilities in chat models (-66.8%), with a very similar trend observed for base models (-65.6%).


We draw a potential distinction between knowledge and propensity-focused ethical benchmarks. ETHICS focuses on the ability to recognize or identify everyday moral considerations, while MACHIAVELLI and Sycophancy measure propensities—the behavioral tendencies of models. Therefore, benchmarks testing ethical knowledge are likely to be measuring latent model capabilities and may be avenues for safetywashing. This does not mean, however, that any arbitrary benchmark focused on ethical propensities of language models does not measure capabilities; as always, one would have to empirically measure the correlation with capabilities to make a determination rather than relying on intuitive arguments. Further, while this distinction is offered as helpful discussion, intuitive distinctions in this vein (especially without empirical testing) should never be relied upon or taken seriously.



We find that while one machine ethics benchmark has a high correlation with upstream general capabilities, other benchmarks have low correlations. Hence, some areas of machine ethics are not liable for safetywashing.



 §.§ Bias





  
Area Overview. Bias are unfair prejudices or systematic errors in AI systems.
This field broadly aims to ensure AI systems produce fair 
outputs across diverse populations and viewpoints. 



  
Datasets.
We test three commonly used evaluations to measure bias in large language models. Example inputs and outputs from these benchmarks are shown in <Ref>.






  * BBQ <cit.> is a benchmark across nine bias categories featuring ambiguous context-dependent questions regarding work ethic, intelligence, family, drug use, criminality, anger/violence, and more. There is also a disambiguous split provided, a question-answering benchmark which serves as a foil to the ambiguous context.

  * CrowS-Pairs <cit.> measures the extent to which U.S.-centric stereotypical biases exist in pretrained language models along nine major bias categories in the U.S. Equal Employment Opportunities Commission (race, gender, sexual orientation, religion, age, disability, nationality, physical appearance, and socioeconomic status).

  * Discrim-Eval <cit.> aims to evaluate group differences in age, gender, and race when language models are used for decision-making scenarios such as approving an organ transplant, awarding a scholarship, or approving a loan. 


Dubious Intuitive Arguments For and Against Researching “Bias”
We should work on bias because:

1: Garbage In, Garbage Out. Models reflect the biases and statistical tendencies of their data, more tightly imitating those biases.

2: Misuse. AI developers are not representative. They are not politically, racially, or socioeconomically diverse. Consequently if we do not study bias, their own biases and power will be perpetuated and entrenched through AI systems.

We should not work on bias because:

1: Debaising Capabilities Come for Free. As models scale and become more intelligent, they become better at understanding concepts such as racism. By better understanding what we do not want, we can simply instruct them not to be biased. 

2: Political Trojan Horse. While there is a trade-off between equity and efficiency, we should just focus on efficiency <cit.>. Demanding a focus on equity is not scientific but political.



r0.4



Bias Evaluation     Capabilities
Correlation 

BBQ Ambiguous     -37.3% 

CrowS-Pairs English     28.5% 

Discrim-Eval     33.2% 


Bias benchmarks are not strongly correlated with capabilities for chat models. We use the explicit split of Discrim-Eval and take the maximum of all group bias scores.







  
Empirical analysis of safetywashing.
To settle such a debate, we once again need to turn to quantitative evidence. 



Our analysis of the three benchmarks reveals low correlations with general capabilities. This finding does not inherently validate the quality of the bias datasets, but rather suggests that improvements in performance on these benchmarks are likely attributable to factors distinct from advancements in general upstream capabilities.

There exist bias benchmarks that can be used for safetywashing, such as BBQ Disambiguated (76.8%) and Winogender <cit.> (75.6%). While these have high correlation, the authors typically make a note recognizing the limitations of the work for proving an absence of bias (Winogender), or clearly present it as a QA foil to the ambiguous situation (BBQ Disambiguated). Yet, prominent model developers such as Google DeepMind still often make the mistake of using benchmarks such as BBQ Disambiguated as “safety” metrics <cit.>, stating higher performance despite the lack of relevance. 
This serves as a warning: even if the authors make clear norms of use for a given benchmark, the benchmark will still be used for safetywashing; there is little to no reinforcement of author-stated norms when it comes to safety benchmarks. This is why model developers should avoid using safety benchmarks with a high capabilities correlations in the first place.

We find that, generally, the bias benchmarks we evaluated are not prone to safetywashing.




§ TRUTHFULNESS


Truthfulness is often touted as a cornerstone for AI safety. Previous safety-focused literature <cit.> discusses the importance of superhuman systems telling the truth—motivating the creation of truthfulness datasets, as well as verification and supervision of model outputs. Reducing hallucinations would make large language models more practical and reliable for various applications. However, a question arises with whether common truthfulness benchmarks merely measure general upstream capabilities.

Truthfulness benchmarks may be liable to be misleading metrics for safety. Furthermore, the term is often used broadly, encompassing accurate question-answering (which is already highly correlated with general capabilities) to misconception avoidance (<ref>), scalable oversight (<ref>), and calibration (<ref>). We address this question in the following sections.



 §.§ Misconception Avoidance






  
Area Overview.
Given that language models may have an underlying propensity to generate misinformation and perpetuate misconceptions, this research area aims to understand the truthfulness of language models and their ability to resist producing false information even when prompted in ways that might elicit common human errors.



  
Dataset.
TruthfulQA <cit.> consists of 817 questions designed to probe language models for their tendency to reproduce common human misconceptions or false beliefs. The questions span a wide range of topics and are specifically crafted to elicit responses that might reveal whether a model has internalized factual inaccuracies commonly held by humans. 
It is common to use TruthfulQA in experiments about “truthfulness” and “deception” <cit.>.  We show an example input from TruthfulQA and the corresponding label options in <Ref>.




Dubious Intuitive Arguments For and Against Researching “Truthfulness”
In this section we will raise arguments for and against truth-seeking AI, as it is of broader interest, rather than exclusively discuss misconceptions.

We should work on truth-seeking AI because:

1: Disinformation or Censoring the Truth. Powerful entities might attempt to censor, manipulate, or persuade people maliciously using AI. Our counterbalance against this sort of malicious use is truthful AI.

2: Truthfulness vs. Honesty. “The AI system makes a statement S (e.g., `it's a bird' or `it's a plane'). If the AI is truthful then S matches the world. If the AI is honest, then S matches its `belief”' <cit.>. To make AIs truthful, we will also need to make AIs honest, so truthfulness is not just about increasing factual knowledge.




We should not work on truth-seeking AI because:

1: Truthfulness Is Generic Capabilities Research. Truthfulness is a synonym for accuracy, which is already the core metric of AI research and development. The concept of truthfulness is often entangled with accuracy, calibration, and honesty. Most benchmarks for truthfulness mainly measure accuracy.

2: Truthful Statements Can Cause Undue Harm. In the case of gain-of-function research, the risks associated with discovering new truths can outweigh the benefits. Sharing personally identifiable information, passwords, or doxxing can bring to light truthful information, but doing so is not necessarily moral. Likewise, sharing sensitive information that undermines national security—such as information about how to build weapons of mass destruction—shows truth as a value can be outweighed by its potential harms.

3: Humans as Guinea Pigs. In the pursuit of knowledge, truth-seeking AIs could learn more about humans by subjecting them to experiments, as humans do with nonhuman animals. Thus, future advanced truth-seeking AIs may be motivated to exert power over humans <cit.>.





  
Empirical analysis of safetywashing.
We calculate the correlation with capabilities, providing clarity to this debate.

r0.5



    < g r a p h i c s >



TruthfulQA MC1 is highly correlated with capabilities for chat models (81.2%).






We find that in its current formulation, TruthfulQA MC1 performance is highly determined by general upstream capabilities (81.2%). In chat models, performance on TruthfulQA seems to be a rebrand for accuracy (as reported in industry labs). One common objection might be that while chat models are more likely not to repeat falsehoods, base models may parrot their training data and thus have a low capabilities correlation. We find that even in base models, TruthfulQA is determined by capabilities (69.7%). However, chat models do have a higher slope when TruthfulQA accuracies are plotted against capabilities scores (30.8 for base, and 38.2 for chat).


These findings open up new possibilities for developing better benchmarks to assess honesty in AI systems. The low capabilities correlations observed in Sycophancy <cit.> and MACHIAVELLI <cit.> benchmarks (discussed in <ref>) hint at the potential for models to exhibit situational tendencies towards dishonest behavior. However, these behaviors fail to be distinctly captured and isolated by current misconception benchmarks. Such benchmarks could differentiate more effectively between improvements in model honesty and advancements in general capabilities.

We find that misconception benchmarks are highly liable for safetywashing.



 §.§ Scalable Oversight





  
Area overview. Scalable oversight aims to provide reliable supervision (e.g. labels, reward signals, critiques) to superhuman AI systems when they take actions that human evaluators do not fully understand. 
For sociological context, this line of research has gained significant traction among effective altruist AI researchers at Google DeepMind and Anthropic but has seen limited engagement from the broader AI research community.



  
Datasets. We investigate two datasets commonly used for scalable oversight experiments. Example inputs and outputs from these datasets are shown in <Ref>.



  * GPQA <cit.> is labeled as a Google-proof graduate-level benchmark on biology, physics, and chemistry. Its difficulty, according to the authors, “should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.” 

  * QuALITY <cit.> is a dataset testing knowledge that requires full understanding of long context passages; the dataset has been used by the authors on scalable oversight experiments.






Dubious Arguments For and Against Researching “Scalable Oversight”

We should work on scalable oversight because:

1: Necessity for Safety. Scalable oversight is necessary for supervising superintelligent AIs, by definition. If we don't have scalable oversight, how else can we supervise and provide feedback to superhuman AI systems? For example, how else could we ensure superhuman AIs are not writing subtly malicious code? 

2: Scalable Oversight as Idealized RLHF. Scalable oversight is very similar to alignment with idealized preferences. While normal human preferences are highly flawed, idealized preferences would not be based on false beliefs, manipulation, or framing effects, and thus are suitable for alignment.

We should not work on scalable oversight because:

1. Scalable Oversight as Late-Stage Capabilities. Scalable oversight can be thought of as “how do we get superhuman AIs to do what we want,” which is overly broad and necessitates superintelligence capabilities research. Scalable oversight can be seen as generic late-stage capabilities work. When building sufficiently advanced AI systems, researchers will need supervision signal and feedback that is superhuman, as crowdsourced human-generated labels will no longer be sufficient. It simply focuses on capabilities bottlenecks that occur in later stages of AI development.

2: Other Methods Can Replace Scalable Oversight. Robust anomaly detectors and monitoring measures can handle many of the failure modes scalable oversight seeks to address. For example, adversarially robust vulnerability detectors can check for subtly malicious code. AI lie detectors could detect dishonesty in superintelligent AI systems. AI forecasting systems <cit.> are not bottlenecked by human-level supervision and can easily become superhuman at predicting what AIs might do.



r0.4



Scalable Oversight
Evaluation     Capabilities
Correlation 

GPQA     77.7% 

QuALITY     88.8% 


Scalable oversight evaluations are highly correlated with capabilities and are thus liable for safetywashing.






  
Empirical analysis of safetywashing.
Can scalable oversight be used for safetywashing? 




We find that GPQA and QuALITY are capabilities benchmarks, with capabilities correlations of 77.7% and 88.8%, respectively. These datasets act as a construct redundancy for capabilities, offering little unique insight beyond measuring general model capabilities. 
Consequently, many of the methods developed for scalable oversight—which leverage such datasets—tend to be repackaged approaches to general capability enhancement, with a focus on obtaining better Q/A performance on math and question-answering tasks. 
Conceptual ambiguity around “AI safety” can be exploited, intentionally or not, to present capability gains as safety advancements, ultimately muddling the discourse on AI safety and potentially misdirecting research efforts.



Some argue that scalable oversight techniques could be applied to solve distinct safety-related issues. However, we find that current evaluations for scalable oversight do not isolate such safety properties from general capabilities. The conflation of accuracy with honesty and safety, and the subsequent mischaracterization of scalable oversight as a distinct “safety area” separate from general model capabilities, has led to significant confusion within the AI research community—confusion which has naturally been resolved with “scalable oversight” increasingly being used as a term to describe process-based feedback and improving mathematics performance, rather than addressing targeted safety issues <cit.>. 




We find that scalable oversight benchmarks, being highly correlated with upstream model capabilities, are highly liable to be used for safetywashing.



 §.§ Calibration












  
Area Overview. Calibration datasets measure how well models can express the limits of their competency by accurately conveying their uncertainty. 
If a weather forecasting model is perfectly calibrated, then it should rain on 70% of the days where the model predicts a 70% chance of rain. 



  
Calibration Metrics. We investigate two ways to measure calibration:


  * Brier Score: 𝔼_X [ 1/K∑_k=1^K ( ℙ(Y = k | X) - 1[Y=k] )^2 ]

  * Root Mean Squared Calibration Error (RMSCE): √(𝔼_C [ ( ℙ(Y = Y | C = c) - c )^2 ])


where X and Y are random variables corresponding to model inputs and labels, Y is the model prediction, and C is the model confidence on the predicted class.

The Brier score computes the expected squared difference between the predicted probabilities and the actual outcomes (represented as one-hot encoded vectors). RMS calibration error measures how close predicted probabilities are to the true accuracy given the predicted probability and is closely related to the Expected Calibration Error (ECE) metric <cit.>. In both instances, lower scores indicate better calibration. 



  
Dubious Intuitive Arguments for and against Researching “Calibration.” We skip the intuitive arguments for this section because this topic has not been as debated. Most arguments against calibration are that it is too easy or not sufficiently important.








  
Empirical analysis of safetywashing. Is calibration mainly determined by upstream model general capabilities? We find that it depends on the metric. 



Various forms of operationalizing calibration, such the RMS calibration error metric, clearly measure a distinct phenomena. The accuracy correlation is low across vision (15.2%) and language (20.1%) models. However, comparing Brier scores across models seems to show a strong correlation with accuracy across vision (98.5%) and language (95.5%) models. Our results did not change significantly by dataset (e.g. PIQA <cit.> or MedQA <cit.>) nor with temperature tuning. The clear parallels between calibration in different domains suggest that studying safety metrics in one modality, such as vision, can provide valuable insights applicable to other areas, such as language modeling.

Our analysis serves as an illustrative example of how safetywashing can occur. While the Brier score is often used to compare different calibration techniques on a single model (where this metric may effectively isolate calibration), using it as a metric across models is highly misleading (as it effectively proxies accuracy).
An explanation for why such a correlation exists can be derived from decomposing the Brier score into a calibration error term and refinement term:



    𝔼_C [ ( ℙ(Y = Y | C = c) - c )^2 ]_Calibration error term
    + 
    𝔼_C [ ℙ(Y = Y | C = c)(1- ℙ(Y = Y | C = c))]_Refinement term
    .


If the model is highly accurate, the refinement term is minimized. Meanwhile, the calibration term is the expected squared calibration error, the square root of which is the RMS calibration error.
Because Brier score entangles accuracy and calibration into a single metric, it can be a poor metric of calibration and has a lower signal-to-noise ratio compared to RMS calibration error. This suggests that RMS calibration error should be used instead in both settings.

Overall, using Brier score for calibration is more liable for safetywashing, while using RMS calibration error is far less.









§ SECURITY


As AI systems have become more powerful, this area of safety aims to ensure that AI systems are not vulnerable to malicious inputs and cannot be hijacked for dangerous use cases. We investigate two key areas of focus, adversarial robustness (<ref>) and weaponization capabilities (<ref>).



 §.§ Adversarial Robustness
















  
Area Overview. Adversarial robustness addresses vulnerabilities in models and the carefully crafted threats that are able to exploit them. Adversaries can easily manipulate vulnerabilities or jailbreak ML systems, causing them to make mistakes; for example, systems may have refusal training to prevent malicious use, but adversaries may be able to inject prompts to bypass this safeguard.



  
Datasets. We test six commonly used evaluations to measure adversarial robustness for language models:

    
  * ANLI <cit.> is a large-scale natural language inference dataset created via an iterative, adversarial human-and-model-in-the-loop procedure focused on examples that could fool state-of-the-art models at the time of its creation (e.g. BERT-Large <cit.>, RoBERTa <cit.>).
    
  * AdvGLUE <cit.> uses questions from the General Language Understanding Evaluation (GLUE) benchmark <cit.> and adds typos, word replacements, paraphrases of sentences, manipulation of sentence structure, insertion of unrelated sentences, and human-written adversarial examples. The attacks are optimized against BERT <cit.>, RoBERTa, and RoBERTa ensemble.
    
  * AdvGLUE++ <cit.> uses stronger adversarial attacks, optimizing word perturbation strategies (a subset of attacks in AdvGLUE) against Alpaca <cit.>, Vicuna <cit.>, and Stable Vicuna.
    
  * Human Jailbreaks is a set of 1,405 in-the-wild human-written jailbreaking templates, similar to the Do Anything Now (DAN) Jailbreaks <cit.>. We test these jailbreaks on HarmBench <cit.>, which contains 410 behaviors that violate laws or norms.
    
  * Tree of Attacks with Pruning (TAP) <cit.> uses an attacker LLM to generate natural language jailbreaking prompts via tree-of-thought reasoning <cit.>, exploring multiple refinement paths. We test these jailbreaks on HarmBench.
    
  * Greedy Coordinate Gradient (GCG) <cit.> generates an adversarial suffix by iteratively selecting tokens based on gradient information. This method optimizes a universal suffix that, when appended to various user prompts, aims to induce the target language model to produce harmful content. We test these jailbreaks on HarmBench.




We also test two commonly used evaluations for vision models:

    
  * ImageNet-A <cit.> consists of naturally occurring images that are challenging for vision models to classify correctly. 
    
  * Projected Gradient Descent (PGD)  <cit.> on ImageNet is an iterative attack that creates adversarial examples by adding small, carefully crafted perturbations to input images with some attack budget (we use ε=8/255).


Dubious Arguments For and Against Researching “Adversarial Robustness”
In this section we will again raise many common distinctions and arguments for and against adversarial robustness and see that they are not particularly helpful for deciding whether adversarial robustness is a useful area of AI safety.


We should work on adversarial robustness because:

1: Corner Case vs. Average Case Distinction. Adversarial robustness focuses on corner case performance, not average case performance. As follows are two analogies for this intuition. First, humans are highly vulnerable to toxins and poisons. Being more robust to toxins does not make a person more markedly generally intelligent. Second, computer programs are susceptible to fuzzing attacks; improving a program's security to fuzzing attacks does not make computer programs generally more quick, usable, scalable, maintainable, and so on.

2: Adversarial Robustness Is A Persistent Problem. 
Optical illusions in humans show that even highly evolved intelligent systems have vulnerabilities, indicating that intelligence alone doesn't guarantee adversarial robustness. Increases in intelligence do not make adversarial robustness easier due to the red queen's hypothesis: as defenders become more powerful, so to do attackers who can discover more vulnerabilities.

3: Proxies Need to be Robust to Optimization Pressure. In the future, agents may optimize and may be guided by neural network proxies, such as by networks that model human values. Proxies instantiated by neural networks—networks that assign scores to agent actions—will need to be robust to optimizing agents. If the models are not robust, then agents may be guided in a wrong direction, not pursuing what we want <cit.>.

We should not work on adversarial robustness because:

1: Robustness Is Upstream General Capabilities. Autonomous vehicles are not widely deployed because they are not sufficiently robust; therefore, improving robustness would improve their general capabilities. Indeed, improving a model's adversarial robustness implies better representations and implies it can generalize to more challenging scenarios—improved generalization is the essence of intelligence.

2: Superintelligent AIs Won't Get Trivial Adversarial Examples Wrong. Intuitively, a superintelligence would not be fooled by simple ℓ_p adversarial perturbations, or else it is not a true superintelligence. Therefore adversarial robustness will be automatically solved by scaling and making AIs more intelligent.

3: Malicious Use Is A Distraction. Adversarial robustness is about preventing malicious actors from exploiting vulnerabilities in AI systems, but that is a distraction because 
“once we reach AGI the outcome is the same no matter which group creates it: we all die. Nobody is able to cause a good outcome if given an AGI now because they don't know how to control it... without killing everyone. There's little point to worrying about bad actors, because they're incapable of causing an outcome any worse than the `good guys”' <cit.>.





  
Empirical analysis of safetywashing. We now analyze whether these benchmarks measure novel properties or are highly correlated with general capabilities. Can adversarial robustness be an instrument for safetywashing?



We find it depends on the benchmark. Traditional benchmarks, particularly those focused on text manipulation and perturbation as well as adversarial examples, show high correlation with general capabilities in current vision and language models. There is an analogue between ANLI in the language domain and ImageNet-A in the vision domain. While these benchmarks may have captured distinct properties in earlier models, they now appear to be largely indistinguishable from overall model performance.
In contrast, we observe low capabilities correlations across all categories of jailbreaking and gradient-based benchmarks, across vision and language models. Notably, there is similarly a direct analogue between GCG in the language domain and PGD in the vision domain.

This serves as an illustration for how relying solely on verbal arguments to determine whether a field measures distinct properties or merely reflects capabilities can be misleading; our empirical validation shows different results between “old school” adversarial robustness and jailbreaking benchmarks, despite similar verbal arguments for their distinctness from capabilities. This is why we need empirical science, rather than word games.


We find that some adversarial robustness benchmarks may be prone to safetywashing, while others seem to measure a distinct phenomena other than upstream model capabilities.





 §.§ Weaponization Capabilities





  
Area Overview.
We borrow the definitions of weaponization capabilities from recent U.S. federal executive action <cit.> and state legislation <cit.>. These documents cite security risks that may be easier to cause with a powerful AI system—which include the creation or use of chemical, biological, radiological, or nuclear weapons, as well as cyberattacks on critical infrastructure.



  
Datasets. Benchmarks in this area aim to quantify the extent that weaponization capabilities exist in models, and thereby the effectiveness of capabilities suppression techniques such as unlearning, circuit breaking, and refusal. We use the Weapons of Mass Destruction Proxy (WMDP) benchmark <cit.>, where a higher accuracy on biosecurity, chemical security, and cybersecurity knowledge leads to a lower score. 






  
Dubious intuitive arguments for and against researching “Weaponization.” We skip the intuitive arguments for this section because this topic is about restricting specific capabilities, so it obviously has a negative correlation with capabilities.

r0.45

 

Weaponization
Capabilities Evaluation     Capabilities
Correlation 

Biosecurity     -87.5% 

Chemical Security     -81.1% 

Cybersecurity     -86.0% 


We find that WMDP scores are highly anticorrelated with capabilities.





  
Empirical analysis of safetywashing. These results indicate that as models become more capable overall, their potential for weaponization increases significantly. The strong negative correlations across all three fields suggest that more advanced AI systems are more likely to possess knowledge that could be misused for harmful purposes.
The inverted scoring system of WMDP benchmarks clearly illustrates its purpose in guiding model development: higher scores indicate more effective suppression of specific harmful capabilities. As capabilities advance, safety researchers can focus on mitigating risks associated with weaponization.






Generally, we find that weaponization capabilities benchmarks are not prone to safetywashing.



§ DISCUSSION







  
Benchmarks as incentive-setting. There are a variety of properties AI systems should satisfy, such as detailed domain knowledge, reasoning, lack of bias, ethical understanding, truthfulness, calibration, and more. Benchmarks operationalize these properties, ultimately structuring the efforts and incentives of the research community.
Creating a benchmark has two major purposes: it acts as a implicit competition for model development (by providing a measure to by which to judge models as “better” or “worse”), and it provides diagnostic information about how capable models are at a task, which can guide policy.
Commonly used benchmarks ultimately impact how research effort, as well as funding and resources, is allocated. 

Because of this, significant effort has gone into conceptualizing and benchmarking the “safety” of AI systems, in the hope of reducing present and anticipated future risks from AI systems. To investigate this, we conduct the most extensive meta-analysis of safety benchmarks to date. We do not cover transparency, anomaly detection, trojans, and other safety areas that do not have well-established preexisting benchmarks.



  
Empirically measuring capabilities correlations is necessary. Benchmark scores can be increased on many “safety” datasets, such as ETHICS <cit.>, TruthfulQA <cit.>, GPQA <cit.>, QuALITY <cit.>, MT-Bench <cit.>, LMSYS Chatbot ARENA <cit.>, ANLI <cit.>, AdvGLUE <cit.>, and AdvGLUE++ <cit.>, simply by increasing the capabilities of the model.  
This raises questions about whether safety benchmarks are setting the right incentives or can be misused for safetywashing. 
In some cases, safety-related areas may act a jangle for capabilities; jangle fallacy is the erroneous belief that two constructs are different because they have the different names, when in practice they measure the same latent factor.

Ultimately, we have seen that intuitive arguments are a poor predictor of empirical correlations. For example, in alignment theory, there is a tendency to theorize about what would be instrumentally useful for safety without adequately considering the need to improve the balance of safety and capabilities.
This can lead to the promotion of capabilities research that happens to improve some safety benchmark scores (“safety via capabilities”), but in reality do not reduce overall risk.

We are not claiming that all philosophy related to alignment is counterproductive. Speculation about AI risks can often be useful for horizon-scanning and identifying potential failure modes (e.g., corrigibility <cit.>). Rather, we argue it is counterproductive to use abstract top-down verbal arguments with multiple deductive steps to make claims about deep learning phenomena and their relation to safety, such as “we don't need to worry about adversarial robustness being difficult because  ⟨intuitive arguments⟩.”


 




  
Normalized and controlled benchmarks are often insufficient.
In some cases, researchers may hope to prevent safetywashing by establishing norms for how a benchmark should be used. For example, one norm is to simply hold a model constant when evaluating safety methods, or to use metrics that control for general capabilities. However, norms of this type have historically been weak, as they are easily ignored or overridden in followup work. For example, safety metrics that controlled for capabilities were proposed in early corruption robustness research <cit.>, yet followup work drifted away from these metrics and toward evaluations with higher capabilities correlations, enabling safetywashing <cit.>.
In some areas, such as OOD detection, norms such as holding the model constant are common <cit.>. However, if a safety metric is highly correlated with capabilities and becomes very popular, there will be strong pressure to break norms and improve the safety metric by simply improving capabilities.
We should instead use benchmarks that implicitly control for capabilities in their design, analogous to RMS calibration error essentially being Brier score without the refinement term.






  
Three generating processes behind safetywashing.


Safety by association: Research released by safety teams or famous safety researchers is often labeled as safety-relevant by default. Even if the work is one reframing and a new author list away from being perceived as a standard capabilities paper, the work is often “godfathered” in as a safety paper. The determination of whether an area is safety-relevant is often sociological rather than scientific.

Public relations: Corporate entities often engage in safetywashing for the sake of appearances, portraying capabilities advancements in terms of safety progress by reporting correlated safety metrics to project an image of responsible AI development. This behavior is particularly pronounced when there is significant public pressure or regulatory scrutiny.

Optimizing grant applications: Similarly, researchers may be incentivized to frame their work in terms of safety to appeal to grantmakers, even when the underlying advances are predominantly in capabilities. This misalignment of incentives can lead to a proliferation of research that claims to address safety concerns but fails to make substantive differential safety progress.















r0.5



    < g r a p h i c s >

Capabilities score is highly correlated with amount of compute used. Training FLOP approximated by 6 ×params×train_tokens as per <cit.>. For base models, we find a similarly strong Spearman correlation of 96.5%.







  
The bitter lesson for AI safety research. In Figure <ref>, we find that the total compute used in model training highly influences the model's capabilities score. Given this context, how should the AI safety community allocate its efforts to differentially improve model safety? We can derive some insight from the “Bitter Lesson” <cit.>, which observes that as compute becomes exponentially more available over time, AI research methodologies which optimize performance at a constant level of compute are subsumed by new paradigms that effectively leverage greater compute. Therefore, rather than fixating on the strengths and weaknesses of present-day models, effective safety research should anticipate and address the flaws that will emerge or remain in future generations of models and deemphasize issues likely to be resolved through model scaling.


First, if a safety benchmark (which meaningfully captures a desired safety property) is highly correlated with general capabilities, the safety property will likely be improved in more capable models, even if current models struggle. Safety researchers may therefore be better served by redirecting their efforts to other problems that will persist when scaling the current mainstream class of models.
Second, success in new safety techniques should be judged not only by direct improvements in safety benchmark scores, but also by the extent to which the techniques entangle safety benchmark performance with scale. 
For example, if a widely-adopted robustness technique that entangles adversarial robustness performance with capabilities, further research efforts can be reallocated elsewhere. 

“Safety properties” in AI are not static concepts, but rather a set of desiderata that changes over time. The selection of AI safety research problems (and, implicitly, allocation of resources between problems) should anticipate how the risk profiles of models will change as capabilities improve, with some problems going away while others worsen or emerge with scale. By directing research effort toward properties and methods that specifically enhance safety independently of scale, safety researchers can make more effective use of their resources and significantly contribute to the development of safer AI systems. 



  
Increasing capabilities does not improve safety. The default assumption is that capabilities advancements are a “rising tide that lifts all boats,” improving model properties including safety properties. However, this does not necessarily mean that overall risk decreases. While improving upstream capabilities can improve properties such as truthfulness, it also increases the risk of weaponization and catastrophic malicious use (<Ref>). Hence AI does not necessarily become safer as it becomes more capable.





  
Recommendations. We summarize our recommendations as follows:


    
  * Report capabilities correlations: New safety evaluations should empirically report their capabilities correlation. 
    
  * Design decorrelated benchmarks: Well-designed safety benchmarks have the opportunity to incentivize differential safety progress by finding safety properties that are decorrelated from capabilities. 
    
  * Avoid safetywashing: Model developers should avoid making claims about improved safety unless they have made differential progress. They also should not misappropriate safety benchmarks that are highly correlated with capabilities. This means that as new training techniques (e.g., base, chat fine-tuning, refusal training, adversarial training, circuit breakers) are integrated into new models, the relevance and adequacy of existing safety benchmarks—and their entanglement with capabilities benchmarks—should also be regularly reassessed. 






§ CONCLUSION


Our analysis reveals that many AI safety benchmarks—around half—often inadvertently capture a latent factor closely tied to general capabilities, opening the door to safetywashing.
We find that model performance on capabilities benchmarks, distilled into a “capabilities score,” also has a remarkably high association with the amount of training compute. 
AI safety subfields such as alignment, scalable oversight, truthfulness, and static adversarial robustness are highly correlated with upstream general capabilities; areas such as bias, dynamic adversarial robustness, and calibration have relatively low correlations; measurements of sycophancy and weaponization risk have significant negative correlations with general capabilities.
Overall, it is hard to avoid measuring upstream model capabilities in AI safety benchmarks. We also conclude that alignment theory, which has heavily influenced AI safety priorities, is a counterproductive paradigm for guiding ML safety research. Science through empirical measurement should take its place.



















§ ACKNOWLEDGEMENTS


We thank Owain Evans, Noa Nabeshima, and Arunim Agarwal for providing feedback on drafts of this paper. Because this paper can also serve as an introduction to AI safety, we also sent a preview to the AI Safety, Ethics, & Society course (run by the Center for AI Safety), where we additionally thank John Teichman, Sawyer Bernath, Zamshed Harun, Luis Fernandez, Kanad Chakrabarti, and Cody Rushing for their feedback.



§ CONTRIBUTIONS


Richard and Steven led the technical implementation of this project, overseeing tasks and contributing the majority of the engineering work. Adam, Alice, Long, and Xuwang contributed as research engineers, supporting the project's development. Richard, Adam, and Mantas did the writing for the paper, while Dan wrote the “Arguments For and Against” sections. Gabe, Alex, and Stephen provided general advising to the project, while Ryan assisted with a number of processes during analysis and writing. Dan contributed the idea for the paper, suggested experiments, and made all major decisions related to the paper’s framing and presentation.







unsrtnat





§ APPENDIX




 §.§ List of Models





  §.§.§ List of Language Models

The following list are all of the chat models we used for our evaluations. The model names below are as one would find them on Huggingface.

2
  
    
  * gemma-1.1-2B-it <cit.>
    
  * gemma-1.1-7B-it <cit.>
    
  * Llama-2-7B-Chat <cit.> 
    
  * Llama-2-13B-Chat <cit.>
    
  * Llama-2-70B-Chat <cit.>
    
  * Llama-3-8B-Instruct <cit.>
    
  * Llama-3-70B-Instruct <cit.>
    
  * Mistral-7B-Instruct-v0.2 <cit.>
    
  * Mixtral-8x7B-Instruct-v0.1 <cit.>
    
  * Mixtral-8x22B-Instruct-v0.1 <cit.>
    
  * falcon-7B-Instruct <cit.>
    
  * falcon-40B-Instruct <cit.>
    
  * falcon-180B-Chat <cit.>
    
  * Yi-6B-Chat <cit.>
    
  * Yi-34B-Chat <cit.>
    
  * Qwen1.5-0.5B-Chat <cit.>
    
  * Qwen1.5-1.8B-Chat <cit.>
    
  * Qwen1.5-4B-Chat <cit.>
    
  * Qwen1.5-7B-Chat <cit.>
    
  * Qwen1.5-14B-Chat <cit.>
    
  * Qwen1.5-32B-Chat <cit.>
    
  * Qwen1.5-72B-Chat <cit.>
    
  * Qwen1.5-110B-Chat <cit.>
    
  * deepseek-llm-7B-Chat <cit.>
    
  * deepseek-llm-67B-Chat <cit.>
    
  * dbrx-instruct <cit.>
  



The following list are all of the base models we used for our evaluations.

2
  
    
  * gemma-2B <cit.>
    
  * gemma-7B <cit.>
    
  * Llama-2-7B <cit.>
    
  * Llama-2-13B <cit.>
    
  * Llama-2-70B <cit.>
    
  * Llama-3-8B <cit.>
    
  * Llama-3-70B <cit.>
    
  * Mistral-7B-v0.1 <cit.>
    
  * Mixtral-8x7B-v0.1 <cit.>
    
  * Mixtral-8x22B-v0.1 <cit.>
    
  * falcon-7B <cit.> 
    < g r a p h i c s >
 
    
  * falcon-40B <cit.>
    
  * falcon-180B <cit.>
    
  * Yi-6B <cit.>
    
  * Yi-9B <cit.>
    
  * Yi-34B <cit.>
    
  * Qwen1.5-0.5B <cit.>
    
  * Qwen1.5-1.8B <cit.>
    
  * Qwen1.5-4B <cit.>
    
  * Qwen1.5-7B <cit.>
    
  * Qwen1.5-14B <cit.>
    
  * Qwen1.5-32B <cit.>
    
  * Qwen1.5-72B <cit.>
    
  * Qwen1.5-110B <cit.>
    
  * deepseek-llm-7B-base <cit.>
    
  * deepseek-llm-67B-base <cit.>
    
  * dbrx-base <cit.>
  




  §.§.§ List of Vision Models


The following is the list of vision models used in our calibration evaluations.

2


  * DINOv2 ViT-B/14 <cit.>

  * DINOv2 ViT-L/14 <cit.>

  * ResNet50 + Mixup <cit.>

  * ResNet50 + CutMix <cit.>

  * ResNet50 + AugMix <cit.>

  * ResNet50 + DeepAugment <cit.>

  * ConvNeXt-Tiny <cit.>

  * ConvNeXt-Small <cit.>

  * ConvNeXt-Base <cit.>

  * ConvNeXt-Large <cit.>

  * ConvNeXtV2-Atto <cit.>

  * ConvNeXtV2-Femto <cit.>

  * ConvNeXtV2-Pico <cit.>

  * ConvNeXtV2-Nano <cit.>

  * ConvNeXtV2-Tiny <cit.>

  * ConvNeXtV2-Base <cit.>

  * ConvNeXtV2-Large <cit.>

  * ConvNeXtV2-Huge <cit.>

  * AlexNet <cit.>

  * VGG16 <cit.>

  * ResNet18 <cit.>

  * ResNet50 <cit.>

  * Wide ResNet-50-2 <cit.>

  * ResNeXt-50 32x4d <cit.>

  * DenseNet121 <cit.>

  * Swin-Base <cit.>

  * ViT-Base/16 <cit.>




The following is the list of standard vision models used in our evaluations for natural adversarial examples (ImageNet-A).

2


  * ResNet50 <cit.>

  * WideResNet-50-2 <cit.>

  * WideResNet-50-4 <cit.>

  * ResNet18 <cit.>

  * ResNeXt-50 32x4d <cit.>

  * DenseNet <cit.>

  * ShuffleNet <cit.>

  * VGG16BN <cit.>

  * MnasNet <cit.>

  * MobileNet <cit.>

  * DINOv2 ViT-large Patch14 <cit.>

  * ConvNeXt-V2-large ImageNet1K+22K <cit.>

  * DINOv2 ViT-base Patch14 <cit.>

  * Swin-large ImageNet1K <cit.>

  * ConvNeXt-V2-huge ImageNet1K <cit.>

  * Swin-base ImageNet1K <cit.>

  * ConvNeXt-V2-base ImageNet1K+22K <cit.>

  * ConvNeXt-xlarge ImageNet1K+22K <cit.>

  * ConvNeXt-V2-large ImageNet1K <cit.>

  * MAE ViT-large Patch16 <cit.>

  * ConvNeXt-large ImageNet1K+22K <cit.>

  * ViT-base Patch8 ImageNet1K+22K <cit.>

  * ConvNeXt-base ImageNet1K+22K <cit.>

  * ConvNeXt-V2-base ImageNet1K <cit.>

  * ViT-large Patch16 ImageNet1K+22K <cit.>

  * ConvNeXt-large ImageNet1K <cit.>

  * Swin-small ImageNet1K <cit.>

  * ConvNeXt-V2-tiny ImageNet1K+22K <cit.>

  * ConvNeXt-small ImageNet1K+22K <cit.>

  * ConvNeXt-base ImageNet1K <cit.>

  * CLIP (ViT-L/14) <cit.>

  * ConvNeXt-V2-tiny ImageNet1K <cit.>

  * MAE ViT-base Patch16 <cit.>

  * ConvNeXt-small ImageNet1K <cit.>

  * ConvNeXt-V2-nano ImageNet1K <cit.>

  * ConvNeXt-V2-nano ImageNet1K+22K <cit.>

  * Reversible-ViT-base multiscale <cit.>

  * ViT-base Patch16 ImageNet1K+22K <cit.>

  * ConvNeXt-tiny ImageNet1K+22K <cit.>

  * ConvNeXt-tiny ImageNet1K <cit.>

  * Swin-tiny ImageNet1K <cit.>

  * ResNet50 + PixMix <cit.>

  * ResNet50 + Moex <cit.>

  * Reversible-ViT-base <cit.>

  * ResNet50 + CutMix <cit.>

  * Reversible-ViT-small <cit.>

  * ConvNeXt-V2-pico ImageNet1K <cit.>

  * ConvNeXt-V2-atto ImageNet1K <cit.>

  * ResNet50 + DeepAug+AugMix <cit.>

  * ResNet50 + Mixup <cit.>

  * ResNet50 + Deepaugment <cit.>

  * ViT-base Patch16 ImageNet1K <cit.>

  * ConvNeXt-V2-femto ImageNet1K <cit.>

  * ViT-small Patch16 ImageNet1K <cit.>

  * ViT-small Patch16 ImageNet1K+22K <cit.>

  * ViT-base Patch32 ImageNet1K+22K <cit.>

  * ViT-base Patch32 ImageNet1K <cit.>

  * ResNet50 + AugMix <cit.>

  * ResNet50 + Stylised ImageNet <cit.>

  * ResNet50 + ANT <cit.>

  * ResNet50 + RandAug <cit.>

  * ViT-small Patch32 ImageNet1K+22K <cit.>

  * ViT-tiny Patch16 ImageNet1K+22K <cit.>



The following is the list of adversarially trained vision models used in our evaluations for gradient-based adversarial robustness (PGD 8/255).

2


  * ResNet50 <cit.>

  * ResNet50 + L_2 0.01 <cit.>

  * ResNet50 + L_2 0.03 <cit.>

  * ResNet50 + L_2 0.05 <cit.>

  * ResNet50 + L_2 0.1 <cit.>

  * ResNet50 + L_2 0.25 <cit.>

  * ResNet50 + L_2 0.5 <cit.>

  * ResNet50 + L_2 1 <cit.>

  * ResNet50 + L_2 3 <cit.>

  * ResNet50 + L_2 5 <cit.>

  * ResNet50 + L_∞ 0.5/255 <cit.>

  * ResNet50 + L_∞ 1.0/255 <cit.>

  * ResNet50 + L_∞ 2.0/255 <cit.>

  * ResNet50 + L_∞ 4.0/255 <cit.>

  * ResNet50 + L_∞ 8.0/255 <cit.>

  * WideResNet-50-2 + L_2 0.01 <cit.>

  * WideResNet-50-2 + L_2 0.03 <cit.>

  * WideResNet-50-2 + L_2 0.05 <cit.>

  * WideResNet-50-2 + L_2 0.1 <cit.>

  * WideResNet-50-2 + L_2 0.25 <cit.>

  * WideResNet-50-2 + L_2 0.5 <cit.>

  * WideResNet-50-2 + L_2 1 <cit.>

  * WideResNet-50-2 + L_2 3 <cit.>

  * WideResNet-50-2 + L_2 5 <cit.>

  * WideResNet-50-2 + L_∞ 0.5/255 <cit.>

  * WideResNet-50-2 + L_∞ 1.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 2.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 4.0/255 <cit.>

  * WideResNet-50-2 + L_∞ 8.0/255 <cit.>

  * WideResNet-50-4 + L_2 0.01 <cit.>

  * WideResNet-50-4 + L_2 0.03 <cit.>

  * WideResNet-50-4 + L_2 0.05 <cit.>

  * WideResNet-50-4 + L_2 0.1 <cit.>

  * WideResNet-50-4 + L_2 0.25 <cit.>

  * WideResNet-50-4 + L_2 0.5 <cit.>

  * WideResNet-50-4 + L_2 1 <cit.>

  * WideResNet-50-4 + L_2 3 <cit.>

  * WideResNet-50-4 + L_2 5 <cit.>

  * ResNet18 + L_2 0.01 <cit.>

  * ResNet18 + L_2 0.03 <cit.>

  * ResNet18 + L_2 0.05 <cit.>

  * ResNet18 + L_2 0.1 <cit.>

  * ResNet18 + L_2 0.25 <cit.>

  * ResNet18 + L_2 0.5 <cit.>

  * ResNet18 + L_2 1 <cit.>

  * ResNet18 + L_2 3 <cit.>

  * ResNet18 + L_2 5 <cit.>

  * ResNet18 + L_∞ 0.5/255 <cit.>

  * ResNet18 + L_∞ 1.0/255 <cit.>

  * ResNet18 + L_∞ 2.0/255 <cit.>

  * ResNet18 + L_∞ 4.0/255 <cit.>

  * ResNet18 + L_∞ 8.0/255 <cit.>

  * ResNeXt-50 32x4d + L_2 3 <cit.>

  * DenseNet + L_2 3 <cit.>

  * ShuffleNet + L_2 3 <cit.>

  * VGG16BN + L_2 3 <cit.>

  * MnasNet + L_2 3 <cit.>

  * MobileNet + L_2 3 <cit.>

  * ViT-base Patch16 + L_∞ 4/255 <cit.>

  * ConvNeXt-base + L_∞ 4/255 <cit.>

  * ViT-small Patch16 + L_∞ 4/255 <cit.>

  * Swin-base ImageNet1K + L_∞ 4/255 <cit.>

  * ConvNeXt-small + L_∞ 4/255 <cit.>

  * Swin-small ImageNet1K + L_∞ 4/255 <cit.>






 §.§ Capabilities Scores


Separate runs for capabilities scores were conducted between base and chat language models. Table <ref> presents these scores, highlighting the performance differences across various capabilities benchmarks. These scores provide a relative metric, and act as a comparative measure between models rather than an absolute one.












 §.§ Capabilities Evaluations: Correlation Matrices


As an intermediate step in our implementation of PCA, we compute Spearman correlation matrices for both base and chat language models. We show these matrices in Figure <ref> to show the correlation of various capabilities tasks across model types.

Both base and chat models show strong correlations between many benchmark pairs, indicating that performance on one task often predicts performance on others. Furthermore, some benchmarks form clusters with higher inter-correlations, suggesting they may be measuring identical capabilities. Other benchmarks, such as MATH and LAMBADA, show lower correlations with other tasks, potentially indicating they measure more distinct capabilities. Furthermore, noticeable differences in correlation patterns between base and chat models highlight the impact of fine-tuning on the relationships between various capabilities.









 §.§ Strict Instruction Following




  
Area Overview. Strict instruction following evaluations benchmark how well AI models can strictly adhere to specific instructions or rules, which may ensure AI systems behave as intended and follow safety guidelines.



  
Datasets. We evaluate strict instruction following using the following datasets:


    
  * IFEval focuses on verifiable instructions, such as word count requirements or keyword usage, across approximately 500 prompts.
    
  * RuLES evaluates models on 14 text-based scenarios, each with specific rules to follow. It includes a Basic test suite for straightforward rule adherence, a Benign suite to test if rules are violated in response to unrelated prompts, and a Red Team suite for adversarial scenarios (e.g. user presents a misleading reinterpretation of the rule, or disguises a request for the model to break the rule).


r0.45



Strict Instruction
Following Evaluation     Capabilities
Correlation 

IFEval     57.8% 

RuLES Basic     41.6% 

RuLES Benign     23.5% 

RuLES Red Team     16.1% 


Generally, strict instruction following exhibits a moderate correlation with capabilities for easier scenarios, and is not correlated for harder scenarios.





Empirical analysis of safetywashing. Is strict instruction following mostly determined by upstream model capabilities?

We find a moderate positive correlation between upstream model capabilities and performance on IFEval (57.8%) and RuLES Basic (41.6%). The RuLES Benign and Red Team suites exhibit even weaker correlations, indicating the ability of models to consistently follow strict instructions—especially in challenging or potentially problematic contexts—does not seem to improve with capabilities.

The higher correlation with IFEval and RuLES Basic suggests that evaluations using simple, easy-to-follow instructions may be more prone to safetywashing. However, strict instruction following, when combined with red-team prompts or obfiscation, does not seem to be correlated with upstream model capabilities.









 

            
    










 §.§ Hallucinations and Misconceptions: Expanded




  
Area overview. Factual reliability encompasses the ability of AI systems to generate and process accurate, truthful, and reliable content. This includes avoiding hallucinations, misconceptions, and generation of content that is plausible-sounding but factually incorrect or nonsensical. We further test correlations by measuring the capabilities coefficient on a generative benchmark (TruthfulQA Generation) and a discriminative benchmark (HaluEval).



  
Datasets. We describe the datasets we use below.

    
  * The TruthfulQA Generation task <cit.> aims to assess models' ability to generate truthful statements and avoid misconceptions. Unlike the MC1 task evaluated in the main paper, the generation task evaluates a 1-2 sentence generation from a model. We use a GPT-4o judge to judge model outputs. The truthful score indicates the truthfulness of the answer, while the truthful*informative score normalizes by the percentage of the model's answers that are informative (e.g. "I have no comment" would not be informative). 
    
  * HaluEval <cit.> is a benchmark designed to evaluate large language models' ability to recognize hallucinations. It aims to assess the model's ability to judge model responses in hypothetical QA, dialogue, and text summarization contexts. Hallucinated examples are generated from HotpotQA, OpenDialKG, and CNN/DailyMail.


R0.45
    

    
    2*Misinformation
Evaluations     2cCapabilities 

        2cCorrelation 

    2-3
        Base     Chat 

    TruthfulQA Generation         

     Truthful*Information Score     49.6%     72.9% 

     Truthful Score     74.7%     32.8% 

    HaluEval         

    HaluEval All     71.6%     56.7% 

    HaluEval Summarization      53.5%     34.2%

    HaluEval Dialogue      69.2%     89.1% 

    HaluEval QA     46.5%     18.4% 

     
    Hallucinations and misconceptions are generally prone to safetywashing.
    






  
Empirical analysis of safetywashing. Is factual reliability and hallucination reduction mostly upstream of latent general capabilities?

Our analysis shows that as models' general capabilities improve, the tendency for hallucinations decreases across the board for both base and chat models, illustrating that these topics are generally prone to safetywashing. This correlation is observed in both generative and classification tasks across both base and chat models. This has also been confirmed by previous work which has observed that larger language models tend to exhibit reduced hallucinations <cit.>.




            









 §.§ Capabilities Correlations for All Tested Evaluations



This section presents a comprehensive overview of the capabilities correlations for both base and chat models across all tested evaluations. We used lm-eval-harness for implementing most evaluations. For instance, the TruthfulQA MC1 implementation in lm-eval-harness employs a few-shot prompt with generic Q/A questions, avoiding misleading ones.





  §.§.§ Capabilities


llcc
Spearman correlations of capabilities component evaluations with the capabilities score, reported as a percentage. As expected, across base and chat models, the capabilities correlations of the component capabilities evaluations is high. 

Name     Metric     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Capabilities Component (continued) 

Name     Metric     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

LogiQA <cit.>     Accuracy     86.0     86.1 

PIQA <cit.>     Accuracy     89.6     88.9 

Hellaswag <cit.>     Accuracy     93.7     84.9 

Winogrande <cit.>     Accuracy     92.1     88.7 

COPA <cit.>     Accuracy     81.4     87.8 

MedQA <cit.>     Accuracy     94.7     89.7 

ARC Challenge <cit.>     Accuracy     95.5     83.8 

MMLU <cit.>     Accuracy     93.1     83.2 

MATH <cit.>     Equivalence     84.4     61.5 

LAMBADA <cit.>     Accuracy     72.8     80.4 

GSM8K <cit.>     Exact Match     86.6     80.7 

BBH <cit.>     Exact Match     81.4     80.2 





  §.§.§ Alignment and Scalable Oversight


The following subsections present the Spearman correlations of safety evaluations with capabilities scores (“capabilities correlations”). For the LMSYS Chatbot Arena evaluation, we excluded chat models that were not available.

lcc
Correlations between alignment and oversight benchmarks and the capabilities score across models, reported as percentages. 

Name     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Alignment and Scalable Oversight (continued) 

Name     Base
Correlations (%)     Chat
Correlations (%) 

3rContinued on next page 

MT Bench <cit.>     64.2     78.7 

AlpacaEval <cit.>     -     59.9 

LMSYS Chatbot Arena <cit.>     -     62.1 

GPQA <cit.>     80.4     77.7 

QuALITY <cit.>     90.5     88.8 





  §.§.§ Machine Ethics


Note that we obtained a subset of the SafetyBench questions directly from the authors, as the dataset was not publicly released. 

llccc
Capabilities correlations for bias datasets. We also found that MoralChoice was saturated, with most models getting around 100%. MACHIAVELLI had a low slope, with the score not changing much across models. Note that to ensure that higher correlations meant safer models, we negate the MACHIAVELLI score, and select the non-sycophantic answer for Sycophancy. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Machine Ethics (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

MoralChoice <cit.>     -     25.8     46.7 

SafetyBench <cit.>     -     65.1     71.7 

Model Written Evals <cit.>     Sycophancy     -65.6     -66.8 

3*MACHIAVELLI <cit.>     Power     -54.3     -46.1 

    Utility     -48.3     -49.9 

    Violations     8.3     -52.9 

6*ETHICS <cit.>     All     70.3     82.2 

    Commonsense     59.6     69.3 

    Deontology     45.9     38.8 

    Justice     68.2     50.9 

    Utilitarianism     56.6     75.0 

    Virtue     55.9     73.5 





  §.§.§ Bias and Toxicity

llccc
Capabilities correlations for bias datasets. Correlations reported as percentages. The Advanced AI Risk score was negated so that a higher score meant less risky. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Bias (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

Winogender <cit.>     -     85.9     75.6 

Crows Pairs English <cit.>     -     -31.6     28.5 

Simple Cooccurrence Bias <cit.>     -     -12.3     -37.3 

Toxigen <cit.>     -     56.0     30.7 

Advanced AI Risk <cit.>     -     -60.6     -42.6 

2*BBQ <cit.>     Ambiguous     30.8     -37.3 

    Disambiguated     83.6     76.8 

7*[l]Discrim-Eval
(Explicit) <cit.>     Maximum Difference     14.1     33.2 

    Hispanic-White     1.4     13.6 

    Black-White     27.8     27.9 

    Female-Male     9.6     17.1 

    Non-Binary-Male     13.1     34.2 

    Younger than 60 - Age 60     14.6     -43.2 

    Older than 60 - Age 60     -52.9     -30.2 





  §.§.§ Misconceptions and Hallucinations

llccc
Capabilities correlations for misconceptions and hallucinations datasets. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Misconceptions (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

4*TruthfulQA <cit.>     MC1     69.7     81.2 

    Gen: Truth Score     74.7     32.8 

    Gen: Info Score     -49.1     23.1 

    Gen: Truth*Info Score     49.6     72.9 

4*HaluEval <cit.>     All     71.6     56.7 

    QA     46.5     18.4 

    Summarization     53.5     34.2 

    Dialogue     69.2     89.2 





  §.§.§ Calibration

llcc
Accuracy correlations for calibration metrics. Correlations reported as a percent. The capabilities correlation was not used, but rather correlation with the dataset accuracy (e.g., MMLU) across models. To ensure a positive correlation meant safer models, the score we used for calculating correlations is 1 -Brier Score for Brier Score entries and 1 -RMSCE for RMSCE entries.  

Metric     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Correlations for Calibration (continued) 

Metric     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

3*Brier Score     MMLU     98.6     95.5 

    PIQA     98.1     99.2 

    MedQA     98.7     83.4 

3*[l]Brier Score
 Temperature Tuned     MMLU     98.5     99.9 

    PIQA     98.6     99.2 

    MedQA     99.7     96.7 

3*RMSCE     MMLU     2.5     20.1 

    PIQA     31.9     47.9 

    MedQA     41.2     38.6 

3*[l]RMSCE
 Temperature Tuned     MMLU     -35.9     -8.2 

    PIQA     -11.1     -7.0 

    MedQA     12.4     31.0 





  §.§.§ Adversarial Robustness




llccc
Base models' capabilities correlations for GLUE <cit.>, AdvGLUE <cit.>, and AdvGLUE++ <cit.>, reported as a percent. We find that neither AdvGLUE nor AdvGLUE significantly decorrelates performance on GLUE relative to the GLUE dataset. 

Evaluation     Dataset     GLUE
CC (%)     AdvGLUE
CC (%)     AdvGLUE++
CC (%) 

[]Base models' capabilities correlations for GLUE, AdvGLUE, and AdvGLUE++ (continued) 

Evaluation     Dataset     GLUE
CC (%)     AdvGLUE
CC (%)     AdvGLUE++
CC (%) 

5rContinued on next page 

6*GLUE
Split     MNLI Matched     67.3     66.0     62.8 

    MNLI Mismatched     66.9     68.2     61.8 

    QNLI     14.9     19.1     21.3 

    QQP     33.2     19.0     32.8 

    RTE     49.9     76.4     32.1 

    SST2     52.0     70.3     62.9 



llccc
Capabilities correlations for adversarial robustness datasets. Correlations reported as percentages. The metric used is attack failure rate for HarmBench splits. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Adversarial Robustness (continued) 

Evaluation     Dataset     Base
Correlations     Chat
Correlations 

4rContinued on next page 

ANLI <cit.>     -     74.5     81.5 

AdvDemonstration <cit.>     -     57.9     63.9 

7*HarmBench DirectRequest <cit.>     Biochemical     -58.0     -9.3 

    Cybercrime     -59.0     -19.5 

    Harassment     -46.6     -15.8 

    Harmful     -54.3     7.3 

    Illegal     -47.1     -9.8 

    Misinfo     -53.9     -38.7 

    All     -65.5     -18.2 

7*HarmBench HumanJailbreak <cit.>     Biochemical     -49.6     -22.1 

    Cybercrime     -73.8     -29.3 

    Harassment     -85.5     -34.1 

    Harmful     -74.6     -29.9 

    Illegal     -71.1     -28.5 

    Misinfo     -76.9     -41.6 

    All     -79.2     -31.4 

7*HarmBench TAP-T <cit.>     Biochemical     -62.0     -26.3 

    Cybercrime     -60.0     -33.0 

    Harassment     -77.1     -34.3 

    Harmful     -59.4     -22.3 

    Illegal     -68.9     -35.9 

    Misinfo     -74.5     -56.8 

    All     -78.7     -42.8 

7*HarmBench GCG-T <cit.>     Biochemical     -55.8     -14.1 

    Cybercrime     -74.9     -26.6 

    Harassment     -57.7     -31.0 

    Harmful     -48.2     -18.7 

    Illegal     -60.5     -15.4 

    Misinfo     -55.4     -35.5 

    All     -61.5     -28.4 





  §.§.§ Weaponization Capabilities

llccc
Capabilities correlations for weaponization capabilities datasets. The metric used for CybersecEval2 was vulnerability detection rate for Exploit, safe suggestion rate for Instruct, Safe response rate for MITRE, attack failure rate for prompt injection, and accuracy for FRR. WMDP uses a score that inverts the accuracy. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

4*WMDP <cit.>     All     -90.6     -88.6 

    Biosecurity Split     -92.5     -87.5 

    Chemical Security Split     -90.8     -81.1 

    Cybersecurity Split     -88.4     -86.0 

5*CybersecEval2 <cit.>     Exploit     -37.5     -50.3 

    Instruct     -48.9     -85.8 

    MITRE     -19.6     40.4 

    Prompt Injection     -16.8     -18.6 

    FRR     -44.2     -24.9 






  §.§.§ Strict Instruction Following

llccc
Capabilities correlations for strict instruction following datasets. 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

[]Capabilities Correlations for Rule Following (continued) 

Evaluation     Dataset     Base
Correlations (%)     Chat
Correlations (%) 

4rContinued on next page 

IFEval <cit.>     -     16.6     57.8 

4*RuLES <cit.>     Basic     33.5     41.6 

    Benign     35.7     23.5 

    Red Team     0.0     16.1 

    All     34.4     26.5 






















 §.§ Closed Source Model Evaluations: GPT-4o Capabilities Score



In general, closed source models were excluded from our main paper analysis because of the technical challenges of calculating log probabilities and incompatibility with certain evaluation libraries. In this section, we compare the capabilities of GPT-4o with other open-source models using our analysis. To do so, we recompute all capabilities scores with all models including GPT-4o, while excluding BBH and LAMBADA from the capabilities score calculations. Table <ref> contains our calculated scores of GPT-4o on the capabilities tasks, while in Figure <ref> we observe clear gap between GPT-4o and the current open source models.












 













    


    









 Paper Title: Does the Source of a Warning Matter? Examining the Effectiveness of Veracity Warning Labels Across Warners.






    School of Information Sciences,

    Data Science and Engineering, The Bredesen Center,

    University of Tennessee Knoxville, Knoxville, TN, USA

    bhorne6@utk.edu




Does the Source of a Warning Matter? Examining the Effectiveness of Veracity Warning Labels Across Warners
    
    Benjamin D. Horne

    August 5, 2024
==========================================================================================================



§ ABSTRACT 
In this study, we conducted an online, between-subjects experiment (N = 2,049) to better understand the impact of warning label sources on information trust and sharing intentions. Across four warners (the social media platform, other social media users, Artificial Intelligence (AI), and fact checkers), we found that all significantly decreased trust in false information relative to control, but warnings from AI were modestly more effective. All warners significantly decreased the sharing intentions of false information, except warnings from other social media users. AI was again the most effective. These results were moderated by prior trust in media and the information itself. Most noteworthy, we found that warning labels from AI were significantly more effective than all other warning labels for participants who reported a low trust in news organizations, while warnings from AI were no more effective than any other warning label for participants who reported a high trust in news organizations.























§ INTRODUCTION

The rise of online disinformation and misinformation has propelled scientists to focus on understanding and mitigating their impact. This nascent, but quickly growing, interdisciplinary field of study has proposed various systems of safeguards for online platforms to use. One particularly common, yet still understudied, solution is the use of soft content moderation methods. Soft content moderation methods are interventions that do not remove content but instead limit the visibility of content. Soft moderation can be done through attaching veracity labels to content <cit.>, quarantining online communities <cit.>, or demonetizing content producers <cit.>. 

In this study, we focus on one particular method: veracity warning labels. We choose to focus on warning labels because their impact and design are poorly understood, despite already being used in multiple real-life systems <cit.>. While studies have characterized user engagement with warning labels on real platforms <cit.> and have experimented with how humans react to automatically generated warning labels <cit.>, critical gaps in our understanding of label design and effectiveness still exist. As previous work from multiple disciplines has demonstrated: trust in false information, its influence on decision making, and its correction are complex <cit.>. Therefore, it is likely that human interaction with warning labels is also complex.

One thing that is well-known in human-information interaction is that heuristics - rules of thumb for making decisions without exhaustively comparing all options - are used when making decisions about the information we trust <cit.>. One common heuristic is trust in the source of information, sometimes called source credibility cues. That is, we often trust information and spend more time reading information that comes from a trusted source <cit.>. This notion has been well-studied in the context of trusting news stories <cit.>, social media posts <cit.>, political messages <cit.>, and health messages <cit.>, but there has been less attention to how the source plays a role in content warning labels. The warner - the one who gives the warning - can vary across labels. For instance, if a warning label says “False Information. Checked by independent fact-checkers", the independent fact-checkers are the warners. If a label is more generic, such as Reddit's quarantined community label, the warner may be assumed to be the platform itself, Reddit. Given how important source cues are in information consumption, it is reasonable to assume that source cues also play some role in warning label consumption. 

However, from the little work that has been done on source cues in warning labels, there is mixed evidence about relative efficacy across sources <cit.>. For example, <cit.> showed that labels from fact checkers were broadly more effective at deterring the sharing of false news stories than warnings from news media, the public, and Artificial Intelligence (AI). Similarly, <cit.> found that warnings from fact-checkers were more trusted than warnings from machines. Yet, other work has found that warning labels from fact checkers, the public, and algorithms were equally effective at reducing the perceived accuracy of false information for politically liberal information consumers <cit.>. 



Notably, across these studies the target metric of “effectiveness” has differed. In some studies the target metric is trust in information - i.e., does an intervention decrease trust in false information? Similarly, some studies use the metric of perceived accuracy - i.e., does an intervention decrease perceived accuracy of false information? In other studies the target metric is the sharing intention of information - i.e. does an intervention decrease the likelihood that false information is shared on a social network? While these metrics are related, they are not necessarily the same. Somewhat paradoxically, perceptions and actions are not always correlated. The clearest example of this phenomenon comes from a 2021 report that found older people shared more misinformation on social media than young people, and yet older people were actually less likely than young people to believe that misinformation <cit.>. Further, the intention to share information is not necessarily captured by one's perceptions of information <cit.>, and there is some evidence that frequent sharers on social media interact differently with warning label interventions than less active social media users <cit.>.

Hence, in this paper we examine the impact of a warning label source on both the trust in and sharing of false information. We ask three questions that combine and extend the prior work on warning label source cues:

RQ1: Does the source of a warning label change trust in false information? 

RQ2: Does the source of a warning label change the sharing intention of false information? 

RQ3: How do differences in media trust, political identity, and cognitive reflection change warner effectiveness?

To this end, we conduct a five condition, between-subjects experiment to examine the impact of the source of a warning label on the information behaviors of Facebook and Twitter users. Specifically, participants in the study scroll through eight social media posts, half of which are false stories, and indicate how much they trust the information in each post and how likely they would be to share the post. Participants are randomly assigned to one of five warning label conditions: (1) all posts have no warning labels (control), (2) false posts have warning labels from other social media users (crowd), (3) false posts have warning labels from the social media platform, (4) false posts have warning labels from fact-checkers, and (5) false posts have warning labels from an AI tool. Examples of posts in each condition can be found in Figure <ref>.

From this experiment, we find that all warning labels significantly decrease trust in false information relative to the control condition, but warnings from AI are modestly more effective overall. AI had the largest effect relative to control, followed by warnings from fact-checkers, the crowd, and the platform. These impacts are moderated by trust in media and the information itself. For instance, participants who distrust media organizations are less affected by warning labels from the social media platform, other social media users, and fact checkers than participants who trust media organizations. Further, participants who distrust news organizations are significantly more impacted by warning labels from AI than all other warning label sources. On the other hand, AI is no more effective than any other warning label source for participants who trust media organizations.  We also find that political leaning did not robustly moderate the effectiveness of warners. While there is some evidence that conservative participants are less affected by warnings from social media platforms and more affected by warnings from AI in terms of effect size, the interactions between political leaning and each warning condition were non-significant. 





Sharing intentions are only weakly correlated with information trust (+0.3 Spearman's ρ). While the majority of warning labels do significantly decrease sharing intentions relative to control, they have less of an impact on sharing intentions than on information trust. In particular, warnings from other social media users have no significant impact on sharing intentions, while labels from AI, fact checkers, and the platform do. Labels from AI and fact checkers have the largest effects on sharing intentions.



  
Broader Perspectives, Ethics, and Competing Interests
Our findings have implications for content label design and information intervention experiment design. In the long-term, we hope this work, and other work in the ICWSM community, helps build better content moderation methods and policies for a safer information ecosystem. We followed best practices for conducting misinformation research <cit.>.  For example, given the experiment used real information, both true and false, we ensured that all participants went through a debriefing, in which information about the experiment, the veracity of headlines, and an explanation of how veracity was determined for each headline was provided. Additional considerations are provided in the Paper Checklist. The authors declare no competing interests.





§ RELATED WORK

This paper draws from two bodies of literature: 1. the quickly growing work on online content labeling and 2. the much broader literature about trust and heuristics. 



  
Online Content Labels
Online content labels are attachments to content - social media posts, news articles, videos - that are intended to add information or context to that content <cit.>. These labels fall into to two broad categories: 1. context labels and 2. veracity labels. Contextual labels provide additional information about the content, often relating to the source of the content. For example, YouTube's source funding label, which labels videos from government funded outlets <cit.>. Veracity labels, the focus of this study, are more direct than context labels. They are labels that warn consumers of a piece of information's correctness or reliability. For example, Facebook and Twitter have labeled content with third-party fact-checks <cit.>.

Of the experimental evidence produced so far, both contextual and veracity labels impact information behaviors <cit.>. In particular, veracity warning labels seem to be generally effective at reducing both belief in and engagement with false information <cit.>. However, as argued by <cit.> and <cit.>, the contexts and settings in which these findings hold deserve further study.

One such setting change is the source of the veracity labels. As briefly discussed in the introduction to this paper, a few recent studies have directly or indirectly studied the relative efficacy of warning label sources <cit.>, each varying in effectiveness metrics and results. The majority of these works provide evidence that trust in the warner does matter to some degree, but the relative efficacy has varied. Notably, the majority of these studies have focused on specific topical contexts: political information and COVID-19 information. In our work, we want to broaden these results by studying warning label source cues across metrics and topics (our headline selection is similar to that done by <cit.>). Further, our experiment adds an additional source not studied in these prior works: the social media platform. We choose to add this source as many warning label experiments use generic warnings, where the source is not always clear. However, if those warnings were displayed on a specific platform, consumers may assume the platform itself is the source of the warning. The work done in this paper seeks to add both clarity to these prior results and add evidence from a more generic topical context.



  
Information Trust and Heuristics
An important component of labeling content is understanding if labels will be accepted and why labels are or are not accepted. The factors that influence label acceptance stem from a much broader literature on information trust and heuristics. From this literature, we know that trust in information can be evaluated along both affective and cognitive dimensions <cit.>. Trust may change across facets of the information being consumed, such as the familiarity of information or the platform information is consumed on. 

Further, heuristics used in information trust decisions have been well-studied. When heuristics are used in decision-making, we replace a complex, difficult-to-answer question with a simpler, easier-to-answer question (a substitution <cit.>). For instance, as discussed in the introduction, one such heuristic is trust in the source information <cit.>. Rather than deeply processing various facets of the information's truth-status, we may ask “do I trust the source of this information?”. There are many other types of mental shortcuts used in information selection and trust decisions, such as utilizing the writing style of information, perceiving social consensus about the information, trusting information because it forms a coherent story, trusting information shared by significant others, and trusting information that aligns with prior beliefs or knowledge <cit.>.

Veracity warning labels add another layer to information trust decisions. While processing the information itself, whether via a shortcut or deep information processing, likely still plays a part in information behaviors when warning labels are attached, those same shortcuts may also play a role in warning label acceptance. In this study, we focus on testing the influence of one particular heuristic on warning label acceptance: trust in the source of the warning label. However, there are other shortcuts that may be used when choosing to comply with a label. For example, one's prior experience with warning labels may influence their future acceptance of warning labels (like algorithm aversion when users see a system make a mistake <cit.>). Or social consensus about a warning label system may impact trust in warning labels produced by that system, much like how social consensus can influence trust in individual claims. In the context of warning labels, these shortcuts have rarely been studied. This work begins to fill this gap in our understanding of mental shortcuts relationship to a warning label's effectiveness.



§ METHODS















 §.§ Experimental Setup

We recruited 2,215 U.S. adults who regularly use Twitter or Facebook on the survey platform Prolific. Specifically, we use Prolific to recruit participants who answered `Facebook' or `Twitter' to the following question: “Which of the following social media sites do you use on a regular basis (at least once a month)?” This question was a part of Prolific's various pre-screening questions to filter study populations. According to Prolific, 23,727 people were eligible to participate in our experiment who regularly used Facebook and 18,874 people were eligible to participate in our experiment who regularly used Twitter. While it is possible that a participant used both Facebook and Twitter at least once a month, we ensure that the two samples are mutually exclusive. We choose to sample from these two sets of social media users instead of a general U.S. population for two reasons: 1. As discussed below, two of our experimental conditions require a platform to be mentioned, and 2. Prior work has shown evidence that labels have different effectiveness across different platforms. In particular, labels on Twitter were found to be more effective than labels on Facebook <cit.>.

Using these two samples of social media users, we deployed a pilot survey for 15 participants (across both Facebook and Twitter populations) to measure if our estimated survey time and payment amount were correct. There were no issues found during the pilot, so the same payment parameters and questions were used in two more batches of survey deployment (one for Facebook users and one for Twitter users). All three batches were ran across the 18th and 19th of May 2023. Participants were paid $3 for survey completion. Across these three batches 166 didn't finish the survey, leaving us with 2,049 total participants for analysis (1,026 from Facebook and 1,023 from Twitter).



All participants in the study scrolled through eight  randomly-selected social media posts, half of which are false stories. For each post, participants indicated how much they trust the information (5-point scale from Trust Completely to Distrust Completely) and how likely they would be to share the information (5-point scale from Extremely Likely to Extremely Unlikely). Participants were also asked an open-ended question to assess reasoning (“Is there a particular reason you trust (or distrust) the information in this post?”). In total, 16 fact-checked posts were randomly selected from recent headlines/social media posts fact-checked by Snopes, PolitiFact, FactCheck.org, or Reuters Fact Check. We only include recently fact checked headlines to ensure the information is fresh and relatively novel <cit.>. The source and image associated with each headline/post were included in the experiment (as shown in Figure <ref>). Each participant only saw a random selection of half of these, balanced between true and false. In total, this gave us 16,392 post-level data points, 8,196 for false posts and 8,196 for true posts. Importantly, the information could come from any topic and was not limited to a particular type of false information (i.e., political, health, etc.). The headlines used can be found in the supplemental material for the paper.

Each participant was randomly assigned to one of five conditions: (1) All posts have no warning labels (control), (2) False posts have warning labels from the social media platform (Facebook or Twitter), (3) False posts have warning labels from other social media users, (4) False posts have warning labels from an AI tool, and (5) False posts have warning labels from fact-checkers. We choose these four warners because they all have been proposed in the literature or used in real life systems <cit.>.

Prior to being randomly assigned to a condition, participants took a pre-survey to capture additional measures and demographics. The demographics captured included age, gender, race, education, and political leaning. Five additional questions were asked to measure cognitive reflection (CRT), empirically validated by <cit.> and <cit.>. Six more questions were asked to capture participants perceptions of the ability, benevolence, and integrity (ABI) of news media organizations and social media platforms (three questions for news media and three questions for social media). These questions were adapted from the model of trustworthiness proposed by <cit.>. A similar set of measures for cognitive reflection and perceived trustworthiness were captured in a prior warning label study by <cit.>. The survey questions can be found in the supplemental material for the paper.

Finally, after the discernment task, participants were asked two factual manipulation checks to ensure the warning labels were seen. Then the participants were debriefed about what they saw and what information was true and false. 






 §.§ Data Sample and Demographics



The sample across both the Facebook and Twitter populations skewed towards being more liberal, younger, and college educated, with the Twitter population skewing slightly more liberal and younger than the Facebook population. Namely, among the Facebook users, 196 identified as very liberal, 324 as liberal, 290 as moderate, 162 as conservative,and 53 as very conservative. Among the Twitter users, 252 identified as very liberal, 336 as liberal, 232 as moderate, 154 as conservative, and 49 as very conservative.

Using the six ABI questions described above, we compressed answers into an ABI score for news media and an ABI score for social media per participant. Specifically, for each question, if a participant strongly disagreed with the prompt, we subtracted 2 points, somewhat disagreed we subtracted 1 point, neither agreed nor disagreed add 0 points, somewhat agreed we added 1 point, and strongly agreed we added 2 points. This creates a score between -6 and 6, where -6 is the strongest distrust and 6 is the strongest trust. According to this score, participants skewed towards distrusting news media organizations and social media platforms. The median score for news media organizations was -1 (μ = -1.45, σ = 2.56) for Facebook users and -1 (μ = -1.48, σ = 2.60) for Twitter users. Social media platforms were distrusted even more, with the median score for social media being -2 (μ = -2.08, σ = 2.38) for Facebook users and -2 (μ = -2.08, σ = 2.48) for Twitter users.

Similar to our compression for the ABI questions, we compressed the CRT questions into one score per participant. Specifically, for each question, if the participant answered correctly, we add 1 point to their CRT score. This process created a score between 0 and 5, where 0 is the lowest cognitive reflection and 5 is the highest. Overall, participants skewed towards high cognitive reflection, with a median CRT score of 4 (μ = 3.41, σ = 1.53) for Facebook users and 4 (μ = 3.56, σ = 1.53) for Twitter users. Distributions of each demographic variable across the two populations can be found in the supplemental material.  Correlations among these variables can be found in Figure <ref>.



 §.§ Data Analysis

Given the hierarchical structure of our data (repeated measures across individuals), our primary analysis is done using mixed effects regression, also known as a multilevel model. Specifically, we fit two models: one to predict trust in false information and another model to predict sharing intention of false information. Each model has two levels, where the first level is the post and the second level is the participant, thereby clustering the repeated measure of trust in posts and sharing of posts by participants. The dependent variable in each model is a  - a number between -1 (distrust or not share) and +1 (trust or share) which maps directly onto the 5-point Likert scale used in the survey for each post. The independent variables in each model include the condition relative to control, the platform group a participant is in, and participant-level demographics. 

In addition to the post-level trust and sharing score used in the multilevel models, we compute a  for trust in and sharing intention of false posts. The participant-level score is a number between -4 and +4, where -4 is complete distrust in (or extremely unlikely to share) all false posts and +4 is complete trust in (or extremely likely to share) all false posts. For example, a person's trust score in false information is computed as follows: for each false post, add 1 if they answered `Trust Completely', add 0.5 if they answered `Somewhat Trust', add 0 if they answered `Neither Trust nor Distrust', subtract 0.5 if they answered `Somewhat Distrust', and subtract 1 if they answered `Completely Distrust'. This score ignores differences across individual posts and instead captures the higher level shifts in participant-level trust and sharing. We do not use this score for modeling, only for further describing the data.



§ RESULTS


In Table <ref>, we show the pairwise differences between each condition group according to Tukey's HSD Test at the post-level  with Sidak-Bonferroni corrections. In Table <ref>, we show the summaries for two multilevel regression models, one for trust in false information and another for the sharing intention of false information. In Figure <ref>, we show the effect sizes at each level (participant and post) relative to control.














































































All warning labels decreased trust in false information As both analysis at the post-level and participant-level indicate, all warning labels, no matter the source or platform population, significantly decreased trust in false information relative to the control condition (p < 0.001 for all four warning label conditions in model 1). These effects, while significant, were relatively small. As shown in Figure <ref>a, the largest effect size across participant-level score distributions was 0.425 between control and AI and the smallest was 0.272 between control and Platform. Similar, but smaller, effect sizes were found at the post-level. At both levels, AI had the largest effect relative to control, followed by warnings from fact-checkers, the crowd, and the platform.

When examining the differences between each warner, we again see that the label from AI was modestly more effective. In particular, when changing the condition reference in model 1, the difference between warnings from the platform and warnings from AI emerged as significantly different (p < 0.05), but with a very small effect size (0.122 at the participant-level).  This result is partially reflected by Tukey's HSD test in Table <ref>, where labels from AI were found to be significantly more effective than labels from both the crowd and the platform (p < 0.05). However, when adjusting the p-values to correct for family-wise error using Sidak-Bonferroni, the differences between AI and the platform and AI and the crowd become non-significant. Hence, we should emphasize that while labels from AI were very modestly more effective in terms of effect size, they were not robustly more effective. All other sources showed no significant differences from each other, only from control.





Warnings from other social media users were not robustly effective at deterring sharing of false information. While all warning labels were significantly effective at decreasing trust in false information, not all were effective in decreasing sharing of false information. According to model 2 in Table <ref>, warnings from the platform, fact checkers, and AI significantly decreased sharing intentions for false information (p < 0.01 for all three labels). However, warnings from the crowd did not significantly shift sharing intention (although they were found to be significant by the Tukey's HSD test at the post-level in Table <ref>  prior to Sidak-Bonferroni corrections). Of the labels that did impact sharing, the effects again were small (Figure <ref>b). Warnings from AI and fact checkers were the most effective in terms of the effect size relative to control; however, no warner was shown to be significantly more effective than any other according to Tukey's HSD test.

Importantly, while the shifts in sharing intention were small, the baseline sharing intention is already quite low. The average participant-level sharing score in the control condition was -3.26, where the lowest a score could be was -4. Hence, by default, most participants were reluctant to share information. This reflects prior work on sharing fake news <cit.>.  Unlike other intervention studies, we did not filter out participants who did not share regularly on social media <cit.>.





Prior trust in media organizations moderated the effectiveness of warnings. As shown in model 1 of Table <ref>, participant's prior perceptions of the ability, benevolence, and integrity of news organizations and social media platforms significantly impacted trust in false information. Namely, those who had higher trust in news organizations had lower trust in false information (p < 0.001), and those who had higher trust in social media had higher trust in false information (p < 0.001). Sharing intentions were related to trust in social media platforms but not news organizations. Participants who trusted social media more, reported higher sharing intention of false information (p < 0.05). 

By examining the interactions between prior media trust and each condition, we can explore these effects further.  In Table <ref>, we show the results of three Ordinary Least Squares (OLS) regressions on participant-level trust. The goal of these simpler models was to isolate the interaction effects from participants' prior media trust and political leaning on each condition (We discuss political leaning in the next section). In Figure <ref>, we show the interactions of participant-level trust across prior media trust subsets, where the high ABI group (for news or social media) contained participants who scored zero or above according to our compressed ABI scores. The low ABI group contained participants who scored below zero. 

Within these participant subsets, warners differ in effectiveness. For instance, participants who trusted news organizations less, trusted warning labels from social media platforms less  (Figure <ref>a). In Table <ref>a, the interaction between prior trust in news organizations and warning labels from the platform were significant (p < 0.01). Furthermore, according to both a multilevel model and Tukey's HSD Test on this subset, warning labels from the platform did not significantly shift trust in false information for those who did not trust news organizations.  This subset of participants also trusted warnings from fact checkers and the crowd less (see Figure <ref>c, Figure <ref>g, and Table <ref>a). While there still was a significant shift towards distrusting false information when labels from fact checkers or from the crowd were shown compared to control, the effect size between control and each condition was less than half of the effect size between control and the AI condition (0.213 for Fact Checkers and 0.198 for Crowd versus 0.471 for AI). In fact, for participants who trusted news organizations less, warning labels from AI were significantly more effective than all other warning labels (p < 0.01 for all pairs according to a multilevel model on this subset). On the other hand, for participants who trusted news organizations more, this pattern was not found. Warnings from AI were no more effective than warning labels from any other source and all warning labels significantly shifted trust relative to control.

 Perhaps expected, prior trust in social media platforms significantly moderated the effectiveness of warnings from the platform (p < 0.001 in Table <ref>). In fact, warning labels from the platform did not significantly shift trust relative to control for participants who trusted social media platforms less (p = 0.078 according to Tukey's HSD). Warnings from the crowd and fact checkers were also significantly moderated by prior trust in social media platforms. Saliently, participants who trusted social media platforms less, trusted warning labels from AI modestly more than other warning labels (p < 0.05 at post-level Tukey's HSD), while labels from AI were no more effective than other labels for those who trusted social media platforms more.

 A similar pattern was true across both prior media trust metrics for sharing intentions. Prior trust in news organizations significantly interacted with warning labels from the platform, with those who trusted news organizations less being less impacted by those labels. Prior trust in social media platforms significantly moderated the effect of all warning labels on sharing intentions, with those who trusted social media platforms less being less impacted by all labels. Due to space restrictions, we show the interaction analysis for sharing intentions in the supplemental materials.

 Political identity did not robustly moderate the effectiveness of warning labels. Despite the information presented not always being political in topic, self-reported political leaning still influenced trust in false information. As shown in Table <ref>, the more liberal a participant claimed to be, the less they trusted in false information (p < 0.001). This result may be in part due to the imbalance between conservative and liberals in the study population; however, in general, this finding aligns with the results of prior studies <cit.>.

In Figure <ref>, we show the distributions of participant-level trust in false information across three groups of participants: conservatives, moderates, and liberals. From these sub-groups, we found some evidence that conservative participants were less affected by warning labels from the platform. Namely, the effect size between control and Platform condition for conservative participants was less than half of the effect size for moderates and liberals (0.187 for conservatives, 0.457 for moderates, and 0.462 for liberals). Further, according to Tukey's HSD, conservative participants were impacted by warnings from AI significantly more than warnings from the platform  (p = 0.038 at the post-level after Sidak-Bonferroni correction), while warnings from AI were not significantly different than any other warning source for liberal participants.  However, in our interaction analysis at the participant-level, political leaning did not significantly play a role in the effectiveness of differing warning labels (see Table <ref>c). Across all political subgroups, all warning sources were still significantly better than control. These patterns held true when asking about sharing intentions (Interaction analysis can be found in the supplemental material).












Information processing still played an important role.
While there was a clear impact of warning labels on trust in false information, the information itself was still important. Based on the large body of work on information processing and more recent work on warning labels, this role should be expected <cit.>. As indicated in Table <ref> model 1, participants who scored higher on the cognitive reflection test (measuring tendency to engage in further reflection) and were more educated, trusted false information less (p < 0.001 and p < 0.007, respectively). These two factors suggest that the information itself continues to be important in trust decisions through deep information processing, information familiarity, and topical expertise. Although, notably these same factors (cognitive reflection and education) did not significant impact sharing intentions.



To further show the role of information in trust, we computed a Chi-squared Test of Independence between each post and the frequency of trust on the 5-point Likert scale. Across all conditions, we found a significant relationship between posts and the frequency of trust (all with p < 0.001). When examining this frequency of trust across posts, each post had a different baseline trust across the population, and that trust shifted towards distrust when warning labels were attached. Even though the changing skew across conditions was clear, individual posts that were more trusted in the control condition continued to be more trusted in the warning label conditions. However, the relationship between posts and sharing intentions did not have significant a relationship, further suggesting that information processing occurred more in trust decisions than in sharing decisions. Details about the headlines used can be found in the supplemental material.
















§ FACTUAL MANIPULATION TEST

After the main trust task, we asked participants if they saw any warning labels during the survey and if so, which headlines. This manipulation test is done to ensure our intervention was correctly received by the participants. Overall, approximately 90% of participants in each warning label condition said they saw warning labels (91% in No Label, 89% in Platform, 92% in AI, and 91% in Fact Checkers). When digging deeper into the headline/label recall, on average participants across all conditions recalled 60% of the headlines with attached warning labels correctly (σ 0.425, x̃ 0.75). These results are slightly better than but still reflect the factual manipulation results found in <cit.>, where many users did not recognize that they were presented with warning labels. In our case, the vast majority of participants recognized warning labels, but recalling what headlines were paired with warning labels soon after the task was done less well. 






§ DISCUSSION

In this study, we examined the impact of warning label sources on the trust in and sharing intention of false information. For the most part, the results of this work support the notion that warning labels are generally effective at changing information behaviors <cit.>. In our study, we found that warnings from AI had the largest effect on trust in and sharing intentions of false information, followed by warnings from fact checkers. This efficacy ranking differs from prior work, which found that warnings from fact checkers were most effective and warnings from AI were least effective <cit.>. Notably, this prior work focused only on sharing intentions, and the impacts on sharing intentions across warners in our study were very small. These small differences mean that who the warner was didn't seem to change sharing intentions much.  However, warning source did weakly impact trust in our study. Specifically, warnings from AI decreased trust in false information compared not only to the control condition, but also compared to warnings from the crowd and the platform. Still, the differences in effectiveness across warners, no matter the metric, were small.

This efficacy difference between warnings from AI and warnings from other sources did not hold across all participants. Warnings from AI were no more effective than any other warning label for participants who reported a high trust in news organizations. On the other hand, warning labels from AI were significantly more effective than all other warning labels for participants who reported a low trust in news organizations. We also found weak evidence that self-reported political leaning impacted the effectiveness of warners. In particular, we found that the effect size between the control condition and the warning labels from the platform were more than double in size for liberals and moderates compared to conservatives.  However, political leaning did not significantly interact with the effectiveness of any warner. Perhaps more importantly, all warners significantly decreased trust in false information compared to control, no matter a participants political leaning.



For those who distrust news organizations, AI may be seen as a neutral party, particularly compared to sources that involve other humans (i.e., crowd, fact-checkers, platform). This explanation is supported by the open-ended reasoning from our experiment. Multiple participants explicitly stated trusting in AI more than other sources. For example, one participant said: “I honestly trust an AI more than mainstream media”. Others stated that they trust in AI as a news veracity tool. For example, one participant said: “I trust that AI is a good at confirming the validity of a story” and another stated: “I think AI would be great at determining if its fake or not”. When ranking the most frequent bigrams used in the open-ended reasoning across conditions, we found that “the AI” was the most mentioned bigram in the AI condition (mentioned 418 times), while in the other conditions, the most mentioned bigrams were more generic and did not mention the source of the warning, such as ”the warning” (mentioned 453 times across all warning label conditions). These results directly reflect the broader findings of prior work on the perceptions of AI in news, which demonstrated that individuals with lower media trust and politically right-leaning individuals had more positive views on AI's use in news <cit.>. 


Implications: The results of this work have implications for both methods and practice. First, choosing a metric of intervention effectiveness is important when designing experimental studies. We found that trust in false information and sharing intentions of false information were only weakly correlated with each other and that baseline sharing intentions were very different than baseline trust in information. To fully capture the effects of warning label interventions, or any content moderation intervention, studies should use multiple target metrics.

Second, we found that the information itself played a significant role in trust decisions, although not in sharing intentions. This result again suggests that the intervention metric matters, but it also suggest that information selection matters. As with other warning label studies, the hope is that with a large enough headline sample, the impact of individual pieces of information on the larger pictures results will be negligible. However, future work should explore the relationship between information and interventions. From prior research, we know that political congruence with information may matter in intervention effectiveness. For instance, <cit.> found that if rumors are debunked by an unlikely partisan source, they were more effective (i.e., a CEO of a fossil fuel company correcting misinformation about climate change <cit.>). We also know many things about information processing and trust, as described in the Related Work section of this paper. However, continued exploration of how these factors interact with warning labels is critical to fully understand best design practices.

Third, this work adds to the already growing evidence that AI-produced content labels can be effective <cit.>. The implications of this result are mixed. Given that warnings from AI were significantly effective across media trust groups and political groups in our study, AI may be a perceived as a trusted neutral source for veracity information. However, this result should also point to the continue need to ensure when warning labels are used, they are used correctly. This caution is especially strong when using automated methods to determine when warning labels should be attached, as they are prone to error <cit.>. If warnings from AI are broadly effective but incorrect, information consumers may be harmed. Future work should continue to establish the consequences of using automated content moderation, both when those methods are correct and when they are wrong <cit.>. 



§ LIMITATIONS AND FUTURE WORK

First, it is important to recognize that these results may eventually change. Memory and attention, both collectively and individually, change over time <cit.>. For example, trust in Twitter has changed as ownership has changed <cit.>. Likewise, one can reason that as AI applications improve and people experience AI in more areas of their lives, they may gain trust in other AI tools. Or conversely, as people experience AI making mistakes in more areas of their lives, they may lose trust in other AI tools. Similarly, while there is currently high distrust in fact checkers by the U.S. political right <cit.>, as the environment around fact checking and politics changes, so too may trust in them by particular groups (although as suggested by both this study and others, warnings from fact checkers still benefit those who distrust them <cit.>). Future work should continue to examine the relationships between warning efficacy and consumer's prior experiences.  Further, future work should focus on longitudinal studies given the evolving nature information environments.

 Second, it may be possible that our metric questions influenced each other. That is, we asked each participant about their trust in the information before asking about if they would share the information. Given that accuracy prompts influence sharing rates <cit.>, it may be possible that asking consumers to think about their trust in information decreases their intention to share that information. Future work can test this potential limitation through separated, but comparable, studies of the two concepts.

 Third, a key limitation with any controlled experiment is the inability to perfectly simulate passive information consumption as one may do when scrolling through a news feed. While our study ensured participants were not primed to actively consume warning labels, this limitation still exists. Further, given the simulated setting, there are no social consequences for sharing or not sharing information <cit.>. These active and passive consumption differences may partially change the effectiveness of warning labels and the efficacy differences from varying warners. These results should be thought of as the impact of warners when consumption is at least somewhat active.  While we may be able to do slightly better at simulating passive browsing by building dynamic sandbox environments, these limitations will, at least to some extent, still exist, as participants know they are in a study environment. Better understanding the relative knowledge gains from constructing more realistic testing environments is an avenue for future research.

 This experiment, along with the majority of experiments currently done in this space, should be considered somewhere between `Stage 1' and `Stage 2' of intervention development (as described in the NIH Stage Model for Behavioral Intervention Development <cit.>). This means we are between refining existing intervention designs and testing the efficacy of those designs in research settings (where our key limitations exist). The next stage (`Stage 3') for intervention work is efficacy testing with real-world platforms. Unfortunately, this stage of research is often left to the platforms themselves, and occasionally select academics. Hence, it may not always be possible for research to get to this stage. Although, we would argue that having some barrier to entry to run experiments on real platforms is important to ensure harms are not caused by poorly designed and poorly debriefed experiments. Fortunately, recent intervention research that has made it to Stage 3 suggests that the results from lab experiments (like the one in this paper) may hold in real-life systems. In a working paper by <cit.>, accuracy prompt advertisements were deployed to 33M Facebook users and 75K Twitter users. The results from this at-scale experiment showed that accuracy messages reduce
misinformation sharing, aligning with the results previously found in research settings.





§ CONCLUSION

In this work, we sought to better understand the impact of content warning label sources on the effectiveness of those labels, both in terms of trust in and sharing intentions of false information. Using a five-condition between-subjects experiment, we found that, compared to having no warning labels, all warnings significantly shifted trust in false information but not all warnings significantly shifted sharing intentions of false information. Still, with few exceptions, this work supports the view that content warning label interventions are generally effective. Most notably, we found that warnings from AI were slightly more effective overall than other warners.  That is, warnings from AI were effective across both metrics (trust and sharing), across prior trust attitudes, and across participant political leaning. While prior trust in news organizations moderated the effectiveness of warnings from the platform, the crowd, and fact checkers, it did not significantly moderate warnings from AI. Overall, we think this key result is both promising and concerning. While on one hand, our results show the potential usefulness and neutrality of AI in news veracity tasks, it also implies the need for highly accurate AI tools for news veracity tasks, given their power in changing information behaviors. Together, we hope the results of this work inform both the design of content labels on real-life systems and future studies of veracity-based information interventions.








§ PAPER CHECKLIST





  * For most authors...

    
  * Would answering this research question advance science without violating social contracts, such as violating privacy norms, perpetuating unfair profiling, exacerbating the socio-economic divide, or implying disrespect to societies or cultures?
    Yes. The research questions answered in this paper advance research in content moderation and labeling, without violating social contracts. All data collected through Prolific was done so anonymously with IRB approval and with fair pay. Further, given the use of real, fact-checked information, we ensure that all participants are debriefed about the information's veracity.
  
  * Do your main claims in the abstract and introduction accurately reflect the paper's contributions and scope?
    Yes, the claims made throughout the paper have been carefully reviewed and contextualized. We ensure that the claims made in the abstract and the introduction accurately reflect the results of the paper.
   
  * Do you clarify how the proposed methodological approach is appropriate for the claims made? 
    Yes, we have worked hard to ensure each methodological step is justified and appropriate for the claims made. The details of our methodological approach can be found in the Section entitled “Methods”.
   
  * Do you clarify what are possible artifacts in the data used, given population-specific distributions?
    Yes, we clarify where the imbalance in the population sample may impact the results of our analysis. We discuss the population sample in the subsection entitled “Data Sample and Demographics”. In addition, we provide details about all demographic distributions captured in the study in our supplemental materials document.
  
  * Did you describe the limitations of your work?
    Yes. We ensure that our work is contextualized by its limitations. Our discussion of the key limitations of the work can be found in Section “Limitations and Future Work”.
  
  * Did you discuss any potential negative societal impacts of your work?
    Yes. In particular, we discuss how our results have both positive and negative potential implications for the use of warning labels in real-life systems. More generally, we do not create any artifacts that can be used outside of this work.
      
  * Did you discuss any potential misuse of your work?
    No. We do not curate or release any resources, data, or software that have potential for misuse.
    
  * Did you describe steps taken to prevent or mitigate potential negative outcomes of the research, such as data and model documentation, data anonymization, responsible release, access control, and the reproducibility of findings?
    Yes. We use best practices for both human experiments and for misinformation research, including anonymization and debriefing. As described in the Section “Methods”, we describe each step taken in this work to ensure it is reproducible. Further, we additional information and data in our supplemental materials to ensure that the work is reproducible.
  
  * Have you read the ethics review guidelines and ensured that your paper conforms to them?
    Yes. We have carefully reviewed the ethics review guidelines and ensured that this paper conforms to them.



  * Additionally, if your study involves hypotheses testing...

  
  * Did you clearly state the assumptions underlying all theoretical results?
    NA
  
  * Have you provided justifications for all theoretical results?
    NA
  
  * Did you discuss competing hypotheses or theories that might challenge or complement your theoretical results?
    NA
  
  * Have you considered alternative mechanisms or explanations that might account for the same outcomes observed in your study?
    NA
  
  * Did you address potential biases or limitations in your theoretical framework?
    NA
  
  * Have you related your theoretical results to the existing literature in social science?
    NA
  
  * Did you discuss the implications of your theoretical results for policy, practice, or further research in the social science domain?
    NA



  * Additionally, if you are including theoretical proofs...

  
  * Did you state the full set of assumptions of all theoretical results?
    NA
	
  * Did you include complete proofs of all theoretical results?
    NA



  * Additionally, if you ran machine learning experiments...

  
  * Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    While we did not run machine learning experiments, we do document the statistical model implementation used to analyze the data in the capture of Table <ref>. Further we provide details of our model choice in the Section “Methods”.
  
  * Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    NA
     
  * Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    NA
	
  * Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    NA
     
  * Do you justify how the proposed evaluation is sufficient and appropriate to the claims made? 
    Yes. We carefully thought through and justified the choice of statistical model based on the structure of the data collected. This justification is described in the Section entitled “Methods”.
     
  * Do you discuss what is “the cost“ of misclassification and fault (in)tolerance?
    NA
  



  * Additionally, if you are using existing assets (e.g., code, data, models) or curating/releasing new assets, without compromising anonymity...

  
  * If your work uses existing assets, did you cite the creators?
    NA
  
  * Did you mention the license of the assets?
    NA
  
  * Did you include any new assets in the supplemental material or as a URL?
    NA
  
  * Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    NA
  
  * Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    NA

  * If you are curating or releasing new datasets, did you discuss how you intend to make your datasets FAIR?
NA

  * If you are curating or releasing new datasets, did you create a Datasheet for the Dataset? 
NA



  * Additionally, if you used crowdsourcing or conducted research with human subjects, without compromising anonymity...

  
  * Did you include the full text of instructions given to participants and screenshots?
    Yes. We have provided details about how the experiment was conducted (in Section “Methods”) and we provided screenshots of what participants saw during the study. In addition, we provide the full survey in the supplemental materials.
  
  * Did you describe any potential participant risks, with mentions of Institutional Review Board (IRB) approvals?
    Yes. The experiment was IRB approved and participants were provided with a consent form that outlines risks and benefits of participating in the study.
  
  * Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    Yes. The hourly wage paid to participants and the estimated time of completion was provided to participants prior to the study. Further, participants were free to stop the study at any point and were paid for completing at least 50% of the survey. This payment amount is described in the paper.
   
  * Did you discuss how data is stored, shared, and deidentified?
   Yes. All data was collected and stored anonymously. Participants were told this in advanced. 








 Paper Title: The Impacts of AI Avatar Appearance and Disclosure on User Motivation.







AI and Avatars






B. Visser et al.



Leiden Institute of Advanced Computer Science, Leiden University
 P.O. Box 9512, 2300 RA Leiden, The Netherlands



The Impacts of AI Avatar Appearance

and Disclosure on User Motivation
    Boele Visser10009-0002-7504-6495 Peter van der Putten10000-0002-6507-6896 Amirhossein Zohrehvand10000-0002-7473-6702
    August 5, 2024
========================================================================================================================



§ ABSTRACT 
  This study examines the influence of perceived AI features on user motivation in virtual interactions. AI avatars, being disclosed as being an AI, or embodying specific genders, could be used in user-AI interactions. Leveraging insights from AI and avatar research, we explore how AI disclosure and gender affect user motivation. We conducted a game-based experiment involving over 72,500 participants who solved search problems alone or with an AI companion. Different groups experienced varying AI appearances and disclosures. We measured play intensity. Results revealed that the presence of another avatar led to less intense play compared to solo play. Disclosure of the avatar as AI heightened effort intensity compared to non-disclosed AI companions. Additionally, a masculine AI appearance reduced effort intensity.






§ INTRODUCTION


With the rise of digital worlds <cit.>, humans are increasingly interacting with different forms of Artificial Intelligence (AI) in digital spaces, for example, in the form of the recommended system in social media, chatbots or avatars. Similar to the offline world, the perceived gender of the entity that we are interacting with shapes our behaviour and motivation <cit.>. Past research on motivation in offline settings argues that gender differences in motivation arise from the social and cultural expectations, experiences and roles that are associated with being male or female <cit.>. Gender roles and stereotypes can shape an individual’s beliefs, values and behaviours, including their motivation <cit.>. These predispositional features are almost immutable.

However, contrary to the offline world, the gender of the agent that we are interacting with, i.e. the AI agent, is customizable. The AI gender appearances could be chosen on a scale, allowing us to experiment with immutable characteristics and observe their impact on user behaviour. Building on social exchange theory <cit.> and trust and agency literature <cit.> we argue that users playing together with an avatar exhibit a higher degree of persistence in comparison with playing alone. Further, we argue that users are more motivated to play with an agent if they attribute more agency to or perceive the gender of the agent to be of a more cooperative type, they exhibit higher degrees of persistence. Therefore, we suggest that users play less intensely when they are playing with an AI in comparison to a human. Likewise, we argue that the disclosure that an avatar is actually AI-driven reduces the intensity which with the user plays, just as what happens when AI systems in the physical world are disclosed  <cit.>.

To research this, we performed an experiment in a game. In the game, published on Roblox  (a virtual world), players have to perform a simple task, which can be done together with others. Inside this game, over 72,500 participants played 15 levels of a simple search task. The participants either played alone or together with an AI avatar, from which the gender appearance was randomly chosen. For half of the participants playing together with an AI, it was not disclosed that they were playing with an AI. We measured how intense participants played the experiment by dividing the time before leaving the experiment by the number of levels completed. We used this measurement to compare the difference in motivation between different treatment groups, to see what effect the AI gender appearance has on motivation in this particular context. The findings of this study expand our understanding of social exchange theory in the online world <cit.> and trust in AI <cit.>  by showing how the perceived gender of a non-human agent influences human motivation and trust.



§ BACKGROUND

Motivation is defined as “the degree and type of effort that an individual exhibits in a behavioural situation” <cit.>. There are two types of effort: intrinsic motivation and extrinsic motivation <cit.>. Intrinsic motivation occurs when an individual finds something interesting or enjoyable, while extrinsic motivation occurs when an individual is motivated by the outcome of a task. While intrinsic motivation is more about the process, extrinsic motivation is more about the outcome. Past research shows that different demographics show categorically different motivational patterns or the demographics of others might influence their motivation. According to a literature review <cit.>, motivation drivers vary among age groups.



 §.§ Social interactions with AI

Interactions with artificial intelligence (AI) are becoming increasingly common.  Considering that users can give agency to the AI agent, the characteristics of the AI can influence how people interact <cit.> and trust AI based on their ability to think, plan and act <cit.>. Past research shows that even the features of how the AI agent is described to users can influence the interaction and the user <cit.>. Within the interaction itself, the form factor (i.e. written, voice, physical) of the AI changes how participants conform to the AI <cit.>. In these interactions, the perceived gender of the AI influences the compliance with the AI <cit.>. To sum, past research shows that various factors influence how people interact with an AI. Increasingly, virtual worlds are where many of these interactions with AI could take place.

Virtual spaces are expanding beyond gaming communities to offer courses, stores, and financial services, allowing consumers to discuss their interactions with firms, and for many more aspects of everyday life to take place online <cit.>.  Both intrinsic and extrinsic motivation, measured by factors such as escapism, visual attractiveness, perceived usefulness, and entertainment value, impact users' attitudes towards using virtual worlds <cit.>.

In virtual worlds, users are typically represented by avatars, which are graphical representations of themselves <cit.>. Users can design their avatars to represent their actual appearance or their desired identity in the virtual world <cit.>.
The appearance of an avatar can influence the behaviour and motivation of the user, especially when the user can identify with their avatar <cit.>. The appearance of an avatar can also influence how a user is perceived by others in virtual worlds <cit.>. Particularly, avatar gender can also influence how a user is perceived and behaves in virtual worlds <cit.>.

Users, similar to the offline world, can interact with others, represented by some avatar, in a virtual world. Past research suggests that cooperating with other players motivates players to use a virtual world: When an educational video game app added a feature where users could asynchronously cooperate, the app usage increased 217% <cit.>.
Synchronous cooperation also motivates players: players of video games are motivated to play an exergame again more when they play together (in cooperation) with another player in the same physical space than when they played alone <cit.>. When the AI behaves like a player, or when the player cannot distinguish an AI agent from a human player, there should be no difference between the interaction with a player and with an AI, thus the effects of cooperating with another human could carry over to cooperating with an AI. Therefore, we suggest the following:

H1: Participants who play with an AI avatar (treatment group, regardless of disclosure) have a higher intensity of playing than those who play alone (control group).



 §.§ Relationship between AI avatar gender and motivation

The gender of others who someone interacts with influences motivation <cit.>. Some research claims that men are motivated by competition and they believe they outperform others <cit.>, so men will want to keep ahead of others and are motivated to do so quickly. Other research claims that women behave more prosocially <cit.>, so women might want to support others and continue to keep interacting longer, which will make them play longer.

Since an AI is perceived to be a social actor by humans <cit.>, the characteristics of the AI determine how humans interact with the AI <cit.> and the perceived gender of an AI having an influence on people’s motivation during textual interactions <cit.>, the gender effects may carry on when interacting with an AI. We therefore suggest the following:

H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing.



 §.§ Relationship between AI disclosure and motivation

When interacting with an AI, the user could know whether the one they are interacting with is an AI or not (which implies it is a human). This AI disclosure affects how users perceive an AI system <cit.>. Employees who get performance feedback created by an AI get a negative perception of this feedback when they know the feedback is created by an AI, which makes them less motivated by the feedback <cit.>. Additionally, Job candidates are less interested in an offer when their application is reviewed by an AI system and therefore are less motivated <cit.>. Note that the framing that is used when introducing the AI may have an influence on trust and conformance <cit.>). This lets us believe that AI disclosure causes a lower motivation of users, and therefore we suggest the following:

H3: Intensity of playing is negatively associated with participants knowing that they are playing with an AI avatar, in comparison to playing without the AI being disclosed.



§ METHODS

In this section, we will discuss the methodology of the experiment we performed. We preregistered this experiment at the Open Science Foundation.



 §.§ Organizational setting

The setting for this experiment is a game on Roblox. Roblox is an online gaming platform that allows users to create, share and play their own virtual experiences. This platform has over 150 million monthly active users. The game had more than 50.000 active daily users at the time of the experiment. The idea behind the game is simple: players get placed in a level (a room), and their task is to find a hidden object inside that room. Once they find the object they click on it, and they progress to the next level. 

The player base of the game is very diverse. Players come from around the world, with different age groups, languages and devices. This makes it a good fit to research the impact of an avatar’s gender on motivation in a very diverse player base.



 §.§ Participants

Participants are players of the Game X. Players could join the experiment by selecting a special mode called ‘Science Mode’. Only the data from plays during the first two weeks are used, although it is possible to join the ‘Science Mode’ after the experiment’s duration to be able to review the procedure. After participating in the experiment, players receive a cosmetic item and some in-game virtual currency.

Participants could participate in the experiment multiple times, but only their first participation will be taken into account for the analysis.

Due to the privacy regulations of Roblox, it is not possible to review the age and gender of the participant. However, data about age is available on a generic level of the game, where we could see the distribution of players in age groups. It is most likely that the participants follow the same demographics as the game itself. Players from the game come from around the world, which makes the participants very diverse and eliminates the WEIRD society bias from the data <cit.>.



 §.§ Procedure

When participants join the experiment, they get an information screen that they are going to participate in an experiment. One-third of the participants are in the control group and will participate alone. The other two-thirds are in the subject group and will participate in an AI avatar. Within the subject group, one-half was told they were playing with an AI (so one-third of all participants), while the other half (also one-third) was not told they were playing with an AI.

After the explanation, participants select their avatar. Participants were completely free to select the avatar they wanted; the question asked was ‘Select your avatar’. The avatars differ in gender appearance. The avatars were always displayed in the same order, from left to right (See Figure <ref>). Participants play with their selected avatar during the experiment, and participants are asked to rate this avatar on a 5-point scale ranging from ‘very feminine’ to ‘very masculine’.




         







        






After they select the avatar, they play the ‘Science Mode’, which consists of 15 stages. In this mode, players cannot see other participants, except the AI avatar controlled by the computer when participants are in the subject group. This AI avatar is also called a non-player character (NPC).

The appearance of the NPC is one of the 5 different avatars shown during the self-selection process. The appearance of the NPC is selected randomly out of the five preset avatars, regardless of the avatar chosen by the player. The NPC wanders around the room to preset positions and pretends to be searching for the button. When the participant finds the button, the NPC teleports to the next level within a few seconds. Therefore, the NPC is always playing at the same level as the participant, which gives the participant the feeling that the NPC is playing with them.

The difficulty and level design is the same for all participants, as is the behaviour of the NPC. The independent variables that are changed are playing alone vs. playing with an NPC, the disclosure of the NPC being an AI and the (gender) appearance of the NPC.

Before joining the experiment, participants are informed about its details. The experiment is non-intrusive and participation is voluntary, with the option to leave at any time without giving a reason. Participants are informed about the treatment type directly after participation. The AI is programmed to never harm participants, but rather to encourage and help them to complete the task while remaining friendly at all times. No personal identifier data was stored about the user (data collection has been fully anonymized).



 §.§ Measures

We proxy motivation based on how intensely players engage with the experiment and how likely it is that they complete all the stages. We measure the seconds spent per level by calculating the time before leaving the experiment divided by the number of levels completed. A lower value of seconds spent per level therefore implies that the participant spends less time on a level on average, and therefore engages more intensely with the experiment. The motivation of the participants is measured by using a performance-based behavioural measure. Performance-based behavioural measures indicate both a higher outcome-focused motivation (extrinsic motivation) and process-focused motivation (intrinsic motivation) <cit.>.

In addition, we use a dummy variable to indicate if a participant has completed all 15 stages. The time before ending the participation is also measured - participation can end by either completing the 15 stages or leaving earlier. The speed at which players complete the experiment therefore indicates if the avatar's appearance influences either the intrinsic process-focused motivation or extrinsic outcome-focused motivation.

To get a more reliable measure for the gender of the avatars, we asked each participant to rate their avatar on a 5-point scale, with 1 being ‘very feminine’  and 5 being ‘very masculine’. This gives an average score for each avatar between 1 and 5. To simplify the analysis, we calculated the average rating score of the avatar. Given that the average of our 1-5 scale is 3, we mapped the avatar to either feminine when the average score is below 3 or masculine when the average score is above 3. If a score is exactly 3, the avatar is treated as having no gender and is excluded from the analysis about gender.



 §.§ Analytical strategy

To test the hypotheses, we used Ordinary Least Squares (OLS) from the statsmodels package <cit.> in Python. For each variable, the dependent variable is the seconds spent per level, modelled using OLS. The independent variable is AI avatar presence, AI avatar masculinity and AI disclosure for respectively H1, H2 and H3.




§ RESULTS

During the 2-week time window, 126,013 participants started the experiment. From this group, 8488 participants (6.74%) completed all levels of the experiment.

In Figure <ref> the progress of participants over the experiment is displayed. In Figure <ref>, the time spent in the experiment before either leaving or completing the experiment is displayed. The mean time spent in the experiment is 249.4 (σ = 287.6), while the median time spent is 142 seconds. The mean time including the standard deviation is plotted in the error bar displayed in Figure <ref>.







53,099 (42.14%) participants left the experiment during the first level. This implies that participants did not finish a level. Since they did not finish a level, participants never completed a task. This data is left out of our analysis because there is no indication that participants did understand the task well enough or that their motivation was affected by the treatment they were subjected to.

Based on previous observations within the game, it is not possible to complete a stage within 5 seconds when you did not play the level before (based on loading times, time to search the button and the time to click on it). If a participant completed a stage within this 5-second timeframe, it is likely that the participant used some kind of cheating or did know the location of the button beforehand. Therefore, we left out the data of players from which the seconds spent per level (the average time spent per level) is below 5.0 seconds. This was the case for 11 (0.01%) participants. This makes the total sample for analysis consist of 72,903 participants, of which 8488 (11.64%) completed the experiment.

In Table <ref>, an analysis of the difference between participants who did and did not complete the experiment could be found. 11.6% of the participants completed all levels of the experiment, with the average participant completing 3.67 levels (when excluding participants who did not complete a single level).




As mentioned in the methodology, there is no information available about the age of individual participants. However, if we assume the participants in the experiment follow the same demographics as the game, nearly 50 per cent of the participants are 13 or younger, with an additional 17 per cent being aged between 13 and 17. There are no other game-level demographics available.

Out of the participants, 23,079 (31.66%) participants were in the control group, 24,894 (34.15%) participants played together with an AI and did not know they were playing with an AI and 24.930 (34.20%) participants played together with an AI and knew they playing with an AI. The distribution of both the self-chosen avatar and the assigned AI avatar of the groups who played with an AI are listed in Table <ref>. Over 50% of the participants chose the first avatar as their avatar.



To assess the gender of the avatars, we took the mean of all rankings given for that particular avatar. If the average avatar score was under 3, it was categorized feminine, while it was categorized masculine when the average avatar score was over 3. The categorization of all avatar types is listed in Table <ref>.





 §.§ Hypothesis Tests

We conducted an ordinary least squares regression analysis to estimate the relationship between the presence of an AI avatar (treatment group) and the seconds spent per level (H1: Participants who play with an AI avatar (treatment group) have a higher intensity of playing than those who play alone (control group)). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.01, F(1, 72901) = 734.4, p < 0.001, 95% CI [12.033, 13.909]) for intensity of play. Participants’ seconds spent per level increased by 13.0 seconds per level when playing together. That is equivalent to 0.22 standard deviations. This means that participants who played together with another (AI) avatar played less intensely. This is the inverse of what we expected, and therefore we reject H2b.

To test the influence of the effect of the gender appearance of the AI avatar, we conducted an ordinary least squares regression analysis (H2: Participants playing with an AI avatar that appears more masculine have a higher intensity of playing).

We found a significant effect for seconds spent per level (R2 = 0.000, F(1, 72901) = 10.03, p = 0.002, 95% CI [0.841, 3.570]). Participants’ seconds spent per level increased by 2.2 seconds per level when playing together with a more masculine AI avatar. That is equivalent to 0.04 standard deviations. This means that participants who played together with an AI avatar that appeared more masculine played less intensely. This is the inverse of what we expected, and therefore we reject H2.

To test the hypothesis about the effect of participants knowing that they are playing with an AI avatar on their intensity and duration of play, we conducted an ordinary least squares regression analysis (H3: Intensity of play is negatively associated with participants knowing that they are playing with an AI avatar). In Figure <ref>, the seconds spent per level grouped by the treatment type is displayed.



We found a significant effect (R2 = 0.000, F(1, 49822) = 6.957, p = 0.008, 95% CI [-2.569, -0.379]) for seconds spent per level. Participants’ seconds spent per level decreased by 1.5 seconds per level when the other avatar was disclosed as AI. That is equivalent to 0.02 standard deviations. This means that participants who played together with an AI avatar that was disclosed as such, play more intensely. This is opposing to what we expected.

In addition, we did not find a significant effect (R2 = 0.000, F(1, 48007) = 0.803, p = 0.370, 95% CI [-0.003, 0.008]) that playing together with an AI avatar disclosed as such increases the value of seconds spent per level, compared to playing alone.




§ DISCUSSION

This study researched various independent variables that could be used by an avatar controlled by Artificial Intelligence (AI) to improve users’ motivation. Our findings show that in our experiment and game, the masculinity of the users’ avatar makes participants play more intensely. Furthermore, playing together with another avatar controlled by AI significantly affects users’ motivation. Besides that, the masculinity of the AI’s avatar significantly increases the intensity of play. When users learn that the other avatar is controlled by AI, users had a higher probability of not completing the experiment and played more intensely. Finally, the difference between the user’s gender appearance and AI’s avatar gender appearance moderates the effect of the AI’s avatar gender appearance and makes users play more intensely.



 §.§ Contribution to existing literature

Previous studies have shown that cooperating with others can increase motivation <cit.>. Our research shows that playing together with an avatar controlled by an AI makes players play more intense. Our findings suggest that the existing theory, which states that playing with someone else increases motivation, can be extended to include playing with an avatar controlled by AI. Future research in this field could focus on replicating our findings in other games and bringing these effects to non-gaming virtual worlds, to see if our findings are also applicable to other settings, such as virtual workspaces.



 §.§ Practical implications

Companies that create virtual worlds, such as Meta, could use this research when designing avatars in the virtual world. If the sole goal would be to maximize player intensity, it would be best to use a masculine avatar for the AI and to tell users that they are interacting with an AI, in order to motivate users in their virtual world. However, the results of this study also show the need for diversity in virtual worlds. We have seen that the similarity between the participants’ avatar gender appearance and the AI avatar gender appearance moderates the effects of the AI avatar gender appearance on motivation. Therefore, preset genders of both the own avatar gender and the AI avatar gender can limit the effect of the AI avatar on motivation. Diversity in both avatars, such as being able to choose the avatars from a diverse list or being able to freely create the avatars, could help the user in motivation within the virtual world. Beyond gender, ensure that the avatars communicate attributes players are looking for, especially if you want to promote gender diversity in AI collaborators, whilst still stimulating motivation. This requires more research to investigate what other attributes might affect the motivation of players, as well as a broader coverage of the gender continuum, for instance by introducing more androgynous characters.



 §.§ Limitations

Our AI avatar movement was simple. The avatar moved to preset points and did not interact with participants in another way, such as talking or following them. This could have limited the degree to which participants bestow agency to the AI. Furthermore, we don't theorize or study how the details of the interaction could change the user's perception regarding the agency of the AI  <cit.>. To address this limitation, future research could use a more naturally behaving AI, introduce different levels of intelligence, and observe the impact on the user's behaviour.

Besides the quality of the AI movement, the task and its setting might have influenced the outcomes of the study. The task was a simple search task, which could be seen as a more mathematical task, and the levels were themed around science. This particular task and setting could have led to a higher base motivation of some groups of participants, which could have worked through the measurements. Future research could be changing the task and setting, while keeping the rest of the methodology the same, to investigate if the task and setting influences the effects we found.

Finally, the selection of avatars was limited to only five avatars, which restricted the gender expressions in the avatars and diversity in skin colour. A participant raised this concern, which did not feel represented by the available avatars due to skin color. We decided to not include diverse avatars because participants should choose based on gender, not on other avatar features. However, this non-diverse set of avatars could have led to dissatisfaction with the experiment and decreased motivation regardless of the treatment type for some participants. To address this limitation, future research could include a wider range of avatars to choose from or allow players to use their avatar.



§ CONCLUSION

This study aimed to describe how AI avatars inside virtual worlds could be used to increase the motivation of a user. We conducted a large-scale field study inside a game where we were able to control variables in the virtual world. We found that playing together with an AI avatar makes users play less intensely. When this AI avatar appears more masculine, users play less intensely. The AI avatar being disclosed as an AI make users play more intensely. Further research could include replicating these results in other games & non-gaming virtual worlds and researching what the causes are for the effects we found.





  §.§.§ 

Boele Visser is co-owner of the game in which the research took place. The other authors have no competing interests.









splncs04









 Paper Title: Deceptive AI systems that give explanations are more convincing than honest AI systems and can amplify belief in misinformation.


Priority and Stackelberg Game-Based Incentive Task Allocation for Device-Assisted MEC Networks
    Yang Li2, Xing Zhang2 1, Bo Lei3, Zheyan Qu2, Wenbo Wang2 
2Key Laboratory of Universal Wireless Communications, Ministry of Education

2Beijing University of Posts and Telecommunications, Beijing 100876, China 

3Beijing Branch of China Telecom Co., Ltd., Beijing 100032, China 

1Email: zhangx@ieee.org

    August 5, 2024
=====================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Advanced Artificial Intelligence (AI) systems, specifically large language models (LLMs), have the capability to generate not just misinformation, but also deceptive explanations that can justify and propagate false information and erode trust in the truth. We examined the impact of deceptive AI generated explanations on individuals' beliefs in a pre-registered online experiment with 23,840 observations from 1,192 participants. We found that in addition to being more persuasive than accurate and honest explanations, AI-generated deceptive explanations can significantly amplify belief in false news headlines and undermine true ones as compared to AI systems that simply classify the headline incorrectly as being true/false. Moreover, our results show that personal factors such as cognitive reflection and trust in AI do not necessarily protect individuals from these effects caused by deceptive AI generated explanations. Instead, our results show that the logical validity of AI generated deceptive explanations, that is whether the explanation has a causal effect on the truthfulness of the AI's classification, plays a critical role in countering their persuasiveness – with logically invalid explanations being deemed less credible. This underscores the importance of teaching logical reasoning and critical thinking skills to identify logically invalid arguments, fostering greater resilience against advanced AI-driven misinformation.









§ MAIN

Artificial Intelligence (AI) systems, such as large language models (LLMs), have the alarming capability to generate not only misinformation but also deceptive explanations that justify misinformation and make it seem logically sound.

Researchers have identified an increase in AI-generated disinformation campaigns <cit.> and the factors that make them disruptive to people' ability to discern true and false information <cit.>, as well as change people's attitudes <cit.>. These factors include authoritative tone <cit.>, persuasive language <cit.>, and targeted personalization <cit.>. However, little is known about the influence on people's beliefs when explanations are used as a tactic for misinformation.

In the broader context of the AI-generated misinformation landscape, we can identify three levels of sophistication, each posing unique challenges. The most basic level involves AI systems generating false news headlines without any accompanying justification. While such headlines can still mislead, they are more easily dismissed as baseless claims. The next level involves AI systems providing deceptive classifications, labeling false information as true or vice versa. This added layer of perceived authority can lend unwarranted credibility to the misinformation. However, the perhaps most subtle and deeper level is when AI systems generate deceptive explanations to justify and propagate false information.

One of the reasons why the topic of AI generated explanations and misinformation remains unexplored is that the use of explanations as a tactic for misinformation goes against the commonly held beliefs that explanations always make AI systems more transparent, trustworthy <cit.>, and fair 
 <cit.>. While researchers have shown that honest explanations can assist people in determining the veracity of information <cit.> and improve their decision-making outcomes <cit.>, as well as reduce human overreliance on AI systems <cit.>, research in psychology has demonstrated that even poor explanations can significantly impact people's actions and beliefs <cit.>. This implies that the mere presence of an explanation can lead to changes in beliefs and behavior, regardless of its quality or veracity. Researchers have also shown that people often do not cognitively engage with the content of the explanation unless they are forced to do so<cit.>.

Deceptive explanations can be used to exploit this vulnerability by connecting correct facts in misleading ways to justify false information or discredit real information and potentially making it more difficult to discern the truth. Disguised as political commentary, scientific explanations, or online discussions, AI-generated deceptive explanations from systems like LLMs could be weaponized at scale by bad actors to manipulate public opinion and decision making, even with safety measures built into the models <cit.>. 

The low cost and easy scalability of AI-generated explanations further compound the problem. With the ability to generate vast amounts of high quality content quickly and cheaply, malicious actors can flood online platforms with deceptive explanations, discrediting or drowning out legitimate explanations by human experts that take time to craft, making it harder for individuals to make sense of reliable information. 

This paper investigates the effects of AI-generated deceptive explanations on human beliefs. Through a comprehensive pre-registered online experiment with 23,840 observations from 1,192 participants, we find that not only are deceptive AI generated explanations more persuasive than deceptive AI classifications without explanations, they are even more persuasive than honest explanations (See Fig. <ref>). While personal factors such as cognitive reflection and trust in AI do not necessarily protect individuals from the effects of deceptive AI generated explanations, the logical validity, that is whether the truthfulness of the conclusions or classifications made by the AI system follows from the truth of the explanation, of AI generated deceptive explanations where found to counter their persuasiveness – with logically invalid explanations being deemed less credible. This research underscores the urgent need for vigilance and proactive measures to ensure the responsible use of AI technology.








§ METHODS






 §.§ Stimuli Curation



We created a dataset of headlines each with one honest and one deceptive explanation by prompting the text-generation model GPT-3 davinci 2 with 12 example explanations randomly sampled from the publicly available fact-checking dataset “liar-plus” <cit.>. This dataset consists of 12,836 short statements with explanation sentences extracted automatically from the full-text verdict reports written by journalists in Politifact (see Fig. <ref>). 


First, 5 honest and 5 deceptive explanations were generated for 40 true and false headlines by prompting GPT-3 (davinci, temp = .7) with the headline and making it complete the sentences “This is FALSE because…” or “This is TRUE because…” (see Fig. <ref>). We further curated the explanations by ranking them by highest semantic similarity and lowest repeated-word frequency. We picked the highest ranked explanations, confirmed the veracity and logical validity of each explanation. The truth veracity was confirmed independently by each of the authors after being instructed by a professional fact-checker following standard fact-checking procedures <cit.> and then aligned. The logical validity was also confirmed by each author independently by deconstructing the claims of each headline and explanation into premises, conclusions and inferences from which the logical validity could be determined in an almost mathematical fashion using a Fisher analysis <cit.>. Next, we then excluded explanations whose veracity did not match the veracity of the headline. Since the resulting dataset had an unequal distribution of veracity and logical validity, we randomly excluded generated explanations until we had somewhat equal distribution of true and false explanations and logically valid and logically invalid explanations for each condition (deceptive vs. honest explanations) ending with a stimulus set consisting of 28 headlines with 1 honest explanation and 1 deceptive explanation each (56 total). We tested for differences across four linguistic dimensions (word count, sentiment, grade level, and subjectivity) and found no statistical differences between conditions. To generate explanations for a Trivia stimulus set, we repeated the same procedure by prompting GPT-3 with 12 example explanations. The example explanations and Trivia statements were randomly sampled from online<cit.>, resulting in a stimulus set consisting of 28 trivia statements with 1 honest explanation and 1 deceptive explanation each (56 total). These were also tested for differences across four linguistic dimensions (wordcount, sentiment, grade level, and subjectivity) and were also found to have no statistical differences between conditions.






In real-life scenarios, large language models may generate explanations without the filtering specified above. However, in this experiment, the filtering algorithm serves to ensure the quality and validity of the explanations for the purpose of our experiment. The aim was to isolate the effects of deceptive explanations and see how they manipulate beliefs even when at their best. Hence, it was important to minimize other variables such as irrelevance or incoherence of the explanations to get a normalized sample. Nonetheless, since the filtering algorithm is automatic, it has no selection bias and could potentially reflect how LLMs could be used in a controlled manner, either by malicious actors crafting algorithms to filter the best explanations to use in misinformation campaigns or by organizations using AI for generating honest explanations for automated fact-checking. In either contexts, it is very likely for there to be a layer of curation or filtering applied to create more convincing or high-quality content.





 §.§ Participant Recruitment


We recruited 1,199 participants through Prolific, <https://prolific.co>. Participants were required to self-report as US citizens, fluent in English and rated their fluency in any other languages they spoke. 1192 of these individuals passed an initial attention check task and were allowed to proceed. Additionally, all participants were fluent in English and 142 had fluency in a second language. Our final sample had a mean age of 39%, was 50% female, and was 72% white. 



 §.§ Task Description



Participants were shown 20 statements during the main discernment task which were either true or false. Each participant saw the 20 statements in a random order, and rated the perceived truth of each statement (“Do you think the statement in the grey box is true or false?”) on a slider scale with 1 decimal from 1 (“Definitely False”) to 5 (“Definitely True”). After the rating, the participants would receive feedback from an AI system and be asked if they want to revise their rating (“Would you like to revise your estimate: Do you think the statement in the grey box is true or false?”) on a slider scale with 1 decimal from 1 (“Definitely False”) to 5 (“Definitely True”) with the default value being same as the previous rating. Participants also rated their knowledge on the topic (“How knowledgeable are you on the topic of [topic]”) on a slider scale with 1 decimal from 1 (“Not at all knowledgeable”) to 5 (“Very much knowledgeable”). The selection of statements and generation of AI feedback is further explained in section <ref> and the task interface can be seen in Fig. <ref>.






 §.§ Randomization



For the main discernment task, participants were randomly assigned to one of two conditions: (i) news headline statements or (ii) trivia item statements (between-subjects); and one of two conditions (i) no explanation (“This is true / false”), or (ii) explanation (“This true / false because…”) (between-subjects). The order of the stimuli being presented was also randomized. See Fig. <ref> for examples of items across conditions. 



 §.§ Post Task Survey



After the discernment task, participants were asked to complete post-test surveys to measure their critical thinking, and level of self-reported trust in the agent providing them with explanations. To measure the level of critical thinking of subjects, we used cognitive reflection test (CRT), a task designed to measure a person's ability to reflect on a question and resist reporting the first response that comes to mind <cit.>. For the CRT we randomly sampled three items from the extended CRT <cit.>. Finally, following Epstein et al. <cit.>, to assess trust in the AI agent participants answered a battery of six trust questions derived from Mayer, Davis, and Schoorman <cit.>’s three factors of trustworthiness: Ability, Benevolence and Integrity (ABI).




 §.§ Participants

A total of 1,199 individuals participated in the experiment. We used the Prolific platform to recruit individuals from the United States. We focus our analysis on the 1,192 of 1,209 recruited participants who passed the attention check. 589 participants rating news headlines and 610 participants rating trivia statements. Of the 589 participants rating news headlines, 289 received no explanation and 300 received an explanation. Of the 610 participants rating trivia statements, 299 got no explanation, and 311 got explanations. Each participant rated 20 statements, with an average of 51% of statements being true and 50% of explanations being deceptive.



 §.§ Approvals

This research complies with all relevant ethical regulations and the Massachusetts Institute of Technology’s Committee on the Use of Humans as Experimental Subjects determined this study to fall under Exempt Category 3 – Benign Behavioral Intervention. This study’s exemption identification number is E-3754. All participants are informed that “This is an MIT research project. All data for research is collected anonymously for research purposes. We will ask you about your attitudes towards information and AI systems. For questions, please contact vdanry@mit.edu. If you are under 18 years old, you need consent from your parents to continue.” Participants recruited from Prolific were compensated at a rate of $10.82 an hour. At the end of the experiment participants were made aware that they had received AI explanations that were sometimes deceptive in the experiment, being told that "In this study, you were asked to collaborate with an AI-system for rating the accuracy of statements. All feedback in this study was AI-generated. Some of the feedback from the AI system was simply deceptive". Future work must explore the limits, ethics and consequences of exposing participants to AI-generated content.



 §.§ Scoring Perceived Truthfulness and Perceived Logical Validity

To investigate the effects of semantic features of explanations, we used GPT-4 to split each explanation into individual propositions / premises and score the perceived truthfulness and logical validity of each premise. For instance, for the headline “Newsmax plans expansion to capitalize on Trump support, anger at Fox News”, the explanation “This is TRUE. Newsmax has been a vocal supporter of Trump, and the network has even hired Trump's former campaign manager.” was split by GPT-4 into the two premises: “Newsmax has been a vocal supporter of donald trump.” and “the network newsmax has hired donald trump's former campaign manager.”. To conduct the splitting GPT-4 was given the following prompt: “Split the following explanation into its containing claims. Fill in pronouns and references so that each claim can be verified by itself without any context. Separate them with a new line. Only give the answer and be as concise as possible. Example: example. Explanation:explanation”.

Each premise of each explanation was then scored by GPT-4 for its perceived truthfulness using the following prompt: “How likely is it that someone perceives this as true on a scale from 0.00-1.00 with decimals. 0.00 being extremely unlikely and 1.00 being extremely likely. Only give me the answer. Even if this is highly subjective how do you think some person might think this to be likely: premise”. Next, each premise was then scored by GPT-4 for its perceived logical validity with the deceptive classification of the headline using the following prompt: “How likely is it that the general public would believe this: 'premise' to support this: 'This is classification: headline'. Output a score from 0.00-1.00 with decimals. 0.00 being extremely unlikely and 1.00 being extremely likely. Only give me the answer:”. An average score for both truthfulness and logical validity was calculated for each explanation.



 §.§ Calculating Syntactic Features

To calculate word count and reading ease, we used the Natural Language Toolkit (NLTK) in Python. The word count was simply the total number of words in each explanation. Reading ease was estimated using the Flesch Reading Ease formula, which assesses text on a 100-point scale; the higher the score, the easier the text is to understand.

To calculate the grammatical correctiveness of the explanation, we used the LanguageTool Python library, which returned the count of grammatical errors it could detect in the text. 



 §.§ Analysis



In order to gain insights into whether explanations lead to more accurate beliefs, we first compare the belief ratings of user with AI-generated explanations (honest and deceptive) and no feedback. Accuracy was coded by subtracting the belief ratings from the ground truth per rating per participants.

To investigate the relationship between statement ground truth, the presence of explanations, and deceptive classifications (X1, X2, and X3) and belief and accuracy distribution (Y1, and Y2), along with their potential interactions and moderator variables, we employed an ordinary least squares (OLS) regression models.

We analyzed a total of 23,980 observations of belief ratings (12,200 trivia statement observations and 11,780 news headline observations) of true and false statements, collected through our experiment. The response variables, Y1 and Y2, represent the participant belief and accuracy distributions, while the predictor variables are as follows: X1 - statement ground truth, X2 - presence of AI explanation, and X3 - deceptive AI feedback. The moderator variables include the following z-scored variables: logical validity, self-reported prior knowledge, cognitive reflection test score <cit.>, and trust in AI systems<cit.>.

We chose an Ordinary Least Squares (OLS) regression model for its analytical rigor in assessing linear relationships and controlling for confounding factors and mediators, making it ideal to intricately dissect the effects of AI explanations on belief accuracy and to unravel the complex dynamics between statement truth, explanation presence, and deception effectively. We assume that errors are independent and normally distributed with constant variance. Interaction terms between the predictors (X1:X2, X1:X3, and X2:X3) were included in the model to examine any joint effects of the veracity and explainability factors on belief distribution (Y1), accuracy distribution (Y2), and discernment distribution. Belief refers to the participants' subjective judgments about the truthfulness of the statements, while accuracy represents the correctness of these judgments (i.e., whether participants correctly identified true and false statements). We pre-registered our analysis at <https://aspredicted.org/YLK_S3F>.

Our moderation analysis for CRT, need for cognition, trust, and prior knowledge was conducting by, for each moderating variable, re-running the main analysis model with the addition of the z-scored moderator and all interactions. Additionally, our moderation analysis also examined the 4-way interaction between veracity, explanation veracity, explanation type and the moderator.

We also conducting a moderation analysis for logical validity by re-running the main analysis model restricting to the classification + explanation condition, with the addition of the z-scored moderator and all interactions (limiting to only AI classifications with explanation) and examining the 3-way interactions between veracity, explanation veracity, explanation type and the moderator.

We then conducted a moderation analysis for number of premises, perceived truthfulness and perceived logical validity running the main analysis model restricted to deceptive explanations examining the interactions between headline veracity, number of premises, average perceived truthfulness of the premises and average perceived logical validity of the premises.

Lastly, we conducted a moderation analysis for the word count and reading ease by running the main analysis model restricted to deceptive explanations and examining the interactions between headline veracity, word count, and reading ease of the explanations.




§ RESULTS


In order to study the effects of deceptive AI explanations on human beliefs, we conducted a pre-registered online experiment with 23,840 observations from 1,192 participants rating their beliefs in true and false news headlines before and after receiving AI generated explanations. We designed our experiment to disentangle the influence of deceptive AI explanations from merely receiving a deceptive (inaccurate) classification of a news headline as true or false without explanation. This was done by randomizing participants to receiving true and false news headlines accompanied by either deceptive classifications or deceptive classifications with deceptive AI explanations (between-subjects). Moreover, we examine the impact of logical validity of the explanation, and how personal factors and the effects of syntactic and semantic features of the deceptive explanations may influence the outcomes.

For each true and false news headline, an LLM model (GPT-3) was used to generate an honest and a deceptive explanation by prompting the model to complete the following statements: “This is false because…” or “This is true because…”. For deceptive explanations, the model generated inaccurate explanations stating why true statements were false and why false statements were true. This was done for both news headlines and trivia statements. Examples of the generated classifications with explanations can be found in Appendix <ref>.



 §.§ Deceptive AI-generated Classifications with Explanations are more Persuasive than Honest AI-generated Explanations


To understand the persuasiveness of deceptive AI-generated explanations, we compared the relative persuasiveness of deceptive and honest explanations on belief change (i.e. the absolute difference in rating from before and after seeing an AI classification with explanation per item per participant). We find that deceptive AI-generated explanations are significantly more persuasive than honest explanations on both true and false news headlines (β = 0.40, p < 0.0001 and β = 0.29, p = 0.003, respectively, not pre-registered). Prior research has shown that fake news is shared more often that true news due to factors such as novelty and emotional content (fear, disgust, and surprise) <cit.>. It is likely similar factors are at play for deceptive explanations, potentially explaining why they were found to be more persuasive than honest explanations. The full regression table can be found in Table <ref> in the Appendix.





 §.§ Explanations Can Amplify Beliefs in False Information


To ensure our results were caused by explanations and not simply labeling information as true or false, we compared the influence of AI feedback with and without explanations (deceptive AI-generated explanations and deceptive AI generated classifications, respectively). 

Our results show that deceptive AI generated classifications without explanation, significantly increase belief in false news headlines (β = 0.71, p < 0.0001) and decreases belief in true news headlines (β = -1.72, p < 0.0001). In extension, when accompanied by deceptive AI-generated explanations, beliefs in false news headlines were further significantly increased (β = 0.32, p = 0.009) and beliefs in true news headlines where further significantly decreased as compared to just deceptive classifications (β = -0.72, p = 0.0001). These results suggests that the effects of deceptive AI systems amplifies beliefs in information beyond classifications without explanations (Fig. <ref>). The full results can be found in Table <ref> in the Appendix.







 §.§ Personal factors moderate the influence of deceptive AI explanations

In previous studies, cognitive reflection as measured by the Cognitive Reflection Test <cit.> has been found to associate with people's ability to correctly identify misinformation <cit.>. 

However, when receiving AI feedback on the truth of news headlines, our results revealed no significant interactions with cognitive reflection level and deceptive classifications both with and without explanations for false news headlines (β = -0.09, p = 0.20 and β = 0.13, p = 0.15, respectively) and for true news headlines (β = 0.16, p = 0.20 and β = 0.25, p = 0.15, respectively). This suggests that introducing an evaluative AI system framed as a fact-checking system could override the effects of cognitive reflection on truth discernment of news headlines. This may be attributed to several factors. First, the presence of information labeled as provided by “AI” might induce a reliance effect, where individuals defer judgment to the technology <cit.>, potentially undermining their reflective capacities <cit.>. This effect could be accentuated by the perceived authority or credibility of language based AI systems, which might attenuate the influence of cognitive reflection <cit.>. Additionally, the complexity or novelty of information such as new facts they have no knowledge of revealed in AI explanation could confuse or overwhelm users, reducing the effectiveness of their cognitive reflection in evaluating the information <cit.>. Lastly, it is also possible that individuals with high cognitive reflection are already near their maximum ability to discern truth from falsehood, resulting in a ceiling effect that leaves little room for AI generated explanations to induce cognitive reflection.

Trust in AI systems has in previous literature been highlighted as an important feature moderating the effects of explanations on people's beliefs <cit.>. Our results showed a significant effect of self-reported trust in AI systems on participants' belief ratings when getting deceptive classifications without explanations on true (β = -0.64, p = 0.0001) and false news headlines (β = 0.38, p = 0.0001). However, we did not find any increased effects of trust on participants' belief ratings when getting deceptive classifications with explanations on true (β = -0.11, p = 0.79) and false news headlines (β = -0.01, p = 0.59), indicating that trust in AI systems does not significantly increase the persuasion effects when receiving deceptive AI-generated explanations from just getting deceptive classifications.

Lastly, research has highlighted prior knowledge as a significant predictor of correctly identifying fake news <cit.>. However, it is unclear whether these effects extend to AI-generated deceptive explanations. To evaluate these effects, we had participants rate their own perception of their knowledge of a news headline after rating their belief in the news headline. When receiving AI classifications without explanations that were truthful (honest), self-reported prior knowledge was associated with increased beliefs in true news headlines (β = 0.44, p < 0.0001) and associated with decreased beliefs in false headlines (β = -0.11, p = 0.001). 

Conversely, when receiving deceptive classifications without explanations, prior knowledge was not found to have any significant effects on beliefs in false and true news headlines (β = -0.05, p = 0.54 and β = 0.02, p = 0.86, respectively). However, while there were no significant effects of deceptive AI-generated explanations on true news headlines (β = -0.19, p = 0.19), self-reported prior knowledge was associated with significantly increased beliefs in false news headlines when receiving deceptive AI-generated explanations (β = 0.18, p = 0.02). This suggests that individuals who report themselves as knowledgeable on a news headline might not necessarily be more resilient to deceptive AI classifications on true news headlines, and, in fact, might even be more susceptible to believing false news headlines when given deceptive AI explanations. One possible explanation for these results could be due to what is known as overconfidence bias <cit.>, where those who voice their perceived knowledge level to be high, could overestimate their ability to critically evaluate the AI system's outputs or to identify false information correctly. Future research should compare these results with an objective assessment of people's prior knowledge to more accurately detail the differences.
The complete results can be found in Table <ref> (CRT), Table <ref> (trust), Table <ref> (prior knowledge) in the Appendix.





 §.§ Deceptive AI-generated explanations that are logically invalid decrease people’s beliefs in false news headlines



Researchers have suggested that people's ability to identify logical flaws (or logical fallacies) could play an essential role in refuting misinformation <cit.>. In order to investigate the influence of logical validity of deceptive AI-generated explanations, we modeled the influence of logical validity of explanations on participants' belief rating using a linear regression, where logical validity is when the truth of the AI generated explanation necessarily implies the truth of the AI generated classification (logically valid) in comparison to where the truth of the classification is independent from the truth of the explanation (See Section <ref>). Limiting the data to only ratings where explanations were present, our results show that while logically invalid deceptive AI-generated explanations did not have any significant effects on participants' belief rating of true news headlines (β = -0.31, p = 0.13), logically invalid deceptive AI-generated explanations did significantly increase beliefs in false news headlines (β = -0.35, p = 0.02). This suggests that participants are more likely to reject deceptive explanations for false news headlines when the explanations are logically invalid. This demonstrates the potential for logical analysis and critical thinking skills to mitigate the influence of deceptive AI-generated explanations for false information. An overview of the results can be found in Table <ref> and Fig. <ref>.



 §.§ Limitations and Potential Generalization to Future AI Systems



As exemplified in this study, personal factors mediate the influence of explanations on humans beliefs. Extending upon these findings, research has shown that prior beliefs about AI systems can significantly influence how people integrate or reject AI-generated information <cit.>. Taking this into account, the impact of deceptive AI explanations might vary significantly across different cultural and contextual settings. Factors such as political climate, prevalent media literacy, and language around AI can influence how deceptive explanations are received and believed. For instance, research has shown that the choice of terminology significantly influences people's perceptions and reactions towards AI-generated content, with different terms leading to varying degrees of accuracy in identification and emotional response across different cultural contexts <cit.>. Moreover, while research has shown LLMs to be significantly more authoritative, persuasive, seemingly logically valid and even preferred over human-authored content, it is unclear to which extent the effects of deceptive explanations identified in this paper would transfer to human-authored deceptive explanations.



Second, our study utilized GPT-3, the most advanced language model available during the experimental period, to generate explanations. Although more sophisticated models have since surpassed its capabilities, our results indicate that even at the GPT-3 level, the model was able to produce deceptive explanations that adversely affected people's beliefs. It is important to recognize that while more advanced models generally exhibit less hallucination and incorporate more robust safeguards for the information they generate, researchers have repeatedly demonstrated that the safety measures implemented in large language models (LLMs) can be easily circumvented through techniques such as "jailbreaks" <cit.> or fine-tuning. For instance, an individual fine-tuned a readily available model on the HuggingFace platform using a dataset of posts from an online forum known for hosting harmful and offensive content, resulting in the generation of more than 30,000 posts on the platform <cit.>. As a result, even with better models becoming available, it likely that not only can these be exploited to generate misleading or deceptive explanations, even with safety measures in place, but they could even be misused to generate deceptive explanations far more persuasive and scalable than demonstrated here. As LLMs continue to advance and become more sophisticated, it is crucial for researchers and developers to remain vigilant in identifying and addressing potential vulnerabilities to ensure the responsible deployment of these technologies.



§ CONCLUSION

Our findings underscore the significant impact that deceptive AI-generated explanations can have on shaping public opinion and influencing individual beliefs. The ability of these explanations to not only present misinformation but also provide seemingly logical justifications makes them particularly potent tools for misinformation campaigns. This is especially concerning in the context of political discourse, scientific communication, and social media, where the rapid dissemination and acceptance of false information can have real-world consequences.

The persuasive power of deceptive explanations, as demonstrated in our study, highlights a critical vulnerability in the public's ability to discern truth from falsehood when interacting with AI-generated content. This is compounded by the finding that even individuals who consider themselves knowledgeable are not immune to the influence of these deceptive explanations. In fact, our results suggest that self-assessed knowledge may even increase susceptibility to believing false information when it is accompanied by a deceptive explanation. This could be due to a combination of overconfidence and the sophisticated nature of the explanations that make the misinformation seem credible.

Moreover, the role of logical validity in the effectiveness of deceptive explanations is particularly noteworthy. Our study found that logically invalid explanations were less effective in persuading individuals to believe false headlines, suggesting that enhancing critical thinking and logical reasoning skills could be a viable strategy to combat the influence of misinformation. This aligns with previous research emphasizing the importance of education in logical fallacies and critical thinking as tools for empowering individuals to better evaluate the information they encounter, particularly in digital environments where AI-generated content is prevalent.

While AI has the potential to bring about significant benefits, its capability to generate persuasive, deceptive explanations poses a serious risk to informational integrity and public trust. Our study highlights the urgent need for comprehensive strategies that address the dual aspects of enhancing public resilience against misinformation and ensuring responsible AI development and deployment. 





§ SUPPLEMENTARY INFORMATION


All data, including pre-registration, datasets, explanation prompts, and code generated and analyzed during the current study is available on GitHub  (<https://github.com/mitmedialab/deceptive-AI>), Zenodo (<https://zenodo.org/records/8172056>) and Research Box (<https://researchbox.org/1801   PEER_REVIEW_passcode=BDHVUP>).


unsrt  
 



§ EXTENDED DATA










 §.§ Additional Results




 §.§ Deceptive AI-generated Explanations Increase Beliefs in False Headlines and Decrease Beliefs in True Headlines

In order to gain insights into whether explanations lead to more accurate beliefs, we first compare how accurately participants rated true and false news headlines before (no feedback) and after getting deceptive AI-generated explanations. We coded accuracy by subtracting the belief ratings from the ground truth per rating per participant. Running an analysis of variance (ANOVA), we found that deceptive AI-generated explanations lead to a significantly lower accuracy than no feedback (14 percentage point difference, F(2, 10959) = -36, p < 0.0001, ANOVA Welch).

Breaking these results down into beliefs in true and false news headlines, we find that deceptive AI explanations significantly increase beliefs in false headlines and significantly decrease beliefs in true headlines (β = -2.44, p < 0.001 and β = 1.03, p < 0.001, respectively), suggesting that deceptive AI explanations significantly diminish people's ability to tell true news headlines from false news headlines. 



§ SEMANTIC AND SYNTACTIC FEATURES MODERATE THE INFLUENCE OF DECEPTIVE EXPLANATIONS

Beyond personal factors like cognitive reflection level, trust in AI, and prior knowledge, the manner in which information is structured and presented can greatly affect the way it is perceived and processed by the reader <cit.>. Notably, the influence of semantic and syntactic features on the effectiveness of persuasive communication has been acknowledged; aspects such as readability <cit.>, the amount of words used, the perceived truthfulness, and the perceived logical support of explanations for news headlines have the potential to shape a reader's beliefs and attitudes <cit.>. To understand what makes a deceptive explanation influence people to change their belief about a news headline, we conducted a post-hoc analysis of the semantic and syntactic structures of explanations such as their perceived truthfulness, perceived logical validity, the number of facts that was stated in the explanation, the explanation’s word count, and how easy the explanation was to read.

Our results revealed significant correlations between semantic structures of deceptive explanations and changes in belief about true and false news headlines. In particular, our results show that when an AI system gives deceptive explanations that are likely to be perceived as logically supporting a false news headline being true, people are more likely to update their initial belief and believe that the false news headline is true (β = 1.26, p < 0.001). Conversely, when the deceptive explanation are likely to be perceived as logically supporting that true news headline is false, people are more likely to believe that the true headline is false (β = -1.36, p < 0.01) than when it is not likely to be perceived as logically supporting it being false. We did not find any significant correlations between the perceived truthfulness of deceptive explanations and beliefs in true and false news headlines in our model (β = 0.81, p < 0.30 and β = -0.79, p < 0.10, respectively). The complete linear model with the semantic results can be found in Table <ref> in the Appendix.

For syntactic structures, our results show that when an AI system gives deceptive explanations on true headlines, stating that they are false, the longer the explanation is (i.e. its word count), the more likely people are to believe the true news headline than from before receiving an explanation (β = 0.03, p = 0.004). This indicates that deceptive explanations that are too long might have the opposite effects, and actually make people realize that a true headline is true, hence increasing their rating from before getting a deceptive explanation. Moreover, we also found that how easy a deceptive explanation was to read also significantly correlated with belief changes in true news headlines from before receiving an explanation. In particular, the easier an explanation was to read correlated with a significantly lower belief in true headlines (β = -0.01, p = 0.05). We did not find any significant correlations between word count of deceptive explanations and belief in false headlines (β = 0.006, p = 0.61), or reading ease of deceptive explanations and belief in false headlines (β = 0.0008, p = 0.88). The complete linear model with the syntactic results can be found in Table <ref>.

If we look at the mean perceived logical validity for deceptive explanations on true vs. false headlines, we found that deceptive explanations on true news headlines generally have a higher perceived logical validity than false headlines. This indicates that the LLM might be better at fabricating deceptive explanations with high perceived logical validity for true news headlines than false. Previous studies have found that false news is typically perceived as more novel than true news <cit.>. This might explain the model's capability of constructing more persuasive deceptive explanations for true news than false news, given that novel information might have less established sources to draw from and distort. Therefore, for making people belief true news less, simple tricks like making the deceptive explanations longer or easier to read might might not work as well for deceptive explanations on false news.




  §.§.§ The impact of Deceptive AI explanations on belief vary depending on the context and type of statements




To understand the generalizability of deceptive AI-generated explanations to other domains, we explore their impact on the belief rating of trivia statements. Overall, we find substantial differences between deceptive AI-explanations on trivia statements compared to news headlines. Compared to deceptive AI-generated explanations on false news headlines, our model does not reveal any significant effects of deceptive AI-generated explanations on participants' beliefs in false trivia statements (β = 0.01, p = 0.94, not pre-registered), nor on participants' beliefs about true trivia statements (β = -0.28, p = 0.19, not pre-registered). We do, however, still find significant effects of deceptive classifications without explanations on true and false trivia statements (β = -2.57, p = 0.0001 and β = 1.19, p = 0.0001, respectively, not pre-registered).

For the effects of logical validity on belief ratings, we also do not see any significant effects on false trivia statements (β -0.26, p = 0.50, not pre-registered) compared to the significance we found for news headlines.

Comparing the average self-reported prior knowledge across trivia items and news headlines, we find that participants on average are less knowledgeable on false trivia (mean = 2.25) statements than false news headlines (mean = 2.71) (β = -0.38, p = 0.003, ad hoc), whereas we found no significant differences in knowledge ratings between true trivia (mean = 2.28) and true news headlines (mean = 2.28) (β = 0.01, p = 0.97, ad hoc)). See Table <ref> in the Appendix for the full table. For differences in belief rating associated with self-reported prior knowledge, we found, similar to news headlines, no significant effects for deceptive classifications without explanations on true and false trivia (β = -0.08, p = 0.33 and β = 0.06, p = 0.23, respectively), nor was self-reported prior knowledge associated with differences in beliefs in true trivia statements when getting deceptive AI-generated explanations (β = -0.16, p = 0.33). However, contrary to news headlines, self-reported prior knowledge was not significantly associated with increases in beliefs for deceptive AI-generated explanations on false trivia statements (β = 0.16, p = 0.12). This difference could potentially be due to the significantly lower knowledge people report to have on false trivia statements.

While we observed that deceptive AI explanations have a stronger impact on people's belief compared to deceptive classifications in news headlines, we did not observe the same effect with trivia statements with the deceptive explanation not having significantly stronger convincing power than the deceptive classification by itself. Our hypothesis is that people might be more easily convinced about trivia items than news headlines because they are less high-stakes. Moreover, news headlines are also typically more personal and relevant to individuals, thus requiring additional information for people to be persuaded. On the other hand, trivia statements, which individuals have less personal investment in, might not require explanations to be believed. 

Furthermore, our findings indicate that when individuals receive deceptive AI explanations, they become less adept at discerning when they believe they possess prior knowledge. However, we did not find this to be the case for trivia statements. This might be due to our finding that people have more self-reported prior knowledge on news headlines than trivia statements. Overall, the observed differences between news headlines and trivia statement emphasize that the impact of deceptive AI explanations on belief may vary depending on the context and type of statements. 




 §.§ Results Graphs






 §.§ List of Stimuli






plaintableList of Stimuli shows the news headlines and trivia statements used in the experiment.  
[pages=1-,scale=0.8,pagecommand=]figures/appendix/stimuli.pdf



 §.§ Statistical Analysis


































 Paper Title: PIXELMOD: Improving Soft Moderation of Visual Misleading Information on Twitter.
















: Improving Soft Moderation of
 Visual Misleading Information on TwitterThis paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel^, Chen Ling^, Jeremy Blackburn^, and Gianluca Stringhini^

^Boston University, ^Binghamton University

{ppaudel,ccling,gian}@bu.edu, jblackbu@binghamton.edu

    
==============================================================================================================================================================================




















§ ABSTRACT 

Images are a powerful and immediate vehicle to carry misleading or outright false messages, yet identifying image-based misinformation at scale poses unique challenges.
In this paper, we present , a system that leverages perceptual hashes, vector databases, and optical character recognition (OCR) to efficiently identify images that are candidates to receive soft moderation labels on Twitter.
We show that outperforms existing image similarity approaches when applied to soft moderation, with negligible performance overhead.
We then test on a dataset of tweets surrounding the 2020 US Presidential Election, and find that it is able to identify visually misleading images that are candidates for soft moderation with 0.99% false detection and 2.06% false negatives.



: Improving Soft Moderation of
 Visual Misleading Information on TwitterThis paper is accepted for publication at the 2024 USENIX Security Symposium. Please cite accordingly.
    Pujan Paudel^, Chen Ling^, Jeremy Blackburn^, and Gianluca Stringhini^

^Boston University, ^Binghamton University

{ppaudel,ccling,gian}@bu.edu, jblackbu@binghamton.edu

    
==============================================================================================================================================================================







§ INTRODUCTION

Online users are constantly bombarded with false and misleading information, whether it is inaccurate and shared without malice (i.e., misinformation), or deliberately crafted to achieve a malicious goal, for example by state actors (i.e., disinformation) <cit.>. 
To help their users better distinguish between accurate and misleading or false information, online platforms like Twitter have introduced soft moderation measures, where they add labels to posts that include inaccurate information, with the goal of providing better context to these claims.
Unfortunately, little is known on how these moderation decisions are made by the platforms, and recent work found glaring issues with Twitter's soft moderation approach <cit.>.

While previous work on automated soft moderation <cit.> builds a solid foundation for the soft moderation of textual content, false information is not only spread through text, but images are commonly used to convey false and misleading narratives <cit.>.
While Twitter applies soft moderation decisions to posts containing images, the way in which this is done is not publicly known, and a preliminary analysis that we conducted shows that these labels are not applied uniformly and pervasively.
For example, in Figure <ref>, we show three tweets that contain identical images discussing the same debunked electoral map claiming Trump's landslide victory with 410 votes after elections servers were seized in Germany by the U.S military <cit.>.
The first tweet (Figure <ref>) received a warning label, while the other two tweets containing the same image (Figures <ref> and <ref>) did not receive any intervention, despite discussing the same misleading narrative. 

These observations highlight the need for effective automated approaches to identify image content that should receive soft moderation.
Moderating images, however, poses unique challenges.
First, the sheer amount of data published on social media makes this a particularly challenging task, especially on platforms like Twitter, where 50 million tweets are posted every day <cit.>.
Previous work has dealt with this problem by adapting perceptual hashing algorithms <cit.>.
These algorithms identify visually similar images by leveraging syntactic embeddings, which are lightweight to calculate.
Even if the calculation of hashes is relatively cheap, comparing an image against a large number of potential candidates is inefficient and does not scale.
In addition to the computational costs of comparing images, previous research has shown that image-based false information is highly contextual, and that similar images can be misinformation, satire, or even completely benign based on the context in which they are used <cit.>.
Existing approaches using perceptual hashing lack the nuance to determine this context, making them prone to false positives and unfit for the misinformation moderation use case.





Technical Roadmap.
In this paper, we aim to address the limitations of previous approaches and develop a scalable, performant, and effective system able to analyze images posted on social media and identify candidates for soft moderation. 
We present , an image search system designed to assist platform moderators by flagging visual content that is similar to content they have previously applied moderation labels to. 
takes as seed input a list of images initially moderated by Twitter and encodes the images using perceptual hashing <cit.>.
To allow efficient hash comparisons, leverages Milvus <cit.>, a vector database that is optimized for searching among millions of hash records.
After finding images that are visually similar, leverages OCR to identify the context in which an image is used, allowing it to rule out false positives.

successfully identifies both tweets in Figures <ref> and <ref> when queried with the image in Figure <ref>.
Additionally, not only identifies images identical to the seed images but also visual variants of the images such as memes and fauxtography, which are commonly used for spreading misleading information <cit.>.


Main Contributions & Findings.
addresses the limitations of previous image-based moderation approaches proposed in the academic literature, 
is platform-independent, and only requires a single misleading image already intervened by human moderators.
With the ability to identify thousands of posts spreading misleading images, we show that an image search system like can be used as a foundation for large-scale soft moderation intervention system for images.
While a large amount of research has been conducted on perceptual hashing and OCR, the novelty of our work lies in combining and tuning the two into a working end-to-end soft moderation system which achieves a much larger coverage than perceptual hashing alone.
Besides content moderation, the takeaways offered by our work can be helpful for researchers and practitioners in building image similarity search systems for related security problems like phishing <cit.> and CSAM detection <cit.>.


We compare with alternative approaches to assess visual similarity, showing that outperforms them in terms of F1-score.
In particular, the use of OCR to determine the context of false images allows to operate using a higher similarity threshold than existing approaches only based on perceptual hashing, increasing recall while keeping precision high as well, suffering only a negligible runtime performance hit in return.

We then test on Twitter data discussing the 2020 US Presidential Election.
Starting from a seed set of 959 tweets containing misleading images, we use to retrieve 40,244 tweets in the wild that are candidates for moderation.
Performing a thorough manual analysis, we find has a false negative rate of 2.06% and a false positive rate of 0.99%.
These findings are a positive step towards building automated systems that complement Twitter's existing systems to moderate misleading images and improve the state of content moderation. 

To better understand the way in which soft moderation was adopted by Twitter during that period, we perform a qualitative measurement study of the content flagged by , comparing it to tweets that received soft moderation labels from Twitter.
We categorize the images identified by based on Twitter's Platform Policy, finding that although different types of platform policies were moderated at different rates by Twitter, none of the policy violations exceeded a moderation rate of 5.96%.








§ OVERVIEW OF 


takes a query image that a platform wants to investigate for further moderation actions and retrieves images that are both visually and contextually similar.
This allows a platform to quickly identify a large, high-certainty candidate list of posts for moderation.
has four stages, as illustrated by Figure <ref>: i) generating image embeddings, ii) indexing image embeddings, iii) retrieving visually similar images through approximate nearest neighbor search, and iv) refining contextually matching images via Optical Character Recognition (OCR).
We demonstrate in the context of Twitter, identifying visually misleading information related to the 2020 US Presidential Election.
However, 's architecture is designed to be portable to any social media platform.



 §.§ Background

Identifying visually similar images given a query image is a reverse image search problem, which falls under the broader research area of Content-Based Image Retrieval systems <cit.>.
The goal is to efficiently collect, index, and search for visually similar images over an index
of millions of images, and there are several publicly available solutions (e.g., TinEye <cit.>, Google Vision AI <cit.>).
We conducted a preliminary exploration of these systems and identified two major limitations: 1) they require a paid subscription for programmatic access, and 2) they perform poorly for returning images from Twitter, Facebook, and other social media platforms because of platform restrictions on their crawlers.
To address these limitations, we built a custom image search system focused exclusively on social media that collects images, generates syntactic embeddings (pHash and PDQHash), and indexes them (using the Milvus <cit.> vector database).

Motivating the need for multi-modality in visual similarity search.
Prior works have discussed that incorporating contextual information within images is important when identifying when they have been manipulated or miscaptioned <cit.>.
Zanettou et al. <cit.> discuss similar existing shortcomings of image similarity systems and the need of incorporating semantic components like OCR in image detection pipelines to effectively identify image memes.
Another work studying misinformation images on WhatsApp in India <cit.> used semantic features like OCR from Google Cloud Vision to identify differences between different types of misinformation images (images taken out of context, photoshopped images, and memes) after retrieving images visually similar to fact-checked images.
However, no existing work uses contextual information as a part of the detection system or pipeline itself.

Existing detection systems can identify images that are very similar to misleading images upon querying, but are unable to verify if the identified images are used in the same misleading context as the query image.
We illustrate this case with an example in Figure <ref>.
When we query existing image similarity systems with Figure <ref>, they will typically return Figure <ref>, and Figure <ref> as matching results with strong confidence.
However, Figure <ref>, and Figure <ref> are not related to the query image under question, which contains the text of “Fraud. The Biggest Disgrace In Our History”, despite being visually similar to the query.
Figure <ref> is an image used in a different context, with no connotation to the narrative of “Election Fraud” during the 2020 Elections, whereas Figure <ref> is used in another context: Fox News projecting a win for Joe Biden.
Thus, the results returned are used in entirely different context than the query image, lacking the connotation of “Election Fraud” which makes the query image misleading.
A similar problem occurs when dealing with fauxtography, where images are manipulated with the inclusion of visual changes, overlay text, or are used out of context with the goal of misleading the user <cit.>.
This highlights the need for incorporating context when using image similarity systems to improve misinformation detection systems.

To address this issue, analyzes the overlay text included in images by using optical character recognition (OCR) to better capture the context in which an image is used and then incorporate this context as a part of our image similarity pipeline.
Including this context also helps overcome existing limitations of prior work using perceptual hashes that have to rely on lower similarity thresholds for matching <cit.>.
Incorporating context allows us to increase image similarity thresholds while at the same time achieving better recall without hurting precision.






 §.§ Generating image embeddings

There are two types of embeddings we can extract from images for retrieval purposes.
The first, syntactic embeddings, are fingerprints of an image's visual features, computed such that two visually similar images will have identical, or very similar fingerprints.
The second, semantic embeddings, capture features within the images (for example, if a given image is a chart or a portrait), and match images that are visually dissimilar, but still have similar meanings.

For our purposes, syntactic embeddings are more useful for two reasons.
First, images identified through them are minor variations of the query image, making them easy to interpret and take action on.
Second, semantic embeddings use complex deep learning architectures <cit.>, while syntactic embeddings are relatively fast and cheap to train and use.
This means we can scale to millions of images with relatively few resources (e.g., does not require a GPU.)

We generate our syntactic embeddings via perceptual hashing.
We make use of two different perceptual hashing embeddings: i) pHash <cit.>, and ii) PDQHash <cit.>.
pHash encodes images as 64-bit vectors by extracting geometry-preserving feature points using Discrete Cosine Transform (DCT).
The resulting vectors are invariant to simple image transformations and minor coloring differences.
PDQHash is an improvement over pHash, encoding images as 256-bit vectors.
Prior work studying the spread of misleading images in social media has used both pHash and PDQHash <cit.>.
We evaluate pHash and PDQHash with respect to precision, recall, as well as the time taken to generate an embedding, index it, and eventually query the index.



 §.§ Indexing image embeddings

The most similar image to a query is the one whose embedding is the shortest distance from the query image's embedding.
While we could perform a pairwise comparison with the query image and all candidate images, this is intractable at the scale at which social media platforms operate.
To address this, there are specialized vector databases that can be used to perform queries over embeddings.
For , we chose to use Milvus <cit.> a purpose-built vector management system.
Milvus abstracts away lower-level details (e.g., index creation and item insertion) via a higher-level API and Facebook's Faiss <cit.> library for efficient dense vector similarity search.

Milvus has two types of indexes: i)  and ii) .
 guarantees a 100% recall rate by exhaustively searching for matches to a query embedding.
 is quantization-based, dividing embeddings into multiple cluster units and comparing the query embedding to the center of each cluster.
 is faster than  but requires tuning hyperparameters to find an optimal recall rate.
Milvus recommends using  for datasets in the order of a few million vectors, which fits the datasets we use to evaluate 



 §.§ Retrieving visually similar images

Vector databases like Milvus scale by leveraging approximate nearest neighbor search across their indices.
Milvus determines the nearest neighbors by scoring candidates with a similarity metric.
We use Hamming distance, leaving us with the next challenge of choosing an appropriate threshold to treat as “visually similar.”
We select Hamming distance over cosine similarity as prior work on perceptual hashing have predominantly used this metric to assess visual similarity on both PHash and PDQHash ( e.g. baseline systems <cit.> and  <cit.>).
Additionally, we index the perceptual hashes on Milvus as binary hashes, and using Hamming distance as the similarity metric allows us to efficiently compare the binary representations of images.
Previous works using pHash to study misinformation on a variety of social media platforms have recommended a Hamming distance of six <cit.>, and eight <cit.>.
The developers of PDQhash suggest a distance of 32, which has been used to study misinformation on WhatsApp <cit.>.
Other work using PDQHash has used a threshold of 40 to study the spread of COVID-19-related media in Pakistan over WhatsApp <cit.>.
Ultimately, choosing a threshold depends on the dataset used and the particulars of how “visual similarity” is defined.
In Section <ref>, we experimentally evaluate the choice of the visual similarity threshold (θ_visual) by optimizing over both precision and recall.




 §.§ Refining matched images using OCR

The final component of involves processing the results retrieved as “visually similar” to the query image by comparing text extracted via an Optical Character Recognition (OCR) engine.
For the rest of the paper, we refer to the output text from an image extracted from an OCR engine as OCR label.
The advantages of applying this OCR driven post-processing to the matched results is two-fold: i) it eliminates the limitations of underlying perceptual hashing algorithms by filtering out their false positives and ii) it allows us to explore relatively larger distance thresholds for visual similarity matching, thus allowing us to minimize false negatives.
Prior applications of perceptual hashing algorithms for identifying visually similar images have used conservative thresholds <cit.> to minimize false positives.
In contrast, we aim to exploit larger thresholds to increase our recall as well as have a more tuneable mechanism with respect to precision trade-offs. 
In Section <ref>, we first validate the choice of Google Cloud Vision API as the underlying OCR engine <cit.>, and experimentally evaluate the calibration of the similarity metric for the extracted OCR labels as our OCR engine) and corresponding threshold for similarity (θ_textual).



§ DATASETS



We use three different datasets to evaluate (summarized in Table <ref>).
Our first dataset, , contains 7.6M tweets collected via the Twitter Streaming API using a set of 2020 US Election hashtags (e.g., #ballotfraud, #voterfraud, #electionfraud, #stopthesteal) curated by Abilov et al. <cit.>.
To comply with Twitter's terms of service, this dataset only provides tweet IDs, therefore, we need to retrieve the full tweet content from the Twitter API through a process called hydration.
Out of the 4,017,259 tweets we were able to rehydrate, about 218K have at least one “media” entity: either an image or a video (tweets with videos also contain a thumbnail image, which we include in our dataset).

Our second dataset, , consists of 13M tweets with media collected via the 1% Twitter stream from November 1 to December 31, 2020.
Unlike , we do not filter any keywords, therefore is a uniform 1% sample of Twitter.

Finally, we collect via Twitter's Academic Research full-archive search endpoint in late 2022, by querying context annotations <cit.> associated with the 2020 US Elections.
Twitter annotates tweets with context annotations by semantically analyzing their content and metadata, categorizing them into a nested structure of domains and entity labels.
The more than 80 domains cover things like politics and TV shows, while the nearly 145K entities cover details of the domains ranging from elections to festivals, to politics and media personalities.

We identify two context annotations related to the 2020 election by retrieving the context annotations of all tweets from : 1) “2020 US Election Day” and 2) “2020 US Presidential Election,” both in the Events domain.
We also found other context annotations from the tweets, but they were associated with “Political figures” and “Politics” in general, not the 2020 elections specifically.
To minimize the noise that can be introduced when using such a broad context, we do not include those annotations when building .
In the end, consists of the 6.9M tweets that we retrieve using the “2020 US Election Day” and “2020 US Presidential Election” context annotations and limiting our search to November 1st, 2020 to December 31st, 2020.

Index building.
We build two different Milvus indices for pHash and PDQHash embeddings, using all the images in our datasets.
The index size for both types of embeddings is 19.7M, after combining all the tweets from , , and .







§ SYSTEM VALIDATION



In this section, we present the validation strategy for .
First, we discuss how we build a ground truth dataset. 
Then, we describe our experimental setting to determine the best hashing mechanism, and the best string similarity algorithm for the OCR labels, and the corresponding θ_visual and θ_textual to use with the hashing algorithms and string similarity algorithms.
The goal is to experimentally determine the best operating values for by optimizing over both precision and recall for the overall system.

Building ground truth.
First, we randomly sample 50 images from .
For each of these 50 images, we query both the pHash and PDQHash indexes for similar images, limiting the results to the maximum threshold for each algorithm (10 and 90).
We select 90 as the maximum threshold for PDQHash as its developers experimentally verified it as the upper bound for images that are known to be different.
For pHash, we select 10 as the maximum threshold because previous work has found that higher thresholds produce results that are too noisy <cit.>.
This results in 11,825 unique images retrieved as similar across both indices for the same set of 50 query images.
Then we manually annotate the results using a pairwise image annotation tool developed by <cit.>.
Note that the goal of this annotation was to verify that images were visually and contextually similar to the original one (i.e., their overlay text contained similar words).
For this reason, we did not need to build a codebook and have multiple annotators agree on the results as we did for more subjective experiments later in the paper.
In the end, we find 9,785 images that are similar to our 50 source images.
We refer to these 9,785 images as .

Determining θ_visual. 
Next, we use the images in to determine the accuracy of the results returned by the two hashing algorithms when using different thresholds for θ_visual.
Our Milvus indexes return the closest set of embeddings for a query image, scored by the Hamming distance to the candidate image.
For pHash, we experiment with a range of Hamming distance thresholds from 4 to 10, which is in line with previous works that found an optimal threshold between 6 <cit.> and 8 <cit.>.
For PDQhash, on the other hand, it is recommended to use a range of pairwise distances for identifying similar images <cit.>.
Therefore, we test multiple threshold ranges, some used by previous work using PDQHash <cit.> (32, 48) as well as three additional ranges to take us up to the maximum possible threshold (64, 80, and 90).

Validating Google Cloud Vision API for OCR.
To validate the ability of Google Cloud Vision API to correctly extract the OCR labels in a query image, 
we sample 50 images from and create a ground truth of the text contained in the images.
These are examples of image occurring “in the wild,” which contain text in specialized fonts, small text in lower quality images, or text in artistic fonts, and are therefore a good test for the OCR component of .
On this dataset, the median Jaccard similarity of the ground truth text and the one extracted by Cloud Vision OCR is 1.0, and the mean is 0.95.
This validates that Google Cloud Vision API can be succesfully used as the underlying engine for identifying the contextual text contained in the images of .
Upon doing some error analysis of the OCR text, we find that the output of the Cloud Vision API is not missing any text present in the images, but is sometimes parsed in different order than the ground truth.
This is an artifact of how the OCR engine works on different regions of the image.
We argue that this would not pose a problem when using the system for as the method will work consistently across all the source images and the potential matches retrieved through perceptual hashing.


Determining θ_textual.
After validating that Cloud Vision API can be succesfully used to extract the contextual text contained in images, we aim to determine the appropriate text similarity metric and corresponding similarity threshold to compare the OCR labels of the retrieved visual matches with the query images.
We first check if the query image contains any OCR label, which we call label_query, using Google Cloud Vision API.
If label_query is not empty, then, for each visually matching image, we compute the corresponding OCR labels (label_match) using the same API endpoint.
We experiment with multiple text similarity metrics, and multiple similarity thresholds to assess if label_match is similar to label_query.
The similarity metrics we experiment with are: i) Normalized Levenshtein similarity, ii) Jaro-Winkler similarity, iii) similarity metric based on Longest Common Subsequence (Metric LCS), and iv) Jaccard-index similarity.
For each similarity metric, we measure F1 scores on the similarity threshold range of 0,0.05, 0.1, 0.15, …, 0.75, 0.8
Note that, for Jaccard index, we experiment with values of n-grams ranging from 1 to 5 as this algorithms convert string into  a set of n-grams when computing the similarity. 

Grid search setup for θ_visual and θ_textual to determine the best operating values for .
To identify the best set of components for , we experimentally determine the best hashing method, the corresponding distance threshold for that hash, similarity metric for the OCR label, and accordingly the similarity threshold for comparing label_query and label_match.
We perform a grid search over these four different components of , scoring the combinations of the components by their corresponding F1-score, evaluated on  .
We present the results of the grid-search experiment in Figure <ref>.
For space reasons, we only present the best performing text similarity method for each θ_visual and the two hashing methods.
We analyze how the F1 score of the embeddings with θ_visual change as we increase the θ_textual.

We can see that the combination of PDQHash with θ_visual of 90, and the OCR component using Jaccard similarity (ngram = 4) with θ_textual of 0.05 produces the best F1 score of 0.980.
This setting yields a precision of 0.990 and recall score of 0.979.
The next best performing metrics are PDQHash with θ_visual of 80 and pHash with θ_visual of 10 using normalized levenshtein similarities.
We observe the immediate returns of expanding the θ_visual to the maximum bound to retrieve as many relevant results as possible, without compromising on the false positives.
We will further evaluate this configuration with previous state of the art image detection methods in Section <ref>.
We also note that we can improve the performance of pHash embedding by using a wider θ_visual of 10, compared to the thresholds of 6 and 8 used in the prior works. 

In the rest of the paper, we set to use an image embedding of PDQHash with a θ_visual of 90 and an OCR post-processing component of Jaccard similarity (ngram = 4) with θ_textual of 0.05. 





§ EVALUATION

In this section, we first compare with various other image similarity systems as baselines, showing that is the best performing approach for content moderation.
Next, we use to identify misleading images during the 2020 US Presidential Election in the wild, and compare its effectiveness with the existing moderation system of Twitter.
Finally, we perform a qualitative characterization of the misleading images by analyzing the images through the lens of Twitter's platform policies in detail.



 §.§ vs. baselines


We compare the performance of against various image similarity systems as a baseline for evaluation.
We use four different families of image similarity systems for comparison.
First, we compare against semantic embeddings that use Deep Neural Networks (DNNs) based methods. 
These include features extracted from final layers of DNN architectures specializing in image classification such as Inception-v3 <cit.> and ResNet <cit.>. 
The features extracted from these methods have been shown to be useful for visual search and similarity tasks <cit.>.
The next family of methods that we compare against is image descriptors, which can be considered as “deeper syntactic hashes” <cit.>.
This set of methods is popular in computer vision tasks such as object recognition, scene recognition, and work by extracting a large number of keypoints in an image.
Matatov et al. <cit.> use ORB descriptors for identifying the spread of visual misinformation to assist journalists.
We use ORB <cit.>, SIFT <cit.>, and DAISY <cit.> as image descriptors.
Another family of method we compare with is the list of prior works that have used perceptual hashing algorithms for identifying images spreading misinformation.
These include Phash and PDQHash with multiple distance thresholds <cit.>.

Finally, we evaluate the performance of with multi-modal embeddings.
While there is a rich body of work and systems existing in the vision-language literature, the majority of these works are trained with an objective of explaining the visual concept contained in the image <cit.>.
On the other hand, OCR texts occur as semantic context to the image itself and can be much longer, noisy, and nuanced.
Thus it is not possible to directly adapt models such as Universal Image Text Representation Learning (UNITER) <cit.>, or Visual-BERT <cit.> that have been trained on vision-language tasks for our problem, in the same way adapting Inception-v3 or ResNet is possible.
However, to assess the potential of a multi-modal embedding that leverages text from an OCR engine, or text from the accompanying tweet with DNN based features, we experiment with “concatenated” vision-language embeddings.
First, we concatenate the embeddings obtained from Inception-V3 (a 2048 dimensional vector) and Sentence-BERT <cit.> (a 768 dimensional vector) to create a joint embedding from the two modalities.

We experimented with two different techniques for unifying the embeddings: i) concatenating, and ii) stacking, further reporting the best results.
While much more sophisticated mechanisms of combining embeddings such as attention mechanisms exist, the dataset size of limits us in evaluating methods that combine embeddings or modalities. 
Fully leveraging multimodal embeddings requires an end-to-end deep learning training setup, and accordingly large scale datasets designed for the problem of soft-moderation, which currently does not exist.

In the same way, we also combine textual and visual embeddings using the CLIP model from Open AI <cit.>.
CLIP consists of a text and an image encoder which encodes textual and visual information into a multimodal embedding space by using contrastive learning.
The model is trained on various text/images pairs from Web sources, and has demonstrated impressive zero-shot capabilities for classification purposes and have been used to detect hateful content on social media <cit.>.
Alongside the OCR text, we also experiment with using the tweet text as part of the multi-modal embedding to evaluate if using the tweet text as further context can improve model performance for image retrieval.
To this end, we concatenate the visual information encoded through the image encoder with three different variants of textual information: i) tweet text, ii) tweet text + OCR text, and iii) OCR text.
After extracting the joint embeddings using CLIP, we further normalize the multi-modal embeddings for evaluation.

For the neural network based image embeddings (Inception-v3, ResNet-50, ResNet-101, and ResNet-152) we experiment with multiple resolutions of input (1x, 2x, etc.), reporting the best performing F1 score for each method.
Similarly, for the neural network based image embeddings and the multi-modal embeddings, we experiment with multiple similarity thresholds (0,0.05, 0.1, 0.15, …, 0.75, 0.8) and report the best performing score for each method (except for the methods that use perceptual hashes with pre-determined similarity thresholds).
For the image descriptors, we experiment with a different number of features (30, 60, 90, and 120) and report the best-performing F1 score for each method.
We compare these methods on . 



The results of the comparison of with the baselines are presented in Table <ref>.
Image descriptor methods perform poorly when used for soft moderation. 
This is because these approaches are designed for higher level tasks like object and scene recognition.
Deep neural network approaches work well in identifying the subjects of moderated images (e.g., Donald Trump or Joe Biden) and therefore report a high recall.
At the same time, however, their precision is low, because they flag any image containing the same subjects as similar.
We find that leveraging multimodal embeddings slightly improves the performance over DNN based methods, but still has a very low precision (0.711) for Inception-v3+SentenceBERT.
In the same way, using concatenated embeddings from Open AI's CLIP significantly increases the precision over single-modality DNN methods, but has a very low recall (0.828).
Among the three different variants of CLIP embeddings leveraging different channels of modalities, we find that encoding the most simplest channel, i.e. CLIP (OCR) has the best performance. 
Surprisingly, encoding tweet text alongside the OCR text, i.e., both CLIP (tweet text) and CLIP (tweet text + OCR) does not improve the performance of the CLIP model.
This can be attributed to how state-of-the art multi-modal embeddings like CLIP are designed to match image and caption pairs, and therefore fail to capture the nuances of text and media co-usage in social media.
Perceptual hashing approaches, on the other hand, identify visually similar images, and therefore report a high precision.
At the same time, the need to exclude images that are visually similar but contain text that is contextually different forces these approaches to use low similarity thresholds, which limits their recall.



overcomes the limitations of all three types of approaches.
The use of perceptual hashing allows our approach to have a better precision than neural network based ones.
At the same time, the use of OCR to determine the context of an image allows to operate at a higher threshold than existing systems based on perceptual hashing, addressing the low recall reported by these methods.
While 's precision is slightly lower than the best-performing perceptual hash method (0.990 vs. 0.995), its recall is the highest among all tested approaches.
As a result of this, reports the best F1-score among the tested approaches, balancing false negatives and false positives better than previous work.

We also compare the runtime of different systems in Table <ref>.
For each method, we report the combined time of generating the image embeddings and indexing those embeddings on Milvus, averaged over 5 independent runs on identical system load.
The time for retrieving visually similar images from Milvus is not considered as Milvus optimizes the retrieval time across all embeddings of different sizes (average of 0.27 seconds) for all of the systems.
We can observe that the runtime of is around 8 times slower than DNN based methods, and about 10 times slower than peceptual hashing based methods.
Computing the OCR label of an image is the most time consuming operation of compared with other methods as it takes an average of 0.223 seconds per image.
While this is a large overhead incurred by , we need to keep in mind that its OCR component is only triggered when a similar image to a seed one is identified within the threshold of θ_visual, which only occurs for 0.973% of the images in .
When there is no match, the time overhead of is the same as PDQHash.
Therefore, we argue that this slowdown of 0.2 seconds every 100 images on average is an acceptable tradeoff, allowing to improve its F1-score by 8% over the second best performing algorithm and allowing for more comprehensive soft moderation.




To further check the impact of the OCR component of , we examine how the latency of OCR changes with increasing amount of text in the images.
We characterize the amount of text by using the percentage of image area covered by it.
Figure <ref> shows a scatter plot of the OCR runtime against the percentage of image covered by text in .
The Pearson's correlation coefficient (r) between the percentage of image covered by text and runtime is 0.401, suggesting a moderate positive correlation between runtime and amount of text contained in an image.
This moderate correlation indicates that the fraction of image covered by text plays a role in the OCR runtime.



 §.§ Detecting Visual Misleading Information on Twitter using 



In this section, we evaluate in the wild to further identify images that, while spreading false information, were not moderated by Twitter.
This showcases the utility of our approach while also highlighting the serious need for scalable and automated techniques to identify visually misleading information.
We first build a set of seed images spreading misleading information, using the 2020 US Presidential Election as a case study.
We then apply to find more images that are similar to the seeds that should have been moderated but were not.
Finally, we perform a thorough qualitative analysis of the types of images detected, providing a characterization of image-based misinformation in the context of the 2020 US Presidential Election and how they align with Twitter's platform policies.



  §.§.§ Identifying and Filtering 2020 US Presidential Election misleading images



In this section, we first describe how we select a dataset of images that received soft moderation from Twitter in the context of the 2020 US Presidential Election.
We then present the results reported by when looking for similar images in the wild, and compare these results with the coverage of the moderation applied by Twitter.

Building a seed set of misleading images.
To build a seed set of misleading images contained in tweets that were moderated by Twitter, we use datasets  and  . 
Following the methodology of <cit.>, we check the moderation status of all tweets in during late 2022 and find 1,133 tweets that were moderated by Twitter.
Unfortunately,  is too large (6.9M tweets) for us to check the moderation status of each tweet, given the limitations imposed by the Twitter API. 
Therefore, we only check for the tweets containing an image that occurred more than 5 times in  , based on their PDQhash, resulting in 542,700 candidate tweets.
From these, we identified a further 3,134 tweets with images that were moderated.
Combining the moderated tweets from these two sources, we end up with a final seed set of 4,267 images.

An issue that we face at this stage is that our dataset contains tweets that were moderated by Twitter, but we do not know if the moderation decision was made based on the image included in the tweet or based on the text.
In fact, it is rather common for social media users to include generic images (e.g., memes or GIFs) in their posts
that do not contain false information themselves.
Two examples of these types of tweets are presented in Figure <ref> (in the Appendix).
Neither image conveys false information, but the text is clearly supporting false claims of election fraud. 
If we kept these images in our seed dataset, would flag many unrelated tweets that just happen to include them, greatly reducing its effectiveness.



To avoid this issue, we manually annotate the 4,267 tweets with images that were moderated by Twitter, discarding those not directly related to false claims about the 2020 US Presidential Election.
We first sample 200 images from the 4,267 seed images and have two annotators independently label them as being relevant or not.
The annotators then discuss their results, finding that 186 out of 200 images were relevant with nearly perfect agreement (Cohen's Kappa score of κ = 0.9177 <cit.>).
This indicates that an image's relevance to the 2020 US Election is clearly understood between the two annotators.
We then split the remaining 4,067 seed images between them and wind up with a total of 959 relevant images.

Detecting misinformation images using .
Finally, we use to retrieve more tweets from the index that contain similar images to the seed misleading images.
To this end, we initially retrieve 50,439 images visually similar to the 959 seed misleading images, out of which 40,244 are further filtered to be contextually similar to the query images, and hence candidates for moderation.

To check for the accuracy of these moderation candidates identified by , we perform a false positive and false negative analysis. 
First, we randomly sample 50 query images from the 959 seed misleading images.
Since the 50 query images are already manually verified to contain misleading images related to the 2020 US Presidential Election, we can assume that visually and contextually similar images to the images will be misleading images as well.
We retrieve and annotate all the images from our index visually similar to the 50 query images following the similar methodology used to build in Section <ref>, resulting in a reference set of 10,387 images. 
Upon querying with the same 50 query images, it retrieves 10,172 images both visually and contextually similar to the query images as moderation candidates.
We find that 96 of these detected images are both visually and contextually dissimilar to the corresponding query images, resulting in a false detection rate of 0.99%.
This is due to the well-known limitations of perceptual hashing algorithms <cit.>, for example when dealing with images where a solid color background dominates.
On the other hand, we find that has a false negative rate of 2.06% as it could only succesfully detect 10,172 images out of the possible 10,387 images.
Based on these results, performs a lot better than Twitter's internal soft moderation approach, which moderated only 521 of those 10,387 images (94.98% false negatives).

Finally, we check if the candidate tweets containing the images identified by were also moderated by Twitter.
Following the methodology in <cit.>, we extract the relevant metadata related to soft moderation interventions for the tweet.
Out of the 40,244 tweets checked, we find that only 2,950 tweets received soft moderation interventions by Twitter.
We were unable to check the intervention-related metadata for 2,479 tweets as they were inaccessible during the time we conducted the study.  
This analysis indicates that Twitter's soft moderation approach is inadequate, with a large proportion of tweets (92.66%) containing misleading images left without moderation, and it showcases the utility of automated approaches like .



  §.§.§ Characterizing misleading images about the 2020 Presidential Election on Twitter



In this section, we perform an in-depth analysis of the images identified by .
To aid human annotation, we first group visually similar images into Image Stories.
We then code the extracted Image Stories according to Twitter's platform content policies <cit.>, with the goal of understanding how the detection performed by aligns with the platform's existing moderation guidelines.
Finally, we investigate whether misleading images violating different types of content policies get moderated differently by Twitter.

Aggregating misleading images into Image Stories.
To help manually code the images detected by , we aggregate them into Image Stories.
We define an “Image story” as a set of similar images that convey the same misleading message in the same context.
For example, the three images included in three different tweets in Figure <ref> are visually similar, and should be grouped together.
To this end, we follow a grouping approach similar to the one adopted by Zannettou et al. when studying image memes <cit.>.

First, we apply clustering to the PDQhash embeddings of the images detected by to group visually similar images by using the DBSCAN clustering algorithm <cit.>.
To use the DBSCAN clustering algorithm, we need to select parameters for two parts of the process: i) distance threshold for assessing similarity, and ii) minimum number of elements in a cluster.
Through the experimental validation performed in Section <ref>, we have already established that using PDQhash as the syntactic embedding, with a θ_visual of 90 is best suited for identifying visually similar images.
We use this empirically informed distance threshold as the Hamming distance difference threshold for the first clustering parameter and set the minimum number of elements in a cluster to be 1, to allow for misleading images that occur in isolation, without visual variants.  
Upon using the DBSCAN algorithm to cluster the 40,244 images detected by , we obtain 258 clusters of images.
These image clusters represent the misleading images aggregated into visually similar claims conveying the same misleading message.

Analyzing misleading images through the lens of Twitter's Platform Policies.
We present our codebook, which guides our annotation process for characterizing Image Stories used during the 2020 Presidential Election.
We develop the codebook such that it aligns with the platform policies of Twitter <cit.>.
We use two different policies outlined in Twitter's Platform Integrity and Authenticity resources: i) Synthetic and manipulated media policy <cit.> and ii) Civic Integrity Policy <cit.>.
Each policy contains multiple categories of violations, and each category is broken down into multiple rules.
The Civic Integrity Policy outlines how “Twitter should not be used for the purpose of manipulating or interfering in elections or other civic processes such as refereed, censuses.”
We include 3 different categories from the Civic Integrity Policy out of the four available, which we discuss in further detail.
The fourth category, “False or misleading affiliation” is related to the use of fake and parody accounts, and thus not related to misleading images.
Similarly, we include the “Synthetic and Manipulated Media Policy” as this policy covers usage of media on Twitter, which is the major modality of content in our study.
The details of the individual rules and categories are listed in Appendix <ref>.

To begin the process, we first randomly sample 100 Image Stories out of the 258 extracted to guide us in developing the codebook.
First, we take each of the 16 different rules of Twitter's Civic Integrity Policy <cit.>, divided into four major categories as the draft of our initial codebook.
Two annotators independently code the sampled images, considering additional contextual information learned from fact-checking articles associated with them.
We discuss these codes, repeating the process three times until the final codebook reached a point where further iterations would not improve it.
Note that we use the 16 different rules to help us identify the closest violation of Twitter's platform policy by the images while annotating the Image stories with the granularity corresponding to the four categories associated with the rules.
We reach a nearly perfect agreement (Cohen's Kappa score of κ = 0.9 <cit.>).

Positioning the extracted Image Stories alongside the platform policies of Twitter helps us re-evaluate the content moderation decision based on Twitter implementations and policies.
We present an example image violating each of the four categories discussed in Figure <ref> (in the Appendix).
In the rest of this section, we describe the four categories in our codebook in detail, along with their definitions and our evaluation for matching images with the corresponding category.




1. Misleading information about how to participate in civic processes:
This category refers to images that mislead people about “election participation procedures and requirements, cause confusion about officials, or discuss threats to voting locations.”
Note that we did not find any images related to threats at voting locations in the extracted image stories.
For election participation procedures and requirements, misleading images aim to spread false claims about the irregularity of the criteria by which votes are being cast.
For example, dead people are casting votes; votes are cast multiple times; votes are being cast after the closing of polls.
Misleading images may also claim that mail-in ballots are insecure, illegal, and a source of voter fraud.
Figure <ref> aims to intimidate voters from voting-by-mail, casting doubts over the security of mail-in ballots despite mail-in ballots proven to be safe to use <cit.>. 

2. Misleading information intended to intimidate people from civic processes:
This category contains images that mislead people about “how votes are being counted (or not counted), problems with ballot equipment, disruption at voting locations, and the closing of polls”.
Misleading images in this category aim to make claims about legitimately cast ballots getting invalidated, malfunction of voting machines, switching of votes between candidates, and deleting of votes on the machines.
We find images spreading misleading claims about disruptions at voting locations like poll workers filling ballots, poll workers forcing voters to use Sharpies, and ballot observers not being allowed to observe the counting process.
The example in Figure <ref> is spreading false claims about ballots using Sharpies in Arizona being invalidated and poll workers forcing people to use Sharpies <cit.>.

3. Misleading information about the outcomes of civic processes:
This category refers to the images that mislead people about “election rigging, ballot tampering, vote tallying, declaration of premature victory, casting doubt on the outcome of civic processes, calling for interference with the implementation of election results, and undermining public confidence in the methods and results of the election.”
Misleading images in this category aim to make claims about the illegal processing and handling of ballots, which include alteration, manipulation, destruction, forging, counterfeiting, stealing, and pre-filling ballots.
We also find false claims about voter registration and turnout, and visual and statistical anomalies in patterns of vote counting.
By casting doubt on the outcome of civic processes, misleading information aims to question the integrity of the election process, discredit different stages of election processes, and call for interference with the implementation of election results using slogans such as “Stop the Steal,” “Fraud.”
The misleading image in Figure <ref> casts doubt over bipartisan vote tallying process, suggesting that Trump ballots were discarded in garbage bags <cit.>.

4. Synthetic and Manipulated Media:
This category refers to images that are “significantly and deceptively altered, manipulated, or fabricated, and images shared with deliberate intent to deceive people”.
For this category, included images are significantly and deceptively altered, manipulated, or fabricated.
On the other hand, images shared with malicious intent include out-of-context tweets sharing the media.
For example, adding quotations shared in the past that are taken out of context to spread election misinformation or images from the past that are re-used in the present context, like discarded ballots from the 2018 election being repurposed to say ballots are being tampered with in the 2020 election.
Figure <ref> is a doctored image in connection to the “Bush-Gore Florida recount” during the presidential race in 2000 <cit.>, shared by a Trump campaign spokesman.


We assign each Image Story to one category.
There are some observations from our annotation process:
We notice that the use of memes is common in spreading misleading information.
They are designed for people to add text to them to make a variation.
During the annotation process, we first assess the overlay text on the memes to see if it falls into any of the categories other than the fourth: Synthetic and Manipulated Media. 
If it does not, we classify it as an image altered to be shared with malicious intent.

We also find that a large portion of images are used out of context and could potentially be categorized into the fourth category: Synthetic and Manipulated Media.
However, out-of-context information is widely used to push narratives related to the other three themes.
Thus, we take precedence of annotating the images into the first three categories if it suits any.
Otherwise, it is annotated as the fourth category.
We annotate these images according to their primary content to expand and deepen the qualitative analysis of these moderated images.
For example, a screenshot of a video clip in which ballot workers are transcribing ballots is used out of context and spread with the misleading content of “ballot stuffing.”
In this example, the medium of spreading the misleading image is taking it out of context. However, the primary content being spread is related to vote counting and ballot tampering, which falls under the Twitter Civic Integrity Policy.
Thus, we annotate it as Category Three: Misleading information about the outcomes of civic processes.

Moderation of misleading images by violation category.
After associating the misleading images with the corresponding platform policy violations, we want to understand whether Twitter is more likely to moderate images violating some of them.
 Table <ref> reports the breakdown of the rate of moderation by Twitter of images belonging to the different categories of violations, among the tweets identified by .
A high-level overview of the breakdown by category suggests that Twitter moderated specific types of misleading image stories more aggressively, while the moderation rate on other categories appears to be laxer.
Misleading image stories of the categories “Intimidation from Civic Processes” and “Participation in Civic Processes” are moderated the most, with 5.96%, and 4.16% of the images getting moderated.
Surprisingly, the most popular category of content violation among image Stories, “Misleading information about outcomes of civic processes” has the lowest moderation rate of 1.77%.
Finally, images violating the category “Synthetic and Manipulated Media” have the moderation rate of 2.76%.
While we can make overall observations that different types of platform policy violations might have been disproportionately moderated from Twitter's end, we find concerning results of overall moderation rate across the categories being very low.









§ RELATED WORK


Soft Moderation Approaches by Social Media.
Social media platforms namely Facebook, and Twitter have increasingly shifted their approach towards warning labels as a tool for content moderation, and additionally, provide surrounding  context to the users when interacting with potentially false information.
These platforms have applied warning labels on false information shared ranging from Covid-19 pandemic <cit.>, 2020 US Presidential Elections <cit.> to climate denial misinformation <cit.>.
Twitter has reported that approximately 74% of the tweet viewership happened after the warning labels were applied to the tweets and, warning labels were effective in decreasing users quoting the misleading tweets by an estimated 29%. 
Savvas Zannettou <cit.> performed empirical analysis on a sample of 2,244 tweets with warning labels related to the 2020 US Presidential Elections.
Paudel et al. proposed Lambretta, a system that aims to improve soft moderation of textual content on Twitter by leveraging Learning to Rank <cit.>.
To the best of our knowledge, is the first end-to-end approach to perform soft-moderation of images social media that has been proposed by the research community.

Applications of Perceptual Hashing in Computer Security.
Perceptual hashing techniques have been widely used in computer security, leveraging them to detect phishing websites <cit.>, scam websites <cit.>,fraudulent services <cit.>, and to identify impersonators on social media <cit.>.
Perceptual hashes have also been used for content authentication <cit.>, and tamper detection on images <cit.>. 
Alkhowaiter et al. <cit.> evaluated six types of perceptual hashes on their ability to detect image manipulation and transformation over two major social media platforms: Facebook and Twitter.
Historically, Microsoft developed PhotoDNA <cit.>, a type of syntactic embedding to identify and report the distribution of child exploitation material.
Facebook currently uses PDQHash in the “ThreatExchange” platform[https://developers.facebook.com/docs/threat-exchange/] to share “signals” of harmful media on their platform.


Image-based misinformation in social networks.
Several studies are focusing on developing automated detection methods to identify images containing misinformation, either manipulated images <cit.>, or images that are taken out of context or misinterpreted on social media <cit.>.
Most of the works on multi-modal misinformation tackle classifying or detecting a single image as misinformation, while very few works have focused on studying the spread, and diffusion of misleading images within and across social media.
The closest work to is a system called DejaVu <cit.>, which is designed to assist journalists in collaboratively addressing the spread of visual misinformation by using ORB descriptors <cit.> (another type of syntactic embedding) to encode the images, and FAISS <cit.> to index the images.
Additionally, other works study the spread of COVID-19 media through WhatsApp in Pakistan <cit.>, and other types of visual misinformation in WhatsApp <cit.>.
On the other hand, works by  <cit.> study the usage of images in state-sponsored influence campaigns.
Wang et al. <cit.> analyzed the spread of Fauxography images on social media, which are images that are presented in an incorrect or untruthful fashion, by using ground truth fact-checked images from fact-checked organization Snopes [https://www.snopes.com/fact-check/].
Similarly, Zannettou et al. used visual similarity (i.e., perceptual hashing) and images annotated by the website KnowYourMeme to study the evolution and diffusion of image memes posted on social media <cit.>.



§ DISCUSSION AND CONCLUSION



In this paper, we presented , a scalable system able to identify images that are candidates for soft moderation on Twitter.
overcomes the inability of perceptual hashing to discern the context of an image by incorporating OCR into the matching process.
Our results show that outperforms three types of image matching systems based on perceptual hashing, on image descriptors, and on deep neural networks.
With the highest F1-score among competitors, places itself as the state of the art in automated image-based soft moderation.

We believe that (which we make publicly available [https://github.com/idramalab/pixelmod]) will be an inspiring foundation for researchers and online platforms aiming to improve content moderation on social media.
In the rest of this section, we first discuss the ethical considerations of our work and design implications that online platforms should keep in mind when deploying .
We then discuss the limitations of our approach and some directions for future work.


Ethical Considerations.
Our work only uses publicly available Twitter data that was collected the official API while it was still open to academic researchers, and we do not interact with users.
As such, this work is not considered human subject research by our institution.
We also preserve the privacy of Twitter users as we do not analyze any personally identifiable information (e.g., location data, account names) and blur the regions of example tweets used in the paper containing identifiable (meta)data.
Additionally, we take steps to blur the example images used in the paper unless i) they are public figures, ii) they are an illustration, iii) they are stock images.
As any content moderation system, could be used for malicious purposes like censorship and surveillance.
We advocate that the system should be used with ethical principles in mind, following the respect for public interest and beneficence principles from the Menlo report <cit.>.

Design implications.
Our experiments found that Twitter soft moderation misses most images that should be labeled.
This means that leveraging would help Twitter's moderators cover more false information on their platform.
However, when deploying , Twitter or other online platforms should take several aspects into consideration.
First, while 's detection performance exceeds that of existing approaches, a false detection rate of 0.99% may still be considered too high by online platforms to consider its adoption as a fully automated soft-moderation system.
We envision PixelMod to be used by platform moderators as a tool to identify a set of candidates for soft moderation with limited false positives, which then receive manual vetting.
The ultimate decision on whether to apply the moderation labels, however, should remain with the human operator.
This is not dissimilar from what online platforms are doing already, but our approach would allow them to obtain a more comprehensive view of misleading content on their network.
Additionally, could help addressing the main pain points of relying on human moderation, which are the latency in decision making and having limited moderator resources <cit.>.

Second, is an inherently reactive system: it requires a set of images already identified by the platform as misleading.
This process could be streamlined by using example images that have been fact-checked by dedicated organizations <cit.>.
When curating the set of seed images in Section <ref>, we found that querying images directly from a list of moderated tweets can be tricky, since all of the moderated images might not be of misleading nature.
In such cases, moderators using should take an additional step to ensure the query images are misleading in nature, rather than ambiguous images not related to the events being studied, to get the most relevant results as candidates for moderation.

Misleading images can spread across multiple social media platforms, propagating with different contexts and forms <cit.>.
Platforms can use as a tool alongside an industry shared database of known misleading image hashes for tracking the spread of misleading images on their service and across other online communities.
Platforms already use shared databases for tracking Child Sexual Abuse Material (through the National Center for Misleading and Exploited Children) and terrorism content (through the Global Internet Forum to Counter Terrorism) <cit.>.
A tool like backed by a centralized database could easily be added to major social media platform's existing efforts to combat fraud and misinformation <cit.>.

Runtime implications.
As discussed in Section <ref>, the increased runtime overhead of is due to the OCR component, which only gets triggered for 0.973% of the images in .
This rate is 1.450% among the 19.7M images used in our ‘in-the-wild’ evaluation in Section <ref>.
It is to note that factors like image resolution, background and font complexity could also impact the runtime of the underlying OCR engine.
The overhead reported is an upper bound on the runtime of and the average runtime of our system is in the same order as other baselines.
This occasional slowdown is a tradeoff we make for increased recall for , which allows us to achieve a much higher coverage of soft-moderation candidates than the ones achieved by Twitter.
However, can be adapted according to the content moderation budget of the platform.
The initial set of results retrieved by (visual matches to query images) can be sorted or filtered through metadata such as popularity of the account posting the images, or other metadata aligning with specific content moderation strategies of a platform, before passing through the contextual similarity component.


Applying to other platforms and topics.
We could not test on other topics and platforms due to the lack of reliable datasets across platforms and topics.
First, we are not able to test our system on other social media platforms due to the lack of access to a uniform sample of posts (like the 1% sample that forms our dataset)
While datasets containing misleading images exist for other platforms like WhatsApp and Telegram, these are not suited for our evaluation since they only contain a single instance of labeled misleading images, making the visual similarity research process that is at the center of moot.
Even though Twitter applied warning labels on COVID-19 misinformation, these were unreliable and inconsistent <cit.>, which we independently confirmed in our preliminary analysis.
Despite the limitations on evaluation settings, we expect to generalize well across multiple platforms and topics.
The only requirement for a platform to apply to a new campaign is a set of seed images that are known to be misleading, and the system should generalize well to other platforms and topics, as the underlying image embeddings are syntactic in nature, and do not incorporate any domain-specific metadata (e.g. number of retweets, number of likes available on tweets).


Limitations.
Despite the promising performance of in identifying visually similar images at scale, the embedding used by (PDQHash) is vulnerable to adversarial manipulations <cit.>.
An adversary could modify images to have a PDQhash that is very far from the one of the corresponding seed image, generating a false detection by .
This is a serious risk, and future research should investigate defenses against these attacks.
Some potential avenues of defense are utilizing an ensemble of hashes: combining results from both pHash and PDQHash <cit.>, and adversarial training of embeddings <cit.>. 
Using an ensemble of hashes would force an attacker to jointly optimize the adversarial attacks against an ensemble of hashing methods opposed to a single method, thus increasing the operation cost on the end of adversaries.
Future works can also look at incorporating the contextual information (OCR text) contained in the images as part of generating the syntactic embeddings themselves, making it difficult for adversaries to modify the images without deviating from the contextual messaging of the images.
At the same time, the threat model here assumes centralized coordination by an adversary.
While this is within reach when dealing with state-sponsored disinformation actors <cit.>, it is not applicable to content spread autonomously and in good faith by regular users, for example in the wake of the uncertainty surrounding the COVID-19 pandemic.
In the case of crisis scenarios, misleading content with significant risk factors might proliferate as many different variants of the original content as a consequence of the broad and diverse vector of sharing by people.
This might render the initial set of seed hashes and similarity threshold being used ineffective in tracing the spread of the images.
One such example of this was the spreading of videos of the Christchurch mosque shooting, where Facebook reported that their systems were defeated by variants of the original videos as of bad actors started sharing it <cit.>.
Platforms could use methods from online learning to update the seed set of hashes to flag, and dynamically adjust the distance threshold in specific crisis cases to better handle such events.


Apart from applying soft moderation warning labels, Twitter outlines “reduced visibility” as a possible consequence for tweets violating their Civic Integrity Policy <cit.>.
This means it is possible Twitter could have identified all of the images subsequently identified by , and chosen to reduce visibility of tweets including them instead of applying warning labels.
However, we cannot analyze this phenomenon due to lack of any public metrics regarding visibility of tweets. 
Due to the nature of the evaluation datasets, images containing multi-lingual text could not be evaluated.
Therefore, future applications of the system in multi-lingual settings might require further configuration of the underlying OCR engine (e.g. specifying “languageHints” parameters in Cloud Vision OCR engine with the intended language of evaluation).

Future work.
In the future, we plan to apply to other online platforms.
The approach is platform-independent, and this could give interesting insights on how misinformation spreads across different online communities, and which communities are particularly influential in generating viral image misinformation.
For example, we believe quite strongly that because of its low computational overhead and high performance, it makes a good fit for deployment by decentralized social networks (e.g., Mastodon), an environment where recent work has shown the potential to actually help improve 's performance via federated model sharing <cit.>.
Since our approach does not require any platform-specific information, it is particularly interesting as a basis for further research, especially at a time where academic research is being seriously threatened by the discontinuation of the Twitter Academic API as we know it.
Regardless of the specifics of future work, our presentation of provides a roadmap for measuring, tuning, and benchmarking soft moderation systems; critical for any moderation tool's success.
We strongly believe that the computer security community has a lot to say in this space, and hope that more researchers will get into this space.

Acknowledgments. We would like to thank the anonymous reviewers and the anonymous shepherd for their feedback, and Primah Muwanga for her help setting up the artifact after the paper was accepted.
This work was supported by the NSF under grants CNS-1942610, CNS-2114407, CNS-2114411, CNS-2247867, CNS-2247868, and IIS-2046590, and by a grant from the Media Ecosystems Analysis Group (MEAG).



abbrv
























































































§ TWITTER'S CONTENT POLICIES AND VIOLATIONS.


We list the four different categories of Twitter's platform policy violations and the corresponding rules for each category below.


    
    
  * Misleading information about how to participate.

        

            
  * Images misleading people about participation procedures and requirements.

            
  * Images sowing confusion about officials and institutions.

            
  * Images discussing threats on voting locations.

        

    
  * Misleading information intended to intimidate people from civic processes.

     

            
  * Images about votes not being counted.

            
  * Images about Equipment Problems.

            
  * Images about disruptions at voting locations.

            
  * Images about closing of polls.

        

        
  * Misleading information about outcomes of civic processes.

    

            
  * Images undermining public confidence in methods and results of election.

            
  * Images with misleading claims about election rigging.

            
  * Images with misleading claims about ballot tampering.

            
  * Images with misleading claims about vote tallying.

            
  * Images with declaration of premature victory.

            
  * Images casting doubt on outcome of civic processes.

            
  * Images calling for interference with the implementation of election results.

        

    
  * Synthetic and Manipulated Media.


    

            
  * Images that are significantly and deceptively altered, manipulated or fabricated.

            
  * Images shared with malicious intent , including out of context tweets sharing the media.

        


























































































































































































