Disclosure of AI-Generated News Increases Engagement but Does Not Reduce Aversion, Despite Positive Quality Ratings
Fabrizio Gilardi Sabrina Di Lorenzo Juri Ezzaini Beryl Santa Benjamin Streiff Eric Zurfluh Emma Hoes

HGAMN: Heterogeneous Graph Attention Matching Network for Multilingual POI Retrieval at Baidu Maps
    
	Jizhou Huang, 
	Haifeng Wang,
	Yibo Sun, 
	Miao Fan, 
	Zhengjie Huang, 
	Chunyuan Yuan,
    Yawen Li

    
===========================================================================================================



§ ABSTRACT  
The advancement of artificial intelligence (AI) has led to its application in many areas, including journalism. One key issue is the public's perception of AI-generated content. This preregistered study investigates (i) the perceived quality of AI-assisted and AI-generated versus human-generated news articles, (ii) whether disclosure of AI's involvement in generating these news articles influences engagement with them, and (iii) whether such awareness affects the willingness to read AI-generated articles in the future. We employed a between-subjects survey experiment with 599 participants from the German-speaking part of Switzerland, who evaluated the credibility, readability, and expertise of news articles. These articles were either written by journalists (control group), rewritten by AI (AI-assisted group), or entirely generated by AI (AI-generated group). Our results indicate that all news articles, regardless of whether they were written by journalists or AI, were perceived to be of equal quality. When participants in the treatment groups were subsequently made aware of AI's involvement in generating the articles, they expressed a higher willingness to engage with (i.e., continue reading) the articles than participants in the control group. However, they were not more willing to read AI-generated news in the future. These results suggest that aversion to AI usage in news media is not primarily rooted in a perceived lack of quality, and that by disclosing using AI, journalists could attract more immediate engagement with their content, at least in the short term.








§ INTRODUCTION


The landscape of journalism is evolving significantly with the advancement of artificial intelligence (AI). Traditionally, journalists have used computers primarily for research and data analysis, but this role is rapidly expanding. Modern AI, particularly generative AI, now matches or even surpasses human capabilities in tasks like text composition. This has led to the emergence of “automated” or “robot” journalism <cit.>, a key aspect of the broader computational journalism trend that underscores the increasing role of computation and data in news production <cit.>.

AI applications, especially in natural language generation, have become integral in practical journalism. These applications include news writing for media organizations such as the Associated Press and Forbes, summarizing scientific data via platforms like the Open Research Knowledge Graph, and narrative writing using tools like ChatGPT-4. These advancements enable AI to produce content that is virtually indistinguishable from human writing, demonstrating AI's potential in various writing-intensive domains.

The integration of AI-generated content in journalism presents several implications. On one hand, it offers opportunities for faster, multilingual, and expanded content production with potentially fewer errors, which could enhance news quality and help combat misinformation <cit.>. AI can also handle routine reporting tasks, allowing journalists to focus on investigative stories. On the other hand, however, there are concerns about job losses in the journalism sector and doubts regarding algorithms' ability to fulfill the “watchdog” role traditionally associated with journalists <cit.>.

A crucial issue is the public's perception of AI-assisted journalism. Understanding attitudes towards AI-generated journalism is essential as it may directly impact the public's political knowledge. Indeed, if the public distrusts AI-generated content and chooses not to engage with it, they may miss out on important political information, leading to a less informed citizenry. A recent study revealed that only 29% of Swiss respondents would read fully AI-generated news, 55% would read news if AI assisted in its creation, and 84% would prefer news created without AI involvement <cit.>. This highlights skepticism towards AI-generated news amongst the Swiss population, consistent with surveys from other countries <cit.>. Such aversion could be problematic because it could undermine the credibility and effectiveness of news organizations. Therefore, transparent and responsible use of AI in journalism is essential to build trust and ensure that the public remains informed and engaged with credible news sources.

In this preregistered study, we investigate the perceived quality of AI-assisted and AI-generated versus human-generated news articles prior to disclosing whether the articles were created by humans, AI-assisted processes, or entirely by AI, and whether subsequent disclosure of the source influences self-reported willingness to engage with those articles as well as the willingness to read AI-assisted or AI-generated articles in the future. Our results indicate that all kinds of news articles, regardless of whether they were written by journalists or (assisted by) AI, were perceived to be of similar quality in terms of credibility, readability, and expertise. When participants in the treatment groups were subsequently informed about AI's role in generating the articles, they expressed a higher willingness to read the full news article than participants in the control group, who saw human-written articles. This increased engagement, however, did not translate to an increased openness towards reading AI-generated news in the future. 

These findings imply that aversion towards AI usage in the news media is not primarily linked to concerns that AI-generated content is of lower quality. Moreover, the results suggest that transparency in the use of AI by journalists could enhance immediate reader engagement. Future work should delve deeper into understanding the underlying mechanisms driving both results, exploring factors such as the novelty of AI involvement, trust in AI, and the psychological effects of transparency. Additionally, research should investigate long-term impacts on reader behavior and trust to develop strategies for effectively integrating AI in journalism, ensuring that it complements rather than undermines the journalistic process. 



§ PREVIOUS RESEARCH



Several strands of literature have explored the perception of human versus AI-generated news. The first strand involves research on public opinions about human versus computer-generated news, relying on respondents' prior experiences or perceptions <cit.>. A recent survey conducted across six countries (Argentina, Denmark, France, Japan, the UK, and the USA) revealed that public opinion on the use of generative AI in journalism is mixed <cit.>. While people expect AI-produced news to be cheaper and more up-to-date, they also anticipate it to be less trustworthy and transparent. Most respondents are wary of AI-generated news, especially on hard topics like politics and international affairs, but show more acceptance for soft news like fashion and sports. A large majority favors disclosure when AI is used, but opinions differ on which specific uses should be labeled. Uncertainty remains, with many respondents unsure or neutral about AI's role in journalism. A limitation of these kinds of opinion-based studies is that they could reflect respondents' preconceptions or lack of familiarity with AI-generated texts, which may not align with actual behaviors when confronted with AI news.

The second strand consists of surveys where respondents directly evaluate news stories labeled as either “human-written” or “computer-written.” For instance, <cit.> found that respondents rated human-written news higher in readability but lower in credibility compared to AI-generated news. However, when participants were only given a single article without information on its origin, the differences in their evaluations were minimal, indicating that the perception of the author (human or AI) plays as significant a role in the assessment as the content itself.

The third strand focuses on experiments where participants are presented with identical articles but with different stated authors, to measure the effect of the source. For example, <cit.> found that while news consumers rated the credibility of both human and computer-generated articles equally, journalists perceived human-labeled articles as significantly more trustworthy. This effect varied with the topic, as sports articles were rated as less trustworthy than financial ones. <cit.> used a 2 × 2 design to explore perceptions of article quality based on authorship labels, finding that both journalists and the public rated AI-written articles higher when they were correctly labeled as such. Another experiment by <cit.> varied the attributed author of a human-written text, finding no significant difference in perceived credibility across different labels. Additionally, <cit.> explored how labeling news headlines as “AI-generated” affects accuracy ratings and sharing intentions, finding that such labels decreased perceived accuracy and sharing intent, even when the text was human-written.

The fourth strand investigates the effect of the message itself by providing texts written by different authors without stating the source, thereby reducing source-based biases. <cit.> found that computer-written articles were perceived as more credible, whereas human-written articles scored higher in readability. This approach highlights the challenge of distinguishing between human and AI-generated content. <cit.> extended this analysis across multiple mediums (text, audio, video), finding significant differences favoring human-generated content only in audio and video formats. In a follow-up study, <cit.> discovered that human newscasters were perceived as more credible than AI newscasters.

Some studies merge the third and fourth strands by measuring both source and message effects. For instance, <cit.> examined the effects of declared source, actual article content, and topic, finding that while declared source effects generally favored human-written articles, computer-written articles scored higher in credibility and expertise in fact-based topics. <cit.> explored European readers' perceptions of automated journalism, finding that automated and human content were perceived as equally credible.

Finally, a meta-analysis incorporating results from studies across all four strands found no significant difference in credibility between human-written and AI-generated news, with human-written texts only having a slight advantage in perceived quality and a notable lead in readability <cit.>. However, all studies in this analysis were conducted before 2020, and the rapid advancements in generative AI may have altered these dynamics.

Overall, existing research suggests that while AI-generated news can be perceived as being of equal quality to human-generated news, knowing that an article was created by AI can negatively influence readers’ perceptions. This implies that preconceived notions may lead to distorted preferences for human-generated news. However, previous studies have not separated quality evaluations from the willingness to read AI-generated news, limiting our understanding of public attitudes in this area. Moreover, recent innovations in AI text generation may have made some of the existing findings outdated, underscoring the need for additional research.



§ RESEARCH QUESTIONS AND HYPOTHESES


Our study addresses the following research questions:


    
  RQ1: How do respondents rate the quality (expertise, readability, and credibility) of AI-assisted and AI-generated news articles, compared to human-generated news articles?
    
  RQ2: What impact does disclosing AI’s role in generating an article have on respondents’ willingness to read AI-assisted or AI-generated news after they have rated the article for quality?


By answering these two research questions, we extend the literature in important ways. First, we combine a detailed measurement of “quality” with three realistic levels of AI-involvement in news generation, reflecting current capabilities and practices. Second, we make sure that respondents' views on AI are not asked in the abstract, which may reflect inaccurate preconceptions, but are rooted in specific examples we provide. Third, we separate quality evaluations from willingness to read AI-generated news by disclosing the degree of AI involvement after participants evaluated the articles' quality, which ensures that awareness of AI's role does not affect quality ratings.

The first set of hypotheses focus on quality assessments of news articles with different degrees of AI involvement. Given that a meta-analysis of experimental studies found that human-written news was generally rated higher for quality <cit.>, our pre-registered hypotheses state that news articles generated with AI involvement are rated lower for all dimensions of quality. However, we acknowledge that recent advances in generative AI may challenge these expectations.




  H1: Respondents in the AI-assisted group rate the article's expertise lower than respondents in the control group.


  H2: Respondents in the AI-assisted group rate the article's readability lower than respondents in the control group.


  H3: Respondents in the AI-assisted group rate the article's credibility lower than respondents in the control group.


  H4: Respondents in the AI-generated group rate the article's expertise lower than respondents in the control group.


  H5: Respondents in the AI-generated group rate the article's readability lower than respondents in the control group.


  H6:  Respondents in the AI-generated group rate the article's credibility lower than respondents in the control group.



Our second set of hypotheses is designed to explore people’s attitudes once they learn the true role of AI in the article they have read, after rating them for quality.

<cit.> found that “news consumers get more pleasure out of reading human-written as opposed to computer-written content”. Hence, we expect that respondents in the AI-assisted and AI-generated groups will show a lower willingness to continue reading their articles after being informed of the true generation process.


    
  H7: Respondents in the AI-assisted group express a lower willingness to continue reading the article than respondents in the control group.
    
  H8:  Respondents in the AI-generated group express a higher willingness to continue reading the article than respondents in the control group.


Our final hypotheses address attitudes towards AI usage in news articles in general, that is, not regarding the particular articles they have read and rated. Respondents will be asked to what degree they would be willing to read AI-generated news in the future. This approach is more in line with opinion-based studies asking abstract questions. However, in our study we ask this question after respondents have engaged with examples of the kinds of articles we are asking about. There is evidence that greater technological competence, AI familiarity, and AI knowledge increases trust in AI <cit.>, which suggests that engagement with AI articles may decrease skepticism towards them. Therefore, we expect that respondents in the AI-assisted and AI-generated display express willingness to read AI-generated news articles in the future.



    
  H9:  Respondents in the AI-assisted group express a higher willingness to read AI-generated news in the future than respondents in the control group.

    
  H10: Respondents in the AI-generated group express a higher willingness to read AI-generated news in the future than respondents in the control group.
    






§ EXPERIMENTAL DESIGN



To explore the public's perceptions of AI-generated and AI-assisted news articles compared to traditional human-generated articles, we conducted an online between-subjects survey experiment. We recruited 599 participants from the age of 18 from the German-speaking part of Switzerland through the survey company Bilendi, using quotas based on age and gender to ensure balance across these dimensions. Sample size was determined by a power analysis, assuming an effect size of Cohen's d = 0.3 for the primary outcomes and d = 0.15 for the secondary outcomes.

First, participants completed a pre-survey to determine their socio-demographic characteristics, including age, gender, education level, and political orientation. Participants were then randomly assigned to one of three conditions: a control group that read excerpts from human-generated articles (“human-generated” group), a first treatment group that read excerpts that were rewritten with the help of ChatGPT (“AI-assisted” group), and another treatment group that read excerpts that were entirely generated by ChatGPT (“AI-generated” group).

To ensure consistency and create a set of texts that is comparable across the three groups, all articles are derived from the human-generated articles that are shown to the control group. We used the following procedure to generate the articles. For AI-assisted articles, we copy-pasted the original article into ChatGPT and asked it to rewrite the article without losing any information. This resulted in articles for the AI-assisted group that are similar to the originals, but are rewritten by the AI system. For AI-generated articles, we only provided ChatGPT with the title and lead of the original article. We then asked ChatGPT to generate a short article in the same style as the source of the human-generated articles.

Each participant read two excerpts, randomly drawn from a pool of ten articles on Swiss politics, with each excerpt having a hard cutoff at 150 words to maintain consistency, avoid bias due to article length, and simulate a paywall for the purposes of our third outcome (willingness to “continue reading” the article). The random selection of two articles from a pool of ten texts for each group minimizes the risk that the outcomes depend on specific topics. Moreover, for each article, we asked respondents to answer a simple question on the article's content. This allows us to check that respondents have read the text in sufficient detail.

The first outcome measures how participants evaluate the quality of the articles. Following <cit.>, we ask respondents to rate three dimensions of quality: expertise, readability, and credibility. Each dimension is based on specific items, expressed as adjectives <cit.>: “clear”, ”coherent”, “comprehensive”, “concise”, and “well-written” for expertise; “boring”, “enjoyable”, “interesting”, “lively”, and “pleasing' for readability; and “biased”, “fair”, and “objective” for credibility. Participants rated the articles on each of these items on a 1-5 Likert-scale, and we then aggregate the scores for each dimension. Following our pre-registered procedure, we then computed a single measure of quality because Cronbach's alpha was above 0.7. However, results are robust to using separate scores for each dimension.


Following the initial rating, participants were then informed about the true creation process of the articles. After learning how the texts they read were actually created, respondents were asked if they would be willing to continue reading the full articles after reviewing the excerpts again. In addition, all respondents were asked to report if they would be willing to read AI-generated news articles in the future, on a 1-5 Likert-scale.

Following our pre-registration, we use linear regression models to analyze the data, controlling for socio-demographic variables and using the Benjamini-Hochberg procedure to adjust for multiple comparisons.



§ RESULTS


Figure 1 presents the mean ratings for the first outcome, which assesses the perceived quality (credibility, expertise, and readability) of the articles across the three groups: control, AI-assisted, and AI-generated. The mean credibility ratings were 3.39 for the control group, 3.37 for the AI-assisted group, and 3.38 for the AI-generated group. In addition to the minimal variation, the overlapping confidence intervals suggest no significant differences between the groups. For expertise, the mean ratings were 3.56 for the control group, 3.63 for the AI-assisted group, and 3.62 for the AI-generated group, with overlapping confidence intervals again indicating no significant differences. Readability ratings were 3.15 for the control group, 3.14 for the AI-assisted group, and 3.19 for the AI-generated group, with no significant differences observed, as evidenced by the overlapping confidence intervals. 










 







Table 1 provides a more detailed analysis of the first outcome. The table presents regression coefficients for the AI-assisted and AI-generated groups, with controls for age, gender, education, and political orientation. Each column corresponds to a separate regression model aligned with one of the hypotheses. The coefficients for the AI-assisted and AI-generated groups across expertise, readability, and credibility are not statistically significant, reinforcing the findings from Figure 1 that there are no substantial differences in the perceived quality of the articles between the groups. Among the control variables, only age and political orientation yielded statistically significant coefficients in some models; however, the effect sizes for these variables are very small. These results remain unchanged when estimating the models only for the subset of respondents who passed the manipulation check. 

These results indicate that there are no significant differences in perceived credibility, expertise, and readability between AI-assisted, AI-generated, and human-generated articles. Consequently, we reject the first set of hypotheses (H1-H6), which expected lower perceptions of these attributes in the AI-assisted and AI-generated groups compared to the control group. In other words, news articles generated either with the assistance of AI or entirely by AI are perceived to match the quality of traditional articles written by journalists.


Figure 2 illustrates the mean ratings for the second set of outcomes, which measure participants' willingness to continue reading the articles they were exposed to and their willingness to read AI-generated news in the future. The control group, which read article excerpts written by journalists, had a mean rating of 2.48 for willingness to continue reading, while the AI-assisted group had a rating of 3.08, and the AI-generated group 3.18. The confidence intervals for the AI-assisted and AI-generated groups do not overlap with those of the control group, indicating statistically significant increases in willingness to continue reading for these groups. Additionally, no significant difference is observed between the AI-assisted and AI-generated groups. This leads to the rejection of hypotheses H7 and H8, which predicted lower willingness to continue reading in the AI-assisted and AI-generated groups. For willingness to read AI-generated news in the future, the control group had a mean rating of 2.37, the AI-assisted group 2.34, and the AI-generated group 2.46. The overlapping confidence intervals suggest no significant differences between the groups.

Table 2 provides a detailed overview of the regression results for the second set of outcomes. They reinforce the findings from Figure 2, indicating a higher willingness to continue reading in these groups. For future reading willingness, the coefficients for both the AI-assisted and AI-generated groups are not significant, corroborating the findings from Figure 2 that there is no significant difference in future reading preferences between the different groups. We therefore reject of hypotheses H9 and H10, which predicted higher willingness to read AI-generated news in the future. These results suggest that knowledge of AI involvement and exposure to it do not significantly influence future reading preferences.

In summary, the data show no significant differences in perceived expertise, readability, and credibility between articles created with the help of AI, those fully generated by AI, and those written by humans. However, participants in the AI-assisted and AI-generated groups exhibit a significantly higher willingness to continue reading the articles. These findings suggest that while AI involvement in news production does not negatively impact the perceived quality of news articles when readers are unaware of the original article generation process, it positively influences readers' willingness to continue reading once the process is revealed. However, no significant effect was detected regarding the willingness to read AI-generated news in the future across the different groups. The rejection of the first set of hypotheses aligns with the notion that modern AI technologies produce content of comparable quality to that of human journalists, while the significant findings in the second outcome highlight an unexpected openness to engaging with AI-generated content, but only in the short term.



§ CONCLUSION


The primary aim of this paper was to examine the perceived quality of AI-assisted and AI-generated news articles compared to human-generated ones, and to assess how subsequent disclosure of AI involvement influences readers' willingness to engage with these articles and their openness to AI-generated news in the future. The research design involved a preregistered experimental study where participants evaluated the perceived quality of news articles without knowing their source, followed by disclosure of whether the articles were AI-assisted, AI-generated, or human-written, to measure changes in willingness to engage with the content and future openness to AI-generated news.

We found no significant differences in perceived credibility, expertise, and readability between articles created by AI, those assisted by AI, and those written by humans. However, participants showed a significantly higher willingness to continue reading articles when aware of AI involvement, whether assisted or fully generated, compared to human-written articles. This suggests that AI does not reduce perceived article quality and  enhances readers' engagement once AI origin is disclosed. However, no significant effect was observed regarding future willingness to read AI-generated news, indicating that AI's role may positively influence immediate engagement, possibly through a novelty effect, but not long-term preferences.

The results regarding our second set of outcomes are particularly noteworthy. We found significant and positive effects on respondents' willingness to continue reading the articles after the true source was revealed, applicable to both treatment groups (AI-assisted and AI-generated). This suggests that readers may not be opposed to engaging with AI-generated or AI-assisted texts and may even be curious to learn more about AI. However, it is important to consider that the increased interest in reading the full article may not be a lasting effect changing preferences for AI-generated news. Participants might have become curious about AI involvement after learning the true authorship, but they have not become more (nor less) open towards reading AI news in general. Future research could explore this increased interest using more targeted intervention research designs, which could provide valuable insights for news producers on how to enhance specific target audiences' interest and increase article engagement.

Regarding respondents' willingness to read AI-generated news articles in the future, we found no significant differences between the treatment groups. Simply reading two AI-assisted or AI-generated texts does not appear to alleviate concerns about AI involvement in news production, as overall willingness scores remained below 2.5 across all groups. These findings may indicate a gradual familiarization process with AI among readers. Over time and with further technological advancements, AI might become more integrated into journalism, particularly in news production, with readers becoming more accustomed to it. Therefore, future research should continue monitoring public attitudes toward AI to detect potential familiarization effects.

Future research could focus on tracking changes in public perception and engagement with AI-generated news over time through longitudinal studies, which would provide insights into whether initial curiosity leads to long-term acceptance or trust. Exploring the role of transparency in disclosing AI involvement would also be important, as different types of transparency might influence how readers perceive and trust AI-generated content. Intervention studies could test different strategies to enhance reader engagement to determine what factors increase willingness to engage with AI-produced news. Expanding this research to include diverse cultural and linguistic contexts would be important, to assess whether these findings are generalizable across different populations. Additionally, studying the impact of AI-generated news on public knowledge could reveal how well readers retain and understand information from AI versus human-written articles.





§ ACKNOWLEDGEMENTS


This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement nr. 883121). We thank Fabio Melliger for excellent research assistance.





apsr




