ChatGPT vs Social Surveys: Probing the Objective and Subjective Human Society
Muzhi Zhou Lu Yu Xiaomin Geng Lan Luo



A Software Visualization Approach for 

Multiple Visual Output Devices

    Malte Hansen
Department of Computer Science 

Kiel University

Kiel, Germany 

malte.hansen@email.uni-kiel.de
Heiko Bielfeldt
Department of Computer Science 

Kiel University

Kiel, Germany 

heikobielfeldt@gmail.com
Armin Bernstetter

GEOMAR Helmholtz Centre for Ocean Research Kiel

Kiel, Germany 

abernstetter@geomar.de
Tom Kwasnitschka

GEOMAR Helmholtz Centre for Ocean Research Kiel

Kiel, Germany 

tkwasnitschka@geomar.de
Wilhelm Hasselbring
Department of Computer Science 

Kiel University

Kiel, Germany 

        hasselbring@email.uni-kiel.de

    September 9, 2024
===============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
The extent to which Large Language Models (LLMs) can simulate the data-generating process for social surveys remains unclear. Current research has not thoroughly assessed potential biases in the sociodemographic population represented within the language model's framework. Additionally, the subjective worlds of LLMs often show inconsistencies in how closely their responses match those of groups of human respondents. In this paper, we used ChatGPT-3.5 to simulate the sampling process and generated six socioeconomic characteristics from the 2020 US population. We also analyzed responses to questions about income inequality and gender roles to explore GPT's subjective attitudes. By using repeated random sampling, we created a sampling distribution to identify the parameters of the GPT-generated population and compared these with Census data. Our findings show some alignment in gender and age means with the actual 2020 US population, but we also found mismatches in the distributions of racial and educational groups. Furthermore, there were significant differences between the distribution of GPT's responses and human self-reported attitudes. While the overall point estimates of GPT's income attitudinal responses seem to align with the mean of the population occasionally, their response distributions follow a normal distribution that diverges from human responses. In terms of gender relations, GPT's answers tend to cluster around a single category, demonstrating a deterministic pattern. We conclude by emphasizing the distinct design philosophies of LLMs and social surveys: LLMs aim to predict the most suitable answers, while social surveys seek to reveal the heterogeneity among social groups.







§ INTRODUCTION

sections/1_introduction.tex



§ RELATED WORK

sections/2_relatedwk.tex



§ RESEARCH DESIGN

sections/3_methodology.tex



§ STUDY 1: CHATGPT AND PROBABILITY SAMPLING: THE OBJECTIVE WORLD

sections/4_study1.tex



§ STUDY 2: CHATGPT AND ATTITUDES TOWARDS INCOME AND GENDER: THE SUBJECTIVE WORLD

sections/5_0study2_Newprompt.tex
sections/5_1study2_income.tex
sections/5_2study2_gender.tex




§ DISCUSSION AND CONCLUSION

sections/6_discussion.tex



§ ETHICAL CONSIDERATIONS

sections/7_ethics.tex






§ APPENDIX

sections/appendix1.tex
sections/appendix2.tex
