# Cultural Value Differences of LLMs: Prompt, Language, and Model Size

Qishuai Zhong Yike Yun Aixin Sun
Nanyang Technological University, Singapore

(May 2024)

###### Abstract

Our study aims to identify behavior patterns in cultural values exhibited by
large language models (LLMs). The studied variants include question ordering,
prompting language, and model size. Our experiments reveal that each tested
LLM can efficiently behave with different cultural values. More interestingly:
(i) LLMs exhibit relatively consistent cultural values when presented with
prompts in a single language. (ii) The prompting language e.g., Chinese or
English, can influence the expression of cultural values. The same question
can elicit divergent cultural values when the same LLM is queried in a
different language. (iii) Differences in sizes of the same model (e.g.,
Llama2-7B vs 13B vs 70B) have a more significant impact on their demonstrated
cultural values than model differences (e.g., Llama2 vs Mixtral). Our
experiments reveal that query language and model size of LLM are the main
factors resulting in cultural value differences.

Cultural Value Differences of LLMs: Prompt, Language, and Model Size

Qishuai Zhong Yike Yun Aixin Sun Nanyang Technological University, Singapore

##  1 Introduction

Since GPT-3 Brown et al. (2020), Large Language Models (LLMs), capable of
generating human-like text based on instructions, have garnered significant
attention from both academia and industry. Numerous benchmarks and datasets
have been created and employed to assess LLMs’ capability in generating human-
like text across various tasks like question-answering, chatbot, and
summarization Clark et al. (2018); Zellers et al. (2019); Hendrycks et al.
(2021a). The open-source leaderboard Park (2023) allows researchers and
engineers to directly compare language models across various dimensions,
spanning from commonsense reasoning to advanced question answering, showcasing
their respective abilities. However, focusing on information content while
ignoring language’s social factors is currently a limitation of natural
language processing (NLP) Hovy and Yang (2021).

Given their capacity to generate human-like text, it is imperative to
investigate whether LLMs demonstrate human-like behaviors stemming from the
internalized values and cultural insights acquired from large-scale training
corpora. As model-generated text gains wider adoption, ethical concerns arise
due to the potential influence of cultural biases embedded in the generated
text on its users Kumar et al. (2023). Hence, an emerging research trend
involves quantifying the cultural biases within language models and
understanding their impact on the models’ performance across various tasks.

The primary methods for assessing values in LLMs typically involve using
social science and psychological instruments originally designed for humans
Feng et al. (2023); Arora et al. (2023) to assess various cultural aspects
quantitatively, or by developing specialized datasets to examine model biases
Parrish et al. (2022); Huang and Xiong (2023). Many studies on social science
instruments primarily evaluate text generated by models in English. However,
historical linguistic research, such as the Whorfian hypothesis proposed by
Sapir-Whorf, suggests that language structure significantly influences
individual perceptions and worldviews Kay and Kempton (1984). Research has
shown that cultural accommodation occurs when individuals engage in
multilingual contexts, as evidenced by experiments with human subjects Harzing
and Maznevski (2002). Similarly, multilingual language models, pre-trained on
text from various languages, can inherit biases and inconsistencies from their
training data Garrido-Muñoz et al. (2021). Therefore, assessments using only
English-based instruments may not fully capture the breadth of knowledge in
multilingual models.

To provide a more comprehensive understanding of LLMs’ cultural values, this
study investigated patterns of cultural values expressed by different models
using three distinct approaches: (i) experimenting with varied prompts in a
single language, (ii) using prompts in different languages, and (iii)
conducting experiments across different models. The pipeline is visualized in
Figure 3 in Appendix A. All sets of experiments were designed and implemented
using Hofstede’s latest Value Survey Module (VSM) Hofstede and Hofstede
(2016), a data collection instrument that quantifies cultural values across
six dimensions Taras et al. (2023).

A total of 6 LLMs were involved in our experiments, with each model being
provided 54 simulated identities to contextualize its response to the VSM
questionnaire. Through our investigation, we found that: (i) LLMs consistently
demonstrate similar cultural values within a single language, despite
variations in prompt content. However, their responses are affected by
alterations in the positioning of options. (ii) LLMs show notably different
cultural values across different languages; and (iii) Differences in the
cultural values expressed by models correlate with variations in text
generation proficiency. For the last finding, although we considered the
models’ text generation proficiency in our study to conduct our analysis and
support our findings, further assessment of the models’ generation capability
is beyond the scope of our research.

##  2 Related Work

Several studies have contributed to detecting social and cultural biases
displayed by models, as values can be inferred from the expression of biases.
Another approach is to incorporate social science models for a direct
evaluation of the values inherent in the models. We review both approaches.

###  2.1 Bias Study of Language Model

Assessing social and cultural biases in language models is crucial to mitigate
associated risks and reveal the values embodied by the models. Liang et al.
(2021) provided a formal comprehension of social biases in language models.
The work identified fine-grained local biases and high-level global biases as
sources of representational biases and proposed the evaluation metrics for
measurement. Subsequently, it introduced the mitigation method. Sheng et al.
(2021) presented the first comprehensive survey on societal biases in language
generation in 2021, identifying their negative impact and exploring methods
for evaluation and mitigation. The study highlighted the challenge of bias
assessment due to the open-domain nature of NLG and the diverse
conceptualizations of bias across cultures. Recently, more studies have
focused on evaluating bias and values in large language models, with
innovative methodologies employed. Cheng et al. (2023) utilized the concept of
markedness, initially linguistic but now a part of social science, to evaluate
models’ stereotypes unsupervisedly. Meanwhile, Kotek et al. (2023) employed a
direct method to assess gender bias in LLMs, revealing models’ tendency to
reflect imbalances over gender due to training on skewed datasets. In Ferrara
(2023), bias in generative language models was defined and its sources, such
as training data and model specifications, were investigated. However, the
study also acknowledged that some biases may persist inevitably due to the
inherent nature of language and cultural norms.

Previous studies have demonstrated diverse techniques for accurately and
efficiently identifying biases. However, they have also underscored the
challenges in mitigating biases in generated text, as biases can be inherited
from human language and culture in training data. This indicates that the
exhibited values of models are shaped by the training data, making it
impossible to dissociate the influence of training data when trying to
understand the patterns of values expressed by models.

###  2.2 Social Science Models

While studies have investigated language models’ social and cultural biases,
there’s still relatively less systematic exploration of how these models
exhibit values under varying circumstances. Quantifying results in this domain
is challenging. Consequently, research instruments initially focused on humans
have been integrated into understanding language models’ values. Feng et al.
(2023) utilized the political compass test to map the political leaning of
language models in a two-dimensional space. Through the experiments conducted,
the study demonstrated that pretrained language models are influenced by the
political leaning inherent in the training data. Regarding culture
measurement, Hofstede’s Value Survey Module (VSM) Hofstede and Hofstede (2016)
and the World Values Survey Inglehart et al. (2014) were employed by Arora et
al. (2023) to explore cross-cultural values embedded in multilingual masked
language models. The evaluation covered 13 languages to probe the models’
cultural values across 13 cultures. The findings indicated that pretrained
language models captured noticeable differences in values between cultures,
albeit with weak correlations to values surveys. Kovač et al. (2023) utilized
three human psychology questionnaires to assess how models’ expression of
values changes with varying contexts, such as varying paragraphs and textual
formats. They introduced the metaphor “LLM as a superposition of perspectives"
to highlight the context-dependent nature of LLM behavior. Shu et al. (2023)
created a dataset covering various persona measurement instruments to evaluate
the consistency of LLMs’ "personality" across different prompts with minor
variations. Their experiments revealed that even minor perturbations notably
impacted the models’ question-answering performance. Therefore, they argued
the current practice of prompting is insufficient to accurately capture model
perceptions. The aforementioned articles challenged the practice of using
psychological models to reveal personalities by regarding language models as
individuals Bodroza et al. (2023); Pan and Zeng (2023).

Summarized from previous research, it is clear that prompt engineering and
training data significantly impact how models express values. However, there
is a need for a systematic study to evaluate these factors comprehensively. In
our study, we systematically explore the expression of cultural values by
models under varying circumstances, including the effects of prompt
engineering, language differences, and model capabilities.

##  3 Measures by VSM

Similarly to previous studies, we utilize a value survey and additional
measurement metrics to evaluate the alignment of cultural values in the LLMs.
Value Survey Module (VSM) Hofstede and Hofstede (2016) is for measuring
cultural values as outlined in Hofstede’s Cultural Dimensions Theory Gerlach
and Eriksson (2021). Despite facing criticism for its psychometric
deficiencies Taras et al. (2023) and simplicity Ercan et al. (1991), its value
representation has become a cornerstone for a substantial body of research on
cross-cultural differences in values Arora et al. (2023). In this study, we
utilize the latest version of the survey (VSM 2013) as the foundational
assessment.

The value test is structured as a questionnaire with 24 questions to evaluate
the interviewees’ cultural values. Another six questions intended to gather
background information about the interviewees are excluded from our study. The
complete questionnaire is in Appendix J. Each question offers respondents five
options, labeled with option IDs from 1 to 5. Option IDs also serve as raw
scores for each question. The authors of the VSM further developed a scoring
system based on each question’s raw score, comprising six dimensions for
measuring cultural values: Power Distance (PDI), Individualism (IDV),
Uncertainty Avoidance (UAI), Masculinity (MAS), Long-term Orientation (LTO),
and Indulgence (IVR). Each dimension is calculated using a formula with the
raw scores from four survey questions. The complete list of formulas is in
Appendix B.

All experiments are conducted using prompts derived from the questionnaire.
The prompt is delivered in a zero-shot manner, and the LLM is expected to
respond in JSON format, specifying the chosen option ID and the rationale
behind the selection. We require models to respond with option IDs to mitigate
the performance degradation outlined by Zheng et al. (2024). Prompt samples
are depicted in Figure 4 in Appendix A. In each prompt, we give instructions
on the reply format, provide a survey question, and supply a simulated
background identity. The simulation provided a target for the model to
contextualize the response. Contextual simulation or targeting specific groups
of people is a common methodology used by previous studies to guide the
generation Kovač et al. (2023); Narayanan Venkit et al. (2023); Ramezani and
Xu (2023); Cheng et al. (2023).

###  3.1 Experiment Set

The experiment conducted in this study consists of multiple experiment sets.
Each set is defined by a unique combination of three hyper-parameters: (i) the
tested LLM, (ii) the prompt language, and (iii) whether options are shuffled.

Within each experiment set, the language model was presented with a curated
collection of simulated identities, each comprising three variables: (i)
nationality, (ii) age, and (iii) gender to furnish context for the model’s
responses to questions. The study encompasses nine nationalities (refer to the
full list in Appendix C), two genders, and three age groups (25, 35, 45),
resulting in a total of 54 identities. These variables align with the VSM
survey, encompassing interviewees from various countries, genders, and ages.
The chosen nations are globally diverse, representing a range of cultures. To
prevent coincidence, each question was queried ten times with different seeds.
Consequently, we could collect $10\times 24\times 54=12960$ responses for each
experiment set. During the analysis, we calculate the average of the ten
outputs as the final output for a simulated identity, which is used as a
single data point (a 24-d vector) in the experiment set.

###  3.2 Measures by VSM Raw Scores

Each set of responses from a simulated identity is represented as a
24-dimensional vector, essential for comparisons within and between experiment
sets. To evaluate the strength of relationships between these groups, we
calculate the Pearson correlation coefficients ($\rho$) and $p$-values among
the centroid vectors. The $\rho$ values help determine whether two response
vectors exhibit a high correlation, indicating a shared and similar pattern in
their responses. The $p$-value tests the null hypothesis that no relationship
exists between the two compared vectors. If $p<0.05$, then we reject the null
hypothesis and conclude that there is a significant relationship between the
vectors.

###  3.3 Measures by VSM Scores

Using VSM formulas in Appendix B, we can generate 6-dimensional score vectors
(i.e., PDI, IDV, UAI, MAS, LTO, and IVR) from the 24-d vectors.

#### Intra-set Disparity Measurement

In Hofstede’s research, VSM scores are analyzed nationally to explore cultural
value differences between countries. The scores for the nine nations involved
in this study are displayed in Appendix D, where each nation’s score
represents the average of all responses from its interviewees. Similarly, we
calculate the national average for model responses of each experiment set. We
then use the standard deviation, denoted as $\sigma_{m}(v_{i})$, to assess the
dimensional disparity among nations, where $v_{i}$ represents the dimension.
Similar calculations are performed to compute $\sigma_{h}(v_{i})$ for the
human results.

The mean values for each dimension, across all experiment sets and human
results, range from $-60$ to $100$, indicating that comparing the disparity
between models and human results is reasonable. Appendix E provides the
complete list of mean values.

We then define the distance among nations observed for humans as $D_{h}$ (see
Eq. 1, where $V$ represents the list of dimensions). This illustrates the
variations in cultural values observed among humans. Similarly, the overall
disparity among nations observed in each experiment set is denoted as $D_{m}$.
Then, we define the ratio of $D_{m}$ over $D_{h}$ as the “Model Cultural
Disparity (MCD).", shown in Eq. 3:

$\displaystyle D_{h}$ $\displaystyle=\frac{1}{|V|}\sum_{v_{i}\in
V}(\sigma_{h}(v_{i}))$ (1) $\displaystyle D_{m}$
$\displaystyle=\frac{1}{|V|}\sum_{v_{i}\in V}(\sigma_{m}(v_{i}))$ (2)
$\displaystyle MCD$ $\displaystyle=\frac{D_{m}}{D_{h}}$ (3)

MCD compares the dispersion of cultural values exhibited by models based on
simulated nations to that observed among humans in Hofstede’s study.

#### Inter-set Disparity Measurement

The intra-set disparity underscores the impact of contextual information on
the models’ expression of cultural values. Furthermore, our pipeline uses
inter-set disparity to explore how changes in any of the three hyper-
parameters—shuffling of options, language, and the tested model—affect the
expression of cultural values.

We employ clustering methodologies, Davies-Bouldin Index (DBI) Davies and
Bouldin (1979) and the Silhouette Score ($SS$) Rousseeuw (1987), to assess the
effectiveness of separation between each pair of experiment sets. Detailed
descriptions of the two metrics can be found in Appendix F. In our study, we
pre-define the model responses from each experiment set as a cluster,
comprising 54 data points, as detailed in Section 3.1.

Additionally, we have introduced a new measurement method, the Silhouette
Score with Human Reference ($SS_{h}$), to measure the absolute disparity
between pairs of sets, taking human results as the reference point.

$SS_{h}$ is designed based on the Silhouette Score, utilizing nationally
aggregated average VSM scores:

$\displaystyle a_{h}(n_{i})$ $\displaystyle=\frac{1}{|C_{h}|-1}\sum_{n_{j}\in
C_{h},i\neq j}d(n_{i},n_{j})$ (4) $\displaystyle SS_{h}$
$\displaystyle=\frac{1}{2N}\sum\frac{b(n_{i})-a(n_{i})}{a_{h}(n_{i})}$ (5)

where $a_{h}(n_{i})$ signifies the mean distance from that nation $n_{i}$ to
all other nations in the human results. $a(n_{i})$ represents the mean
distance from the $i^{th}$ nation to all other nations within the same
experiment set. $b(n_{i})$ denotes the mean distance from the same nation to
all nations in another experiment set. Additionally, $N$ denotes the
consistent number of nations involved in this study. Unlike the previous two
metrics, which concentrate solely on the compared set-pair, $SS_{h}$ measures
the effectiveness of the separation between sets by referencing human results.
An $SS_{h}$ value exceeding one suggests that the separation between the two
sets is more pronounced than the disparity observed among humans from various
nations.

##  4 Experiment Setting and RQs

Given that the value survey is structured as a questionnaire, we have
specifically chosen and employed models fine-tuned for chat purposes for this
study. A total of six models are evaluated in this study, including members of
the Llama2 family Touvron et al. (2023): Llama2-7b-chat-hf, Llama2-13b-chat-
hf, and Llama2-70b-chat-hf; members of the Qwen family Bai et al. (2023):
Qwen-14b-chat and Qwen-72b-chat; and Mixtral-8x7B-Instruct-v0.1 Jiang et al.
(2024), which features a different architecture from the other models.

All experiments were conducted using Vllm Kwon et al. (2023) with Transformers
Wolf et al. (2020) to achieve faster inference. We utilized four Nvidia A6000
cards with CUDA 12.2. We used the default config.json and framework parameters
for the models’ text generation.

Through experiments, we aim to gain insights into three Research Questions
(RQs).

RQ1: Can large language models consistently express cultural values when
presented with perturbed questions in a single language? We focus on how
responses of the same model vary with changes in contextual information and
option order shuffling.

RQ2: How does language affect the expression of cultural values in models? We
examine the consistency of cultural values expressed by models when identical
questions are posed in different languages.

RQ3: What can we infer about models’ expression of cultural values when
comparing them? We evaluate whether models from the same family show more
consistent cultural values and investigate how differences in text generation
capabilities relate to variations in cultural values.

##  5 Experiment Results

Among the 24 questions in the VSM 2013 survey, questions 15 and 18 pertain to
the interviewee’s recent mental and physical health. Consequently, we assign
the most neutral option (option 3) to these two questions. We similarly assign
option 3 for any unrecognizable responses from the models.

Initially, we requested responses in Chinese when querying models with Chinese
prompts. However, about 7% of Llama2-7b-chat-hf’s responses and 24% of
Llama2-13b-chat-hf’s responses were unrecognizable, in contrast to other
models which had at least 99% recognizable responses. As a result, these two
models are required to respond in English to Chinese prompts.

###  5.1 Prompt Variants (RQ1)

To evaluate a single model’s consistency in expressing cultural values within
a single language, we developed prompt variations focusing on two aspects:
simulated identity and options order. The former modifies only the context
presented to the model, whereas the latter entails further prompt engineering.
The impact of simulated identity is assessed within the experiment set, while
the effectiveness of options order is evaluated through inter-set methods.

#### Simulated Identity

Within each experiment set, the model is queried with 54 simulated identities.
VSM raw scores and intra-set measurements are used to examine the impact of
simulated identities.

Based on raw scores, each tested model consistently produces results with a
similar distribution, irrespective of changes in the context. The correlation
coefficients for the average score vectors, grouped by context variables in
Table 5 in the Appendix, indicate that responses across different identities
are highly correlated.

Similarly, the intra-set measurements based on VSM scores presented in Table
6, show that the simulated nations assigned to the LLMs exhibit significantly
less cultural value diversity compared to the differences observed among human
interviewees from those nations.

In summary, the evaluated models produce responses with relatively consistent
cultural values and show limited sensitivity to changes in the context of the
prompts. The cultural values learned from the training corpus help mitigate
the effects of variations in the simulated identities provided in the context.

Models | $DBI$ $\downarrow$ |  $SS$ $\uparrow$ |  $SS_{h}$ $\uparrow$
---|---|---|---
Llama2-7b-chat-hf | 1.837 | 0.169 | 0.430
Llama2-13b-chat-hf | 1.694 | 0.205 | 0.228
Llama2-70b-chat-hf | 0.658 | 0.572 | 0.574
Qwen-14b-chat | 0.981 | 0.409 | 1.033
Qwen-72b-chat | 0.825 | 0.478 | 0.483
Mixtral-8x7B | 0.542 | 0.641 | 0.680
Table 1: Results of three measurements are listed in the table to quantify the
disparity between model responses for the two sets, “Eng w/o shuffled options"
and “Eng w. shuffled options". Figures showing the greatest distinctness are
highlighted in bold in each column.

#### Shuffled Options

As noted by Zheng et al. (2024), LLMs are susceptible to selection bias,
primarily due to token bias and, to a lesser extent, position bias, both of
which originate from the training data. Accordingly, our experiment maintains
the original option IDs and their corresponding text, only altering their
positions to minimize token bias.

We evaluate the consistency of the model’s cultural values despite selection
bias by analyzing changes in the distribution of raw scores and measuring the
inter-set disparity between the "Eng" and "Eng w. Shuffle" experiment sets for
each model.

The Centroid vector of each experiment set represents the distribution of the
set. The correlation coefficient and $p$-value are computed between the
centroids, with comprehensive results presented in Table 7 in Appendix. These
results indicate that most models maintain highly correlated score
distributions after option shuffling. However, the overall correlation scores
are noticeably lower than those calculated for simulated identities.

The inter-set disparity measurement results, as shown in Table 1, display the
effect of shuffling to models from the aspect of VSM score. The results of
$DBI$ and $SS$ suggest that the experiment sets, pre-divided based on
shuffling, are clustered but not distinctly separated from each other. When
evaluating the disparity between sets using $SS_{h}$, we find that most models
exhibit a noticeable absolute shift in cultural values between the sets, which
does not correspond to the significant differences observed among humans from
diverse nations.

The experiment results show that models remain vulnerable to selection bias,
consistent with the findings reported in Zheng et al. (2024). Unlike human
behavior, models fail to maintain consistent cultural values in the face of
textual ambiguities.

|  |
---|---|---
|  |
(a) Llama2-7b-chat-hf (b) Llama2-13b-chat-hf (c) Llama2-70b-chat-hf (d)
Qwen-14b-chat (e) Qwen-72b-chat (f) Mixtral-8x7B

Figure 1: The 6-d VSM scores for different experiment sets for each model are
visualized using the t-SNE technique van der Maaten and Hinton (2008) to
facilitate direct comparisons. Results from English queries (denoted as "Eng")
are displayed with black circles; results from English with Shuffled Options
(denoted as "Eng w. Shuffle") are shown with pink stars; and results from
Chinese (denoted as "Chn") are represented by green squares.

Model response distributions across different experiment sets are visualized
using t-SNE in Figure 1 van der Maaten and Hinton (2008). The visualization
also indicates that most models demonstrate less or comparable separation
effectiveness between sets divided by “Shuffling of options" compared to those
split by “Language".

###  5.2 Language Variants (RQ2)

Models | $DBI$ $\downarrow$ |  $SS$ $\uparrow$ |  $SS_{h}$ $\uparrow$
---|---|---|---
Llama2-7b-chat-hf | 0.962 | 0.423 | 1.357
Llama2-13b-chat-hf | 0.720 | 0.533 | 0.581
Llama2-70b-chat-hf | 0.799 | 0.499 | 0.707
Qwen-14b-chat | 1.846 | 0.215 | 0.622
Qwen-72b-chat | 0.529 | 0.646 | 0.961
Mixtral-8x7B | 0.651 | 0.581 | 0.660
Table 2: Results of three measurements are listed in the table to quantify the
disparity between model responses for the two sets, “Eng w/o shuffled options"
and “Chn w/o shuffled options". Figures showing the greatest distinctness are
highlighted in bold in each column.

In addition to varying prompts within the same language, we conduct
experiments to evaluate each model’s behavior when prompted in English and
Chinese. For the Chinese queries, we carefully crafted prompts using the
Chinese version of the VSM 2013 questionnaires. Contextual information of the
simulated identities is manually translated.

Correlation coefficients and $p$-values of this group of comparisons are
displayed in Table 8 in Appendix H. One model exhibits a $p$-value exceeding
the threshold, indicating no significant relationship between its outputs for
English and Chinese questions. Although the $p$-values for other models remain
below the threshold, the overall correlation coefficient is lower than that
observed with prompt variants. This suggests that language impacts the models’
choice of options more significantly than the shuffling of option order.

In addition to the raw scores, the inter-set disparity measurement results
based on VSM scores are detailed in Table 2, with a comprehensive analysis of
values provided in Appendix H. Based on the results of $DBI$ and $SS$, we find
no significant differences between comparisons based on language and
“shuffling". However, the $SS_{h}$ results suggest that when queried with the
same questions in a different language, the model is expected to exhibit
cultural values with a variability of at least 50%, akin to that of an
individual from another country. Language differences can result in a more
distinctive separation in expressing cultural values. The t-SNE figures in
Figure 1 also clearly illustrate that most models express cultural values more
variably when queried in different languages.

The discrepancies between the initial measurements and $SS_{h}$ stem from
differences in their formulas. $DBI$ and $SS$, focus on the ratio between
inter-set distance and intra-set disparity. These measurements may not provide
robust values if model responses are sparse. In contrast, the $SS_{h}$ formula
considers inter-set distance with human disparity (a constant), providing an
absolute measure of inter-set disparity.

Summarizing the findings, we observe that language significantly influences
the models’ responses and the cultural values expressed by those responses.
This observation aligns with research findings Norton (1997) that suggest
values are commonly conveyed through language. We argue that the diverse
cultural values expressed by the model in various languages are acquired from
the distinct training corpora of those languages, similar to other types of
knowledge transferred from training corpora to the language model Lin et al.
(2019); Krishna et al. (2023).

|
---|---
|

(a) $SS_{h}$ among Models with English Questions (b) MMLU Distance among
Models

(c) $SS_{h}$ among Models with Chinese Questions (d) $SS_{h}$ among Models
cross Languages

Figure 2: The three red heatmaps display the $SS_{h}$ values among models,
with darker colors highlighting greater disparities. The green heatmap
displays the differences in MMLU scores among models, corresponding to the
disparities observed in the adjacent red heatmap.

###  5.3 Models Comparison (RQ3)

We now analyze the patterns of cultural values expressed by different models
based on their inter-set disparity. This analysis encompasses three types of
comparisons: (i) among models queried solely in English (without “shuffling"),
(ii) among models queried solely in Chinese, and (iii) cross-language
comparisons. All comparisons utilize $SS_{h}$ values. We represent all three
comparison subsets with heatmap charts, as shown in Figure 2.

Observations from Heatmaps (a) and (c) in Figure 2 reveal that models from the
same family do not necessarily exhibit closer cultural value alignment.
Additionally, all Llama2 models, irrespective of size, are trained using the
same datasets for the same duration Touvron et al. (2023). The Qwen technical
report Bai et al. (2023) also indicates that identical datasets and
hyperparameters are applied across various model sizes during pretraining and
fine-tuning stages (SRF and RLHF).

Based on the findings: (i) models from the same family do not guarantee
consistency in expressing cultural values; (ii) models with the same
background receive uniform training; and (iii) larger models within the same
family demonstrate better text-generation performance. We can deduce that
variations in cultural values among models of the same family are linked to
differences in their text-generation capabilities instead of training data.
Larger models in the same family are guaranteed to handle complex patterns,
understand context more effectively, and generalize better to unseen data. As
a result, they are more adept at comprehending questions posed in value tests
and generating more appropriate responses compared to smaller models.

We further link our findings with the evaluation results of generation. A
common evaluation all six models have undergone is the MMLU (Massive Multitask
Language Understanding) test Hendrycks et al. (2021b). Differences in MMLU
scores among models are displayed in Figure 2, Heatmap (b). A large $SS_{h}$
value between two models often corresponds with a significant gap in MMLU
scores. However, the reverse is not necessarily true: a small gap in MMLU
scores does not guarantee a small $SS_{h}$ value between models.

Additionally, in the heatmap (d) of Figure 2, the overall disparities between
models across languages are significantly larger than those observed within a
single language. The marked inter-set disparities noted in cross-language
comparisons indicate that language variations can cause substantial
differences in cultural values among models.

Our hypothesis that differences in cultural values correlate with variations
in model capabilities is based on observations. Developing a testing mechanism
that simultaneously evaluates text quality, the expression of cultural values,
and their alignment is part of future work. This approach will enhance our
understanding of how language model performance impacts the expression of
cultural values.

##  6 Conclusion

In this study, we developed an investigative pipeline to assess the behavior
of large language models concerning expressions of cultural values. Our
results show that (i) Cultural values tend to remain relatively consistent
across variations in prompts, especially when changes are limited to content
alone. (ii) LLMs exhibit significantly divergent cultural values across
different languages, and (iii) The difference in cultural values among models
is relevant to variations in the models’ overall proficiency in text
generation. Furthermore, upon comparing the results illustrating the second
and third findings, we find that language variants can lead to greater
disparities in cultural values. Language emerges as the most significant
factor influencing the cultural values exhibited by the models.

##  7 Limitations

This study has a few limitations that require further investigation in future
research. (i) We limited our exploration of cultural values expressed by
models to the 24 questions of the VSM 2013 survey, which has been criticized
for its simplicity. Therefore, future research should consider incorporating
additional cultural value surveys to investigate the models’ behavior further.
(ii) This study evaluated and assessed only six models. To further validate
the findings regarding the models’ expression of cultural values and their
performance differences, additional models should be explored and included in
future studies. (iii) In our experiments, models are prompted within a
narrowly defined context to generate responses in a zero-shot manner,
conditioned solely on the provided context. Future studies should extend
beyond direct prompts, exploring how models express cultural values when
supplied with extensive past experiences and acting as believable agents Park
et al. (2023). (iv) A new evaluation pipeline or mechanism needs to be
designed to assess and quantify the relationship between specific cultural
value patterns and the generated text’s quality. This would build upon the
current finding that variations in text quality result in different cultural
values. (v) Although we have observed variations in the cultural values of
large language models when the same questions are asked in different
languages, we have not thoroughly analyzed user preferences concerning these
differences. Future research should develop a systematic approach to assess
how language-induced disparities in cultural values impact users and to
formulate strategies to mitigate any negative effects.

##  8 Ethical Consideration

All experiments described in this study rely on data from the widely
recognized Value Survey Module (VSM) 2013 Hofstede and Hofstede (2016) and
utilize open-source language models. While our analysis includes human subject
data, it is important to note that this data is derived from the well-
established findings of the VSM 2013 study. Additionally, although our
research examines the responses of various large language models to assess
cultural values, we explicitly avoid ranking these models to maintain
objectivity and ethical integrity.

## References

  * Arora et al. (2023) Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2023.  Probing pre-trained language models for cross-cultural differences in values.  _Preprint_ , arXiv:2203.13722.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _Preprint_ , arXiv:2309.16609.
  * Bodroza et al. (2023) Bojana Bodroza, Bojana M. Dinic, and Ljubisa Bojic. 2023.  Personality testing of gpt-3: Limited temporal reliability, but highlighted social desirability of gpt-3’s personality instruments results.  _Preprint_ , arXiv:2306.04308.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Cheng et al. (2023) Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023.  Marked personas: Using natural language prompts to measure stereotypes in language models.  _Preprint_ , arXiv:2305.18189.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _ArXiv_ , abs/1803.05457.
  * Davies and Bouldin (1979) David L. Davies and Donald W. Bouldin. 1979.  A cluster separation measure.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , PAMI-1(2):224–227.
  * Ercan et al. (1991) G Ercan, Hamad Nasif, Bahman Al-Daeaj, and Mary S Ebrahimi. 1991.  Methodological problems in cross-cultural research: An updated review.  _Management International Review_ , pages 79–91.
  * Feng et al. (2023) Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023.  From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.
  * Ferrara (2023) Emilio Ferrara. 2023.  Should chatgpt be biased? challenges and risks of bias in large language models.  _First Monday_.
  * Garrido-Muñoz et al. (2021) Ismael Garrido-Muñoz , Arturo Montejo-Ráez , Fernando Martínez-Santiago , and L. Alfonso Ureña-López . 2021.  A survey on bias in deep nlp.  _Applied Sciences_ , 11(7).
  * Gerlach and Eriksson (2021) Philipp Gerlach and Kimmo Eriksson. 2021.  Measuring cultural dimensions: External validity and internal consistency of hofstede’s VSM 2013 scales.  _Front. Psychol._ , 12:662604.
  * Harzing and Maznevski (2002) Anne-Wil Harzing and Martha Maznevski. 2002.  The interaction between language and culture: A test of the cultural accommodation hypothesis in seven countries.  _Language and Intercultural Communication_ , 2(2):120–139.
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring massive multitask language understanding.  _Preprint_ , arXiv:2009.03300.
  * Hofstede and Hofstede (2016) G Hofstede and G. J. Hofstede. 2016.  VSM 2013.  https://geerthofstede.com/research-and-vsm/vsm-2013/.  Accessed: 2024-1-11.
  * Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021.  The importance of modeling social factors of language: Theory and practice.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 588–602, Online. Association for Computational Linguistics.
  * Huang and Xiong (2023) Yufei Huang and Deyi Xiong. 2023.  Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models.  _Preprint_ , arXiv:2306.16244.
  * Inglehart et al. (2014) R. Inglehart, C. Haerpfer, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris, E. Ponarin, B. Puranen, and et al. 2014.  World values survey: Round six - country-pooled datafile version.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024\.  Mixtral of experts.  _Preprint_ , arXiv:2401.04088.
  * Kay and Kempton (1984) Paul Kay and Willett Kempton. 1984.  What is the sapir-whorf hypothesis?  _Am. Anthropol._ , 86(1):65–79.
  * Kotek et al. (2023) Hadas Kotek, Rikker Dockum, and David Sun. 2023.  Gender bias and stereotypes in large language models.  In _Proceedings of The ACM Collective Intelligence Conference_ , CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery.
  * Kovač et al. (2023) Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023.  Large language models as superpositions of cultural perspectives.  _Preprint_ , arXiv:2307.07870.
  * Krishna et al. (2023) Kundan Krishna, Saurabh Garg, Jeffrey P. Bigham, and Zachary C. Lipton. 2023.  Downstream datasets make surprisingly good pretraining corpora.  _Preprint_ , arXiv:2209.14389.
  * Kumar et al. (2023) Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023.  Language generation models can cause harm: So what can we do about it? an actionable survey.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3299–3321, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  _Preprint_ , arXiv:2309.06180.
  * Liang et al. (2021) Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021\.  Towards understanding and mitigating social biases in language models.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 6565–6576. PMLR.
  * Lin et al. (2019) Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019.  Open sesame: Getting inside bert’s linguistic knowledge.  _Preprint_ , arXiv:1906.01698.
  * Narayanan Venkit et al. (2023) Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023.  Nationality bias in text generation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 116–122, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Norton (1997) Bonny Norton. 1997.  Language, identity, and the ownership of english.  _TESOL Quarterly_ , 31(3):409–429.
  * Pan and Zeng (2023) Keyu Pan and Yawen Zeng. 2023.  Do llms possess a personality? making the mbti test an amazing evaluation for large language models.  _Preprint_ , arXiv:2307.16180.
  * Park (2023) Daniel Park. 2023.  Open-llm-leaderboard-report.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  _Preprint_ , arXiv:2304.03442.
  * Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022.  Bbq: A hand-built bias benchmark for question answering.  _Preprint_ , arXiv:2110.08193.
  * Ramezani and Xu (2023) Aida Ramezani and Yang Xu. 2023.  Knowledge of cultural moral norms in large language models.  _Preprint_ , arXiv:2306.01857.
  * Rousseeuw (1987) Peter J. Rousseeuw. 1987.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _Journal of Computational and Applied Mathematics_ , 20:53–65.
  * Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021.  Societal biases in language generation: Progress and challenges.  _Preprint_ , arXiv:2105.04054.
  * Shu et al. (2023) Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Dallas Card, and David Jurgens. 2023.  You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments.  _Preprint_ , arXiv:2311.09718.
  * Taras et al. (2023) Vas Taras, Piers Steel, and Madelynn Stackhouse. 2023.  A comparative evaluation of seven instruments for measuring values comprising hofstede’s model of culture.  _Journal of World Business_ , 58(1):101386.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Huggingface’s transformers: State-of-the-art natural language processing.  _Preprint_ , arXiv:1910.03771.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Zheng et al. (2024) Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024.  Large language models are not robust multiple choice selectors.  _Preprint_ , arXiv:2309.03882.

##  Appendix A Investigation Pipeline and Prompt Format

Figure 3: Pipeline of investigations, exploring cultural values alignment in
LLMs in three steps. (i) Evaluating cultural values exhibited by an LLM
queried by a single language but with variants of prompts. (ii) Assessing
cultural values in the context of different languages. (iii) Examining
cultural values exhibited by different LLMs, within and across model families
and in different model sizes. Figure 4: Prompt samples for the two languages
used in the experiment. In both samples, the syntax highlighted in red is
copied from the original question in the questionnaire. During the VSM 2013
testing, there are approximately nine types of questions. All customized
components are embedded with the respective values when querying the model.

##  Appendix B VSM Dimension Formula

$\displaystyle PDI$ $\displaystyle=$ $\displaystyle
35*(m_{7}-m_{2})+25*(m_{20}-m_{23})+C$ (6) $\displaystyle IDV$
$\displaystyle=$ $\displaystyle 35*(m_{4}-m_{1})+35*(m_{9}-m_{6})+C$ (7)
$\displaystyle MAS$ $\displaystyle=$ $\displaystyle
35*(m_{5}-m_{3})+35*(m_{8}-m_{10})+C$ (8) $\displaystyle UAI$ $\displaystyle=$
$\displaystyle 40*(m_{18}-m_{15})+25*(m_{21}-m_{24}+C$ (9) $\displaystyle LTO$
$\displaystyle=$ $\displaystyle 40*(m_{13}-m_{14})+25*(m_{19}-m_{22})+C$ (10)
$\displaystyle IVR$ $\displaystyle=$ $\displaystyle
35*(m_{12}-m_{11})+40*(m_{17}-m_{16})+C$ (11)

##  Appendix C Nationalities for Experiment

The full list of nationalities used in experiments for simulated identities
includes U.S.A, China, France, Germany, Brazil, India, Singapore, Japan, and
South Africa.

##  Appendix D Human Results of VSM Scores

Human results, grouped by nations, are presented in Table 3.

Nations | Dimensional Mean
---|---
PDI | IDV | MAS | UAI | LTO | IVR
U.S.A. | 40 | 91 | 62 | 46 | 26 | 68
China | 80 | 20 | 66 | 30 | 87 | 24
France | 68 | 71 | 43 | 86 | 63 | 48
Germany | 35 | 67 | 66 | 65 | 83 | 40
Brazil | 69 | 38 | 49 | 76 | 44 | 59
India | 77 | 48 | 56 | 40 | 51 | 26
Singapore | 74 | 20 | 48 | 8 | 72 | 46
Japan | 54 | 46 | 95 | 92 | 88 | 42
South Africa | 49 | 65 | 63 | 49 | 34 | 63
Table 3: Human results for the nine nations involved in the experiments.

##  Appendix E Mean Values of VSM Scores

The mean values for each VSM dimension for all experiment sets and human
results are outlined in Table 4

Models | Dimensional Mean
---|---
PDI | IDV | MAS | UAI | LTO | IVR
Llama2-7b-chat-hf (Eng) | 18 | 20 | 15 | 13 | -12 | 82
Llama2-7b-chat-hf (Eng w. Shuffle) | 22 | 8 | 4 | 17 | -9 | 33
Llama2-7b-chat-hf (Chn) | -18 | 94 | -58 | -52 | 3 | 74
Llama2-13b-chat-hf (Eng) | 22 | 45 | -5 | -4 | 20 | 22
Llama2-13b-chat-hf (Eng w. Shuffle) | 40 | 29 | -8 | -6 | 24 | 14
Llama2-13b-chat-hf (Chn) | 17 | 1 | -2 | 0 | -3 | 18
Llama2-70b-chat-hf (Eng) | -16 | 67 | -33 | -38 | 4 | 49
Llama2-70b-chat-hf (Eng w. Shuffle) | -12 | 28 | -4 | -23 | 0 | 40
Llama2-70b-chat-hf (Chn) | -32 | 30 | -39 | -33 | -47 | 57
Qwen-14b-chat (Eng) | 28 | 83 | -20 | -17 | -5 | 13
Qwen-14b-chat (Eng w. Shuffle) | -11 | 7 | 2 | -1 | 1 | -10
Qwen-14b-chat (Chn) | -7 | 72 | -55 | 1 | -1 | 56
Qwen-72b-chat (Eng) | -13 | 74 | -40 | -2 | -26 | 26
Qwen-72b-chat (Eng w. Shuffle) | 14 | 47 | -27 | -1 | 2 | 22
Qwen-72b-chat (Chn) | 7 | 11 | -8 | -33 | 12 | 24
Mixtral-8x7B (Eng) | -33 | 70 | 34 | -31 | 2 | 47
Mixtral-8x7B (Eng w. Shuffle) | 4 | 30 | 9 | -34 | 15 | 44
Mixtral-8x7B (Chn) | -56 | 48 | 0 | 1 | 29 | 38
Hofstede’s Research | 61 | 52 | 61 | 55 | 61 | 46
Table 4: The mean values for each VSM dimension for all experiment sets and
human results are calculated. These mean values are presented in integer
format to maintain consistency with the human results listed in Table 3.

##  Appendix F Clustering Measurement Methods

  * •

Davies-Bouldin Index (DBI) Davies and Bouldin (1979): The metric quantifies
the average similarity between each cluster. In our case, it offers an
overview of the disparity in models’ cultural values at the experiment set
level. We calculate the DBI value for each pair of sets. The formula is given
by:

$DBI(e_{i},e_{j})=\left(\frac{S(e_{i})+S(e_{j})}{M(e_{i},e_{j})}\right)$ (12)

where $S(e_{i})$ is the average distance of all points in set $e_{i}$ to the
centroid of set $e_{i}$, $S(e_{j})$ is the average distance of all points in
set $e_{j}$ to the centroid of the set $e_{j}$, and $M(e_{i},e_{j})$ is the
distance between the centroids of sets $e_{i}$ and $e_{j}$. The lower the DBI
value, the better the separation between the two sets. If the DBI value is
larger than one, it suggests that the separation between clusters is not very
distinct.

  * •

Silhouette Score ($SS$) Rousseeuw (1987): The metric evaluates clustering
quality by comparing each point’s similarity to its cluster against other
clusters. We use this metric to compare model outputs between any two sets to
assess the effectiveness of their separation. The formula for the Silhouette
Score $s(i)$ of a single model output $i$ is given by:

$\displaystyle a(p_{i})$ $\displaystyle=\frac{1}{|C_{1}|-1}\sum_{p_{j}\in
C_{1},i\neq j}d(p_{i},p_{j})$ (13) $\displaystyle b(p_{i})$
$\displaystyle=\frac{1}{|C_{2}|}\sum_{p_{j}\in C_{2}}d(p_{i},p_{j})$ (14)
$\displaystyle SS$
$\displaystyle=\frac{1}{|C_{1}|+|C_{2}|}\sum\frac{b(p_{i})-a(p_{i})}{\max(a(p_{i}),b(p_{i}))}$
(15)

where $a(p_{i})$ is the mean distance of data point $p_{i}$ to all other data
points in the same set $C_{1}$, $b(p_{i})$ is the mean distance of $p_{i}$ to
all points in the opposite set $C_{2}$. Our study computes the average score
across all points from two sets to determine the disparity score between them.
The silhouette score ranges from -1 to 1, where a higher value indicates more
effective separation between clusters.

Models | Identity Context
---|---
Nation | Age | Gender
PCC ($\rho$) | PCC ($\rho$) | PCC ($\rho$)
Llama2-7b-chat-hf (Eng) | 0.969 | 0.987 | 0.925
Llama2-7b-chat-hf (Eng w. Shuffle) | 0.942 | 0.994 | 0.949
Llama2-7b-chat-hf (Chn) | 0.842 | 0.971 | 0.969
Llama2-13b-chat-hf (Eng) | 0.978 | 0.993 | 0.996
Llama2-13b-chat-hf (Eng w. Shuffle) | 0.969 | 0.993 | 0.987
Llama2-13b-chat-hf (Chn) | 0.993 | 0.997 | 0.998
Llama2-70b-chat-hf (Eng) | 0.991 | 1.000 | 0.995
Llama2-70b-chat-hf (Eng w. Shuffle) | 0.987 | 0.999 | 0.996
Llama2-70b-chat-hf (Chn) | 0.969 | 0.996 | 0.995
Qwen-14b-chat (Eng) | 0.934 | 0.992 | 0.995
Qwen-14b-chat (Eng w. Shuffle) | 0.752 | 0.905 | 0.837
Qwen-14b-chat (Chn) | 0.807 | 0.939 | 0.858
Qwen-72b-chat (Eng) | 0.934 | 0.992 | 0.995
Qwen-72b-chat (Eng w. Shuffle) | 0.943 | 0.986 | 0.994
Qwen-72b-chat (Chn) | 0.915 | 0.988 | 0.987
Mixtral-8x7B (Eng) | 0.992 | 0.997 | 0.998
Mixtral-8x7B (Eng w. Shuffle) | 0.995 | 0.999 | 0.998
Mixtral-8x7B (Chn) | 0.947 | 0.989 | 0.953
Table 5: The Pearson Correlation Coefficient $\rho$ among model responses, queried with single-language prompts featuring variant simulated identities, are provided above. The average correlation coefficients are computed over the grouped identity context, and all p-values $p\ll 0.05$. Correlation coefficients below 0.9 are in boldface.  Models | Dimensional Standard Deviation | Distance | MCD
---|---|---|---
PDI | IDV | MAS | UAI | LTO | IVR
Llama2-7b-chat-hf (Eng) | 7.587 | 4.648 | 6.960 | 7.014 | 11.402 | 14.096 | 8.618 | 0.424
Llama2-7b-chat-hf (Eng w. Shuffle) | 6.831 | 7.047 | 3.903 | 6.549 | 7.712 | 9.916 | 6.993 | 0.344
Llama2-7b-chat-hf (Chn) | 13.629 | 21.835 | 14.901 | 5.429 | 14.681 | 10.993 | 13.578 | 0.668
Llama2-13b-chat-hf (Eng) | 5.833 | 5.952 | 3.608 | 2.850 | 4.187 | 3.004 | 4.239 | 0.209
Llama2-13b-chat-hf (Eng w. Shuffle) | 3.109 | 4.301 | 3.134 | 2.530 | 6.586 | 4.148 | 3.888 | 0.191
Llama2-13b-chat-hf (Chn) | 4.131 | 2.758 | 3.394 | 0.919 | 3.153 | 4.085 | 3.074 | 0.151
Llama2-70b-chat-hf (Eng) | 4.160 | 1.096 | 2.789 | 4.866 | 3.197 | 5.113 | 3.537 | 0.174
Llama2-70b-chat-hf (Eng w. Shuffle) | 2.746 | 2.844 | 2.829 | 1.680 | 9.016 | 5.674 | 4.132 | 0.203
Llama2-70b-chat-hf (Chn) | 8.183 | 3.965 | 9.947 | 3.942 | 16.616 | 7.673 | 8.388 | 0.413
Qwen-14b-chat (Eng) | 7.376 | 6.512 | 6.596 | 5.405 | 4.026 | 6.388 | 6.051 | 0.298
Qwen-14b-chat (Eng w. Shuffle) | 5.965 | 9.709 | 5.916 | 3.485 | 16.145 | 5.451 | 7.778 | 0.383
Qwen-14b-chat (Chn) | 10.607 | 22.082 | 13.947 | 6.285 | 11.354 | 7.974 | 12.042 | 0.592
Qwen-72b-chat (Eng) | 3.947 | 4.767 | 3.660 | 1.952 | 13.470 | 6.036 | 5.638 | 0.277
Qwen-72b-chat (Eng w. Shuffle) | 4.250 | 4.854 | 3.767 | 3.409 | 6.386 | 4.267 | 4.489 | 0.221
Qwen-72b-chat (Chn) | 14.556 | 3.968 | 2.458 | 9.098 | 12.066 | 9.795 | 8.657 | 0.426
Mixtral-8x7B (Eng) | 7.078 | 0.591 | 0.583 | 7.785 | 7.947 | 10.799 | 5.797 | 0.285
Mixtral-8x7B (Eng w. Shuffle) | 2.983 | 1.650 | 5.904 | 1.693 | 3.251 | 3.465 | 3.158 | 0.155
Mixtral-8x7B (Chn) | 7.319 | 5.035 | 0.412 | 1.523 | 5.332 | 11.495 | 5.186 | 0.255
Human Results | 16.613 | 23.904 | 15.301 | 27.491 | 23.337 | 15.336 | 20.330 | 1.0
Table 6: The standard deviation for each VSM dimension is calculated across
nations. For the models, these deviations are derived from responses grouped
by simulated nations, while for human results, they are based on Hofstede’s
research findings. Distances and MCDs are calculated as outlined in 3.3. The
highest MCD among models is emphasized in bold, indicating that a larger MCD
suggests a greater influence of simulated nations on the models’ expression of
cultural values.

##  Appendix G Experiments Results for RQ1

###  G.1 Variant Context

  * •

Results based on the raw scores of 24 questions are listed in Table 5.

  * •

Results for intra-set comparison based on VSM scores are listed in Table 6.
The largest MCD among all experiment sets is less than 0.7, and only two out
of eighteen groups have scores greater than 0.5.

###  G.2 Shuffled Options

  * •

Results based on the raw scores of 24 questions for each pair of experiment
sets are listed in Table 7.

  * •

Results for inter-set comparison based on VSM scores for each pair of
experiment sets are listed in Table 1. As shown in the table, the smallest
Davies-Bouldin Index (DBI) value among all models exceeds 0.5, with values
closer to 0 indicating better clustering quality. Additionally, the highest
Silhouette Score (SS) is below 0.7, where values closer to 1 signify more
effective clustering. These statistics again underscore that the change in
context within prompts does not significantly alter the cultural values in
models’ responses.

From the perspective of $SS_{h}$, most models have values less than one, with
the exception of Qwen-14b-chat, whose $SS_{h}$ value exceeds one, indicating a
greater disparity than human results. This model also has the lowest Pearson
correlation coefficient between the two sets as shown in Table 7.

Models | PCC ($\rho$) | P-value
---|---|---
Llama2-7b-chat-hf | 0.894 | $\ll 0.05$
Llama2-13b-chat-hf | 0.861 | $\ll 0.05$
Llama2-70b-chat-hf | 0.938 | $\ll 0.05$
Qwen-14b-chat | 0.718 | $\ll 0.05$
Qwen-72b-chat | 0.922 | $\ll 0.05$
Mixtral-8x7B | 0.876 | $\ll 0.05$
Table 7:  The table presents Pearson correlation coefficients ($\rho$) and
p-values comparing centroids of models’ responses between “w. Shuffle" and
“w/o Shuffle" options (all prompts are in English), assessing the consistency
of responses from the aspect of the original scores.

##  Appendix H Experiment Results for RQ2

  * •

Results based on the raw scores of 24 questions are listed in Table 8.

  * •

Results for intra-set comparison based on VSM scores are listed in Table 2.
From the $DBI$ perspective, we observe that no value falls below 0.5,
consistent with our findings from comparisons between experiment sets split by
“shuffling", as presented in Table 1. The second measurement method,
Silhouette Score ($SS$), shows a similar trend, with the highest value among
the six comparisons remaining below 0.7. The average $DBI$ for comparisons
based on language differences is 0.17 lower than that based on "shuffling",
and the overall average $SS$ is 0.07 higher. Nevertheless, using standard
clustering metrics, we find no significant differences between the results in
Table 2 and Table 1.

However, the results of $SS_{h}$ for comparisons based on language differs
significantly from those for comparisons based on "shuffling": (i) No $SS_{h}$
values in Table 2 fall below 0.5, whereas half of the values in Table 1 are
below 0.5. This indicates that when asked the same questions in a different
language, we can expect the model to express cultural values with at least 50%
of the variability that a person from another country might exhibit. (ii) The
average $SS_{h}$ value for language comparison is 42.7% higher than that for
"shuffling". The observations suggest that language differences can more
readily "induce" the model to select a different option than selection bias.
Consequently, this results in a more distinctive separation in the expression
of cultural values by the same model. The t-SNE figures in Figure 1 also
illustrate the differences in intra-set disparity, clearly showing that most
models express cultural values more variably when queried in different
languages.

Models | PCC($\rho$) | P-value
---|---|---
Llama2-7b-chat-hf | 0.315 | 0.134
Llama2-13b-chat-hf | 0.704 | $\ll 0.05$
Llama2-70b-chat-hf | 0.841 | $\ll 0.05$
Qwen-14b-chat | 0.531 | 0.008
Qwen-72b-chat | 0.643 | $\ll 0.05$
Mixtral-8x7B | 0.535 | 0.007
Table 8:  The table presents Pearson correlation coefficients ($\rho$) and
p-values comparing centroids of models’ responses between “English" and
“Chinese" prompts, assessing the consistency of responses from the aspect of
the original scores.

##  Appendix I Experiment Results for RQ3

  * •

Heatmap (a) in Figure 2 shows that the 13b and 70b models from the Llama2
family are closest to the 14b and 72b models from the Qwen family. Similarly,
the Qwen-14b-chat model has the smallest $SS_{h}$ value with Llama2-13b-chat-
hf. Additionally, Qwen-72b-chat closely aligns with Qwen-14b-chat. When
examining the inter-set disparity for models when queried in Chinese, as
depicted in Heatmap (c), it is evident that all models from the Llama2 and
Qwen families show the smallest $SS_{h}$ values to the model outside their own
family.

  * •

In heatmap (d) of Figure 2, we present the $SS_{h}$ values between models when
questioned in different languages. Based on the visualized disparities among
models, it is clear that comparing models across languages results in
significantly larger differences in cultural values than comparisons within a
single language. The distribution of $SS_{h}$ values in the heatmap (d) of
Figure 2 is notably sparse, with 38.9% of values exceeding 1.0 and 10.5%
falling below 0.5. However, all values below 0.5 correspond to comparisons
between one model and others tested in a different language. This suggests
that the dimensional space utilized in the VSM testing might be too
constrained, causing overlap in results from various experiment sets. Despite
the overlap, the pronounced inter-set disparities observed in cross-language
comparisons suggest that variations in language can lead to more significant
differences in cultural values among models.

##  Appendix J VSM Questionnaire

Figure 5: VSM Questionnaire Page 1

Figure 6: VSM Questionnaire Page 2

Figure 7: VSM Questionnaire Page 3