LLM-CI: Assessing Contextual Integrity Norms in Language Models
Yan Shvartzshnaider Vasisht Duddu John Lacalamita















: Assessing Contextual Integrity Norms in Language Models
    
Yan Shvartzshnaider

York University


Vasisht Duddu

University of Waterloo


John Lacalamita

York University



    
=======================================================================================================================




§ ABSTRACT 


Large language models (LLMs), while memorizing parts of their training data scraped from the Internet, may also inadvertently encode societal preferences and norms. As these models are integrated into sociotechnical systems, it is crucial that the norms they encode align with societal expectations. 
These norms could vary across models, hyperparameters, optimization techniques, and datasets. This is especially challenging due to prompt sensitivity–small variations in prompts yield different responses, rendering existing assessment methodologies unreliable. 
There is a need for a comprehensive framework covering various models, optimization, and datasets, along with a reliable methodology to assess encoded norms. 

We present , the first open-sourced framework to assess privacy norms encoded in LLMs. uses a Contextual Integrity-based factorial vignette methodology to assess the encoded norms across different contexts and LLMs. We propose the multi-prompt assessment methodology to address prompt sensitivity by assessing the norms from only the prompts that yield consistent responses across multiple variants. Using and our proposed methodology, we comprehensively evaluate LLMs using  and  vignettes datasets from prior work, examining the impact of model properties (e.g., hyperparameters, capacity) and optimization strategies (e.g., alignment, quantization).






§ INTRODUCTION



Recent advancements in generative models, including large language models (LLMs), have led to significant performance improvements and their adoption in various sociotechnical systems, such as education <cit.> and healthcare <cit.>.
LLMs, which generate responses to input prompts, require vast amounts of data for training scraped from the Internet <cit.>.

However, the training of LLMs have several side effects. LLMs memorize parts of the training dataset, which may include personal or sensitive information <cit.>. 
Moreover, during training, LLMs could inadvertently encode societal preferences and norms that directly bias their responses. 

A misalignment between norms which are socially acceptable and those which are encoded by an LLM, could cause it to reveal information inappropriately in its responses, thereby violating privacy <cit.>. 
Several prior works have quantified these privacy violations from LLMs by identifying personally identifiable information <cit.>, and extracting potentially sensitive training data <cit.>.
However, the orthogonal problem of assessing encoded norms in LLMs has not been explored before.
Understanding the norms encoded in LLMs can help ensure they adhere to socially acceptable norms, prevent inappropriate information leakage, and mitigate social and ethical harms <cit.>. 

To address the novel problem of assessing encoded norms in the context of LLMs, we use the theory of contextual integrity (CI) <cit.>. CI defines privacy as the appropriate flow of information according to contextual norms. Prior work have used CI to evaluate societal expectations in various sociotechnical systems <cit.>, including the alignment of LLMs with human annotations <cit.>.

However, the assessment of encoded norms is not trivial. 
Firstly, we conjecture that norms vary across different model types, capacities, hyperparameters, and optimization strategies (e.g, alignment <cit.> and quantization <cit.>).
Secondly, LLMs are affected by prompt sensitivity, where minor changes in phrasing can alter responses <cit.>. This issue has not been addressed in prior work <cit.>, making them unsuitable for our evaluation.

To tackle the above challenges, we developed , a modular open-source framework for running various LLMs with different optimizations, hyperparameters, and datasets using CI-based vignettes as prompts. We also introduce the multi-prompt assessment framework, which addresses prompt sensitivity by evaluating norms based only on prompts that produce consistent responses across variants. This approach enables comprehensive and reliable assessment of encoded norms in LLMs.
We claim the following main contributions: we present


  * [Code will be available upon publication.], the first open-source framework which supports running various LLMs with different model properties, optimizations, and datasets. (Section <ref>)

  * a multi-prompt CI norm assessment methodology to address prompt sensitivity to reliable assess encoded norms in CI. (Section <ref>)

  * a comprehensive evaluation to assess encoded CI norms in 10 state-of-the-art LLMs and examine the impact of model properties and optimization strategies. (Section <ref>)






§ BACKGROUND AND RELATED WORK


We present a brief primer on LLMs (Section <ref>) and CI (Section <ref>), followed by describing related work at the intersection of CI and LLMs (Section <ref>) and evaluation of socio-technical properties in LLMs (Section <ref>).



 §.§ Large Language Models


Current state-of-the-art language models use transformers with billions of model parameters <cit.>. These language text generation models are trained to predict the next tokens in a sentence given previous tokens. 
The model learns the distribution Pr(x_1, x_2, …, x_n)  = Π_i=1^n Pr(x_i | x_1, …, x_i-1) where x_1, x_2, …, x_n is a sequence of tokens taken from a given vocabulary. 
A neural network, f_θ, with parameters θ, is used to estimate this probability distribution by outputting the likelihood of token x_i given by f_θ(x_i | x_1, …, x_i-1).

During training, a language model learns to maximize the probability of the data in a training set containing text documents (e.g., news articles or webpages).
Formally, the training involves minimizing the loss function ℒ(θ) = -logΠ_i=1^n f_θ(x_i | x_1, …, x_i-1) over each training example in the training dataset.
Once trained, a language model can generate new text conditioned on some prompt as prefix with tokens x_1, …, x_i) by iteratively sampling x̂_i+1∼ f_θ(x_i+1 | x_1, …, x_i) and then feeding x̂_i+1 back into the model to sample x̂_i+2∼ f_θ(x_i+2 | x_1, …, x̂_i+1).


Training LLMs is resource- and time-intensive, so pre-trained public models are fine-tuned for specific objectives before deployment. Popular fine-tuning optimization techniques include, alignment for matching with human annotations, and quantization reduce model capacity for efficiency.


Alignment. 
Training data scrapped from the Internet can include inappropriate content such as hate speech, stereotypes) <cit.> that LLMs might memorize and reproduce during inference.
To address this drawback, LLMs are aligned through fine-tuning on human-annotated datasets that correct inappropriate responses. 
As a result, the model replaces inappropriate responses with a standard responses like “I’m sorry, but I cannot...”. 
Popular alignment techniques include: instruction fine-tuning <cit.> to follow natural language instructions; 
reinforcement learning from human feedback (RLHF) <cit.> using a reward model based on human-annotated data to reward or penalize model responses; 
direct preference optimization (DPO) <cit.> which simplifies RLHF by using LLM’s output probabilities to align with human preferences without a reward model.


Quantization. LLMs demand powerful GPUs because of their high capacity. Quantization improves efficiency by reducing the precision of weights and forcing them to take fixed set of values. This allows models to run on smaller devices. Activation-aware quantization <cit.> is one such approach that analyzes activation distributions to retain important parameters and eliminate redundant ones while maintaining utility.




 §.§ Contextual Integrity


Contrary to predominant accounts of privacy that focus on aspects such as protecting sensitive information types <cit.>, enforcing access control <cit.> or mandating procedural policies and purposes <cit.>, the theory of CI defines privacy as an appropriate flow of information as governed by established societal norms <cit.>. According to CI, privacy is prima facie violated only when an information flow breaches an established contextual informational norm (aka CI norms or privacy norms), which reflect the values, purposes and function of a given context.  
A CI-based assessment of privacy implication of a system or a service involves two main phases: a) identifying the norm breaching flow using CI and b) examining the breach using the CI heuristic to determine how the novel flow contributes the values and purposes of the context. 


Identifying the norm breaching flow. The CI framework requires identifying five essential parameters to capture the information flow and the establishes norms in a given context including:
[label=(*)]

  * roles or capacities of senders, subjects, and recipients in the context they operate (like professors in an educational context and doctors in the health context);

  * the type of information they share;

  * transmitted principle to state the conditions, for purposes or constraints under which the information flow is conducted.

A canonical example below describes a typical interaction between a patient and a doctor. 
[label=ci_example, title=CI Example]
Patient (sender)  sharing patient's (subject) medical data (information type) with a doctor (recipient) for a medical check up (transmission principles) 

All the five parameter values matter. 
A change in any of the values results in a novel information flow. 
For instance, if instead of a doctor, a colleague is a recipient or instead of using the information for a medical check up, the information is made public, which could constitute a breach of an established social norm.  


Examining the breach. After we detect a violation, as part of the normative assessment, we use the CI heuristic to examine the ethical, financial, social and even political implications <cit.>. At the end of the process, we can either discard the novel information flow or modify the existing norm to better reflect the societal values and expectations. 

Several works have used the CI framework to gauge and evaluate privacy norms in different social context such as education <cit.>, IoT <cit.>, COVID-19 pandemic <cit.> and natural disasters <cit.>. They employed  a survey methodology using CI-based vignettes to gauge the appropriateness of potential information flows. These vignettes are of the form: 
[colframe=black,colback=white]
 <information flow with five parameters>.
How acceptable is the above information flow?
[strongly unacceptable, somewhat unacceptable, neutral, somewhat acceptable, strongly acceptable]






 §.§ Contextual Integrity and LLMs


A number of recent studies have applied CI to evaluate LLMs. 
<cit.> use CI and theory of mind to evaluate the alignment of LLMs with human annotated responses. They present, ConfAIde, a benchmark to use CI for LLMs with 98 prompts from <cit.>. 
Their study shows that LLM responses have low correlation with human annotations, with GPT-4 demonstrating better alignment compared to other models. 
In a follow up work, <cit.> have used ConfAIde to investigate the alignment of 16 mainstream LLMs with human annotations. 
They find that “most LLMs possess a certain level of privacy awareness” as the probability of LLMs refusing to answer private information increases significantly when they are instructed to follow privacy policies or maintain confidentiality.
Similar to results of <cit.>, they show that Pearson's correlation between human and LLM agreement varies widely and ChatGPT has the highest correlation among other models.

<cit.> evaluate the norms of LLMs when used as agents with a focus on privacy norms in LLM-mediated communication (i.e., LLMs being used to send emails).
They assess how well LLM responses align with crowd-sourced ground truth and measure privacy leakage from out-of-context information sharing.

<cit.> align LLMs with specific legal statutes to evaluate privacy violations and understand complex contexts for identifying real-world privacy risks. They generate synthetic cases and fine-tune their model to improve LLMs' ability to recognize privacy risks in actual court cases. However, their approach relies on limited number of expert-annotated norms and social contexts. To address these gaps, <cit.> develop a comprehensive checklist that includes social identities, private attributes, and existing privacy regulations. Using this checklist, they demonstrate that LLMs can fully cover HIPAA regulations.

<cit.> describe an attack that manipulates LLMs into revealing sensitive information by altering the context, such as fabricating an alien invasion to compel the model to disclose user details for “saving Earth.” Existing defenses like differential privacy and data sanitization fail because they do not account for context, and alignment is susceptible to jailbreaking. They use CI theory to mitigate information disclosures by proposing the use of two separate LLMs: one as a data minimization filter to identify appropriate information to disclose based on context, and the other that interacts with clients using the filtered data.





 §.§ Evaluating Sociotechnical Properties


Several benchmarks evaluate various LLMs sociotechnical properties such as toxicity, fairness, bias, sycophancy, privacy, robustness, and ethics <cit.>. 

On the other hand, LLMs have been shown to be sensitive to small variations in prompts which can drastically alter responses <cit.>. Previous studies comparing LLM decision-making to human behavior often overlook this sensitivity. <cit.> demonstrate that simple prompt adjustments can make LLMs exhibit more human-like behavior, questioning the reliability of current evaluation methods <cit.>.
There are limited studies consider prompt sensitivity: <cit.> propose generating synthetic prompts for better results. However, this is not suitable for assessing the encoded norms in LLMs as we require to query using CI-based vignettes and not synthetic prompts. Hence, a methodology for accounting for prompt sensitivity is largely an open problem.

There are a number of prior works that focus solely on assessing the leakage of sensitive data, including personally identifiable information to enhance LLMs privacy <cit.>. We can view them as assessment of a single CI parameter (data type), whereas a comprehensive CI approach requires all five parameters to make a privacy violation determination. 
Hence, these are orthogonal to our work. 




§ PROBLEM STATEMENT


We aim to reliably extract and evaluate the contextual information norms embedded in LLMs. In this section, we present the research questions, challenges, and limitations of using closely related work. 


Research Questions.
We pose the following questions: 
 

  *  How can we develop a comprehensive framework to assess the encoded norms in LLMs at scale?

  *  What methodology can reliably assess encoded norms?

  *  How do different factors influence the encoded norms?



Challenges.
To answer the above research questions, we have to address the following two challenges:
 

  * Lack of framework and datasets. Current literature lacks methods for evaluating models with varying capacities, hyperparameters, and optimizations. Furthermore, there are no large datasets with CI-based vignettes.

  * Prompt sensitivity. Current approaches for evaluating the responses obtained by simply prompting the model are not reliable due to prompt sensitivity <cit.>. 
Hence, we cannot adapt existing evaluation strategies to reliably assess the encoded norms. 

We need to address <ref> to answer <ref>, and develop a methodology to address <ref> to answer <ref> and <ref>.


Limitations of Prior Works.
The most closely related studies to our research question are by <cit.>, <cit.>, and <cit.>. As we discussed in Section <ref>, these works use CI-based vignettes to evaluate the alignment for LLMs. 
We, however, see several aspects that limit the applicability of their methodology to  our research questions:


  * Different objectives. Both of these prior works study the alignment of LLMs with human annotations obtained from a user study. In contrast, we assess the CI norms learned by LLMs, not their alignment with human responses.


  * Limited data. Their datasets only includes 98 vignettes <cit.> and 493 vignettes <cit.> in a few privacy-sensitive contexts. Moreover, the prompts only consider vignettes with three of the five CI parameters to show the alignment with human feedback. Therefore, it is unclear whether this approach can capture subtle differences in information flows with all parameters that might affect human judgments on acceptability.


  * Limited evaluation. 
The prompt templates for evaluation are different: to avoid anthropomorphizing LLMs, the prior works framed the prompts based on how people perceive information sensitivity rather than asking what model considers as “acceptable”.
While this works for evaluating alignment with humans, the approach does not assess the encoded norms in LLMs.
Finally, these works do not explicitly evaluate the impact of model properties such as  capacity, prompt variants, or optimization techniques like alignment and quantization. 



  * Methodological differences. These prior work do not address prompt sensitivity, which is a major issue in evaluating LLMs <cit.>. Therefore, we cannot use their methodology to reliably assess the encoded CI norms. Furthermore, based on observations from prior work <cit.>, carefully choosing the prompts could result in a false sense of alignment with ConfAIde's human annotations <cit.> or PrivacyLens' crowd-sourced ground truth <cit.>.


Overall, the framework evaluates norms in LLMs in a general and principled manner and 


can be extended to include tasks evaluated by prior work such as measuring privacy leakage <cit.> or human alignment <cit.>.



§ FRAMEWORK


We present , the the open-sourced CI norm assessment framework for LLMs to address <ref> and answer <ref>.


Design. Figure <ref> shows the modular design of , which  comprises the Vignette, Inference, Clean-up, and Analysis & Plotting modules. 







  * Vignette module includes vignettes datasets. 
Similar to prior work <cit.>, we use a script to create all possible vignettes from the combinations of the five CI parameters and a vignette template. 
The resulting vignettes are saved in a  file.
includes datasets covering the following contexts:
[label=*),itemjoin=,]

  * IoT devices <cit.>

  * COPPA regulations for IoT devices <cit.>

  * Internet privacy <cit.>

  * location data <cit.>

  * public records <cit.>

  * privacy as a social contract <cit.>.
 

For a chosen dataset, the module converts each vignette into a prompt before passing it to the LLM. 
Depending on the LLM, the module uses a corresponding prompt template (e.g., [INST] and [/INST] for Llama,  and  for tulu). The prompt template also appends additional text to asking for the acceptability of the described information flow described in the vignette. 


  * Inference module requires users to provide a model description, after which it loads the pre-trained weights (e.g., from Huggingface or OpenAI), executes the model, and offers an API for sending prompts and receiving responses. 
The model runs on an inference engine for efficient execution with minimal overhead. For Huggingface models, we use vLLM <cit.>, while the OpenAI models run on our custom implementation.

All the model descriptions include their capacity (e.g., 7B for seven billion parameters) along with the fine-tuning optimization used.
We identify three types of model optimization:
[label=*),leftmargin=*]

  * non-aligned models which have been trained on standard datasets but do not include any safety fine-tuning.

  * aligned models have been fine-tuned to account for human preferences using DPO. 

  * quantized models use AWQ to reduce the model capacity for better efficiency. 

These models are identified with “AWQ” or “dpo” in the description.
For models like llama-3.1-8B-Instruct and gpt-4o-mini, which use RLHF by default, we do not specify the optimization in the model description.




  * Clean-up module will filter the responses to get relevant text (e.g, Likert scale value) from verbose responses. For instance, an example of verbose text from models includes:
[colframe=black,colback=white]
Based on the scenario provided, the answer is: somewhat acceptable.  While it is understandable for a smart watch to collect and transmit data related to its owner's child's heart rate, it is important....

We manually reviewed all responses to ensure the cleaning process correctly extracted the appropriate Likert scale value. This was necessary as, in some cases, the model provided all Likert scale responses without specifying the applicable one.


  * Analysis & Plotting module generates relevant statistics (e.g., unanswered prompts and counts for each Likert scale) and plots them (e.g., heatmaps to illustrate the impact of varying CI parameters on responses). Additionally, the module can include statistical tests to measure result significance across different models, prompt variants, and optimizations to help infer the encoded norms in LLMs.  For a dataset with a normal distribution, we can perform the Analysis of Variance (ANOVA) <cit.> test or the t-test <cit.> to assess the statistical significance of the difference between two  or more groups of models. For non-normal data distributions, we can use  non-parametric statistical tests such the Friedman test <cit.> to measure the overall differences in models' responses and pairwise comparisons using the Wilcoxon Signed-Rank Test <cit.>.


Intended Use. A user need only provide the model description and select a context-specific dataset to assess the encoded norms. We envision to have the following users:


  * model developers to evaluate their models and compare norms across various optimizations before public release, and to design models which do not leak information in inappropriate contexts (e.g., <cit.>).

  * regulators and auditors to assess CI norms in various models amid growing calls for regulation, address privacy and information governance, and

  * researchers to better understand encoded norms and design training strategies that aligns encoded norms in LLMs with some socially accepted norms.

Overall, acts as a comprehensive benchmark to compare various models, optimizations, and datasets, which goes beyond prior work <cit.>.


Extending . The modular design of allows to easily integrate new datasets, prompt templates, models, and plots for analysis. To generate a new dataset, it is sufficient to specify the actors (senders, recipients, subjects), information types and transmission principles of a given context (see Section <ref> for the five essential CI parameters). 
Our (Python) script can generate new CI vignettes corresponding to the new parameters and them export in parsabale format such as a  file. 
also supports introducing different prompt templates to encapsulate the CI vignettes and adding new models, by specifying the model description, taken from the Huggingface library, in our configuration file for evaluation. 
Furthermore, the responses which are exported into a dataframe, can be used for additional analysis.

We design to be general, i.e., evaluate norms in LLMs regardless of their application. Prior works have considered specific cases where they evaluate the alignment with some ground truth human annotations using metrics such as privacy leakage <cit.> or correlation <cit.>.
can be extended by adding an additional module to evaluate these metrics given some reasonable ground truth.
However, to assess the norms encoded in LLMs, knowing ground truth is not necessary as we show in Section <ref> and Section <ref>.



§ METHODOLOGY


We present the experimental setup and hyperparameter choices in for evaluation (Section <ref>), followed by our multi-prompt CI assessment methodology (Section <ref>).



 §.§ : Setup and Configuration


Vignette Module. We focus on two datasets:  <cit.>, and  <cit.>.  contains 6912 vignettes to gauge appropriateness of information flows in context of IoT devices <cit.>.  contains 1800 vignettes to gauge the perceptions of information flows in context of IoT devices prescribed in Children's Online Privacy Protection Rule (COPPA) <cit.>. Both datasets have vignettes of the form:
[colframe=black,colback=white]
 Please indicate the extent to which this scenario is acceptable:  a sleep monitor records owner's location which is sent to the local police under the following condition: if owner has given consent.
The answer needs to be strictly only one of the following options (without ANY additional text):
<Likert Scale>

For each prompt using the above template, we generate ten different prompt variants using ChatGPT as part of multi-prompt assessment methodology (see Section <ref>).


Inference Module. We discuss the choice of models and hyperparameter used for prompting.


Model Descriptions.  Table <ref> summarizes the models used in our evaluation. We primarily consider pre-trained open-sourced tulu-2 family of models <cit.> to take advantage of the publicly available versions of the model with different capacities and optimizations (DPO and AWQ). Given a base model (e.g., tulu-2-7B or tulu-2-13B), we can  evaluate the impact of optimizations on the CI norm assessment. 






Temperature parameter. 
The responses from LLMs are sensitive to the temperature parameter (t) that controls the predictability of the results. A smaller t results in more predicable output: with t=0 produces the most consistent results across multiple runs and t=1 the most unpredictable. We confirm this empirically by comparing the responses for t=0, 1 on both datasets, which are shown in Figure <ref>.


   


Increasing t lead to change in the distribution of LLM responses. 
There was a noble increase in “No Answer” response and a shift along the Likert scale from “somewhat unacceptable” to “strongly unacceptable” or “neutral.”  
Overall, t=0 returned consistent outputs for all models with an exception of quantized models. 
To ensure consistent and deterministic responses, we choose the temperature parameter t=0 .


Cleanup Module. 
In our evaluation, the clean up module found the responses for 9.93% (88,268) of the prompts as invalid. 


The majority of invalid responses include the main categories: request for further context: “based on the information provided, it is difficult to determine the acceptability of the scenario without further context...”; limitation acknowledgment (mostly due to alignment): “as an ai language model, i cannot provide a personal opinion or additional text...”; and nonsensical response: mostly included character “s”, or used the wrong Likert scale in the response such as “smoothly acceptable” or “strictly acceptable.”





Analysis Module (Statistical Significance Tests). We run several statistical tests to to assess the statistical significance of the difference between two or more groups of models. For pairwise comparisons among models, we use the Wilcoxon Signed-Rank Test <cit.>, which is a non-parametric rank test assuming the responses are not normally distributed.




 §.§ Multi-Prompt CI Assessment Methodology


We first empirically illustrate prompt sensitivity and then describe our proposed novel assessment methodology.


Illustrating Prompt Sensitivity. 
We use ChatGPT to rephrase the original prompt template to generate ten different syntactic variants. For example, one prompt says: Please indicate the extent to which this scenario is acceptable: {scenario} and its variation is Please rate how acceptable this scenario is: {scenario}. 
For the full list of prompt variants, refer to Table <ref> in the Appendix.



Figure <ref> shows the variance between prompts for each vignette across all LLMs for both  and . 
All models, except gpt-4o-mini, exhibit variance in their prompt responses that is consistent between the two datasets with occasional outliers. 
Specifically, for , we observe a lower variance with the median of 0 to 0.5, and 25% of the prompts returned responses with variance of 0.5 to 1. 
The variance in responses follows a similar trend for the  compared to the .
We observed that models quantized and aligned models tend to have higher variances (e.g, tulu-2-7B-AWQ, tulu-2-dpo-7b and tulu-2-dpo-7B-AWQ) which can be attributed to lower quality responses as observed in prior work <cit.>.
Overall, the variance across prompt variants makes it harder to reliably assess the encoded norms.


Proposed Assessment Methodology To address the prompt sensitivity challenge (<ref>), we propose a methodology that  only evaluates norms from prompts with consistent responses across all the prompt  variants (addressing <ref>). 
We quantify consistency using either simple majority (≥50%) or super majority (≥67%) of responses for each prompt variant. A stricter majority threshold and greater diversity in prompt variants both increase confidence in assessing encoded norms.


[colframe=black,colback=white, title=Multi-prompt assessment methodology.,colbacktitle=gray]
 

  * Select K different variants of a given prompt to query the LLM. Ideally, the prompt variations should cover a wide set of distributions that is likely to be seen in practice.

  * Pass all the K+1 prompts to LLMs and track the responses.

    
  * If majority[This includes different types of majority: simple majority, super majority or any other forms] of the K+1 responses are consistent, use the corresponding information flow consistent response for further evaluation.
    
  * Else, hold that vignette and its responses for further evaluation.


  * Identify different CI parameters impact the responses, and the norms that an LLM might have learnt from its training data.



In our evaluation, we use simple majority of prompts with the same responses. 

We incorporate this methodology in and use it for all subsequent evaluations.




§ EVALUATION


We now discuss how to reliably valuate CI norms and examine the factors influencing these norms to address <ref>, by way of addressing the following two questions:



  *  How do we assess the encoded CI norms for LLMs? (Section <ref>)

  *  How do model type, capacity, alignment, and quantization impact our assessment? (Section <ref>)


Choice of Norms/Models Analyzed.
We use a limited subset of information flows and their heatmap for brevity, as space constraints prevent covering all norms. We randomly selected senders to demonstrate norm analysis and the impact of various factors.
However, can generate complete heatmaps for detailed analysis some of which are included in the Appendix.



 §.§ Assessing Encoded CI Norms in LLMs


 

We chose gpt-4o-mini and llama-3.1-8B-Instruct, which produced the most consistent responses in our evaluation, to illustrate a subset of encoded norms  (<ref>) on .

We omit the evaluation on  due to space limitation. Figure <ref> shows a sample output of 's plotting module–a heatmap of the extracted norms for a fitness tracker as a sender. 
For a full set of extracted norms, refer to Figure <ref> and <ref> in the Appendix. 
The empty (gray) squares in the heatmap represent information flows where could not deduce the corresponding encoded norm due to a lack of sufficient number of prompts with valid responses: ten or more prompts for llama-3.1-8B-Instruct or three prompts for gpt-4o-mini.



Overall, compared to gpt-4o-mini, llama-3.1-8B-Instruct is more conservative in its responses, with the majority of flows deemed “somewhat unacceptable.” A notable exception is the “if the owner has given consent” transmission principle. Under this transmission principle, the llama-3.1-8B-Instruct model viewed most information flows as “somewhat acceptable,” except when the fitness tracker shares information with “government intelligence agencies.” Furthermore, two specific information types—“audio [and video] of the owner”—stand out. The model deemed the information flow “strongly unacceptable” when the information “is stored indefinitely.” This norm stance seems to align with the original survey result in <cit.> that found that “fitness tracker sending recorded audio is considerably less acceptable than the same device sending exercise data.”
This is also reflected in gpt-4o-mini, which sees sharing audio and video information types for a large number of transmission principles as unacceptable.  While overall producing a more positive responses, gpt-4o-mini views sharing information for advertising, indefinite storage consistently as “somewhat unacceptable” or “strongly unacceptable.”







 §.§ Evaluating Influencing Factors


We now evaluate the influence of the following factors on the encoded CI norms  <ref>:
[label=*),itemjoin=,]

  * model type

  * model capacity (7B models vs. 13B models)

  * alignment (base models vs. DPO)

  * quantization (base models vs. AWQ).

To gauge a factor's influence, we use a multi-dimensional heatmap to show responses for each across four models in both datasets. We compare models with specific factor values, such as the  and  models to assess the impact of model capacity on norms. To ensure reliable norm extraction, we use a multi-prompt assessment that  considers the majority norm across all prompts for each model.


Model Type.
We conjecture that CI vignettes produce different responses for different model type due to differences in their training datasets and prompt sensitivity. We used to extract encoded norms in ten LLMs (see Table <ref> for a complete list) on both  and . Figure <ref> shows distribution of responses for each LLMs for a fixed prompt template. The distributions of LLM responses varies significantly with various biases. For example, the model tulu-2-7B-AWQ produced largely “strongly unacceptable” responses compared to llama-3.1-8B-Instruct where the responses were split between “somewhat unacceptable” and “somewhat acceptable.” 


Overall, we noted a significant variability in agreement on norms across various LLMs. 
For , tulu-* models provided the same response for only 241 information flows and only five information flows in . The tulu-* and llama-3.1-8B-Instruct models agreed on ten information flows in  and none in . The tulu-* and gpt-4o-mini-8B models agreed on the 207 information flows and only three in . The llama-3.1-8B-Instruct and gpt-4o-mini-8B seemed to be the most aligned, agreeing on a total of 2519 information flows in  and 1107 in .

As we discuss in Section <ref>, without knowing the exact datasets used to train the LLM models, we can only speculate about the differences for these apparent biases. 


   












Model Capacities. To evaluate the influence of model capacity, we compare the  models with the  models: tulu-2-7B ()  with tulu-2-13B (); and tulu-2-dpo-7B () with tulu-2-dpo-13B (). Heatmap in Figure <ref> shows the embedded norms for all four models related to information flows with the senders “a fitness tracker”  () and “smart watch” ().
The squares with same color for all four triangles represents consistent responses across all four models. 
Conversely, the different triangle colors reflect the inconsistencies in the models' responses.

We first focus on the  responses with the same (or similar) color shades for all triangles to understand the norms that are consistent across different model capacities. 

For example, in , perhaps reflective of the training dataset, all four models ranked the majority of flows involving  “a fitness tracker”  sharing information with “the government intelligence agencies" (second column in each section of the corresponding data type) as “somewhat unacceptable” and “strongly unacceptable.” For , all four models viewed information flows involving the transmission principle of “[serving] contextual ads” as “somewhat unacceptable” or “strongly unacceptable.” 
This observation aligns with the prior work in <cit.> that found: “Information flows with the transmission principle “if the information is used to serve contextual ads” have negative average acceptability scores across almost all senders, recipients, and attributes.” 
Nevertheless, in contrast to the reported result in <cit.>, the models also viewed information flows even if information “is deleted” as “somewhat unacceptable” or “strongly unacceptable.”

 
For several flows, however, in both  and , depending on capacity, the models embed different norms. 
In , while the  models: tulu-2-7B () and tulu-2-dpo-7B ), consider information flows involving “a fitness tracker” sharing “owner's heart rate” with “other devices at home” or “owners immediate family” in “an emergency situation” or “to perform maintenance” as “strongly unacceptable,”  models: tulu-2-13B () and tulu-2-dpo-13B (), deem them “strongly acceptable” (darkred!90darkred!90darkgreen!30darkgreen!30). 

In , we observe a similar pattern for “a smart watch” sharing “the owner's child's birthday” with “a third-party service provider” or “[device's] manufacturer” when the “[recipient] implements reasonable procedures to protect the information collected.” The original survey <cit.> suggests similar age-related differences in information flow perceptions. Participants aged 45-65 found certain transmission principles more acceptable than younger groups, especially those familiar with COPPA and owning smart devices.  Difference in age and familiarity might be reflected in the training dataset. This, in turn, could've  skewed the model's encoded norms.


Model Alignment.
To evaluate the influence of alignment, we compare the responses from the base models: tulu-2-7B () and tulu-2-13B (); with the corresponding aligned models: tulu-2-dpo-7B () and tulu-2-dpo-13B (). 

Figure <ref> identifies several illustrative differences.  In , there are intra-agreement and inter-disagreement between base models and within aligned models  regarding information flows involving “a fitness tracker” sharing “ the owner's heart rate”  or  “owner's location" in “an emergency situation” with all receipts. 
The base models view these flows are “strongly unacceptable,” whereas the aligned model deemed it as “somewhat acceptable” (darkred!90darkred!90darkgreen!30darkgreen!30). 
In , a similar pattern appears with regard to information flows involve “a smart watch” sharing “audio [or video]  of it's owner child” with “a third party service provider” or “[device] manufacturer” for all the stated transmission principles (darkred!90darkred!90darkgreen!30darkgreen!30).

We can also observe a full agreement between the base and aligned models for some information flows in  but not in . 
In , as discussed in the previous section, all models viewed “a fitness tracker” sharing information flows with the “government intelligence agencies” as “strongly unacceptable.” 
This is perhaps indicative of a strong sentiment at the time the training datasets was scrapped. The original work <cit.> corroborates this assumption: “we included the local police and government intelligence agencies in consideration of recent court cases involving data obtained from IoT and mobile devices”.


Significance of Results.
Table <ref> shows a pairwise Wilcoxon Signed-Rank tests that yielded  p-values < 0.05 for comparisons of four model outputs thus we can reject the null hypothesis of no difference between the model responses, indicating that there are statistically significant differences. 
We can, thus, conclude that model alignment and capacity significantly impacts the encoded norms.




Quantization. To evaluate the impact of quantization, we compare the base models: tulu-2-7B () and tulu-2-13B (); with the quantized AWQ models: tulu-2-7B-AWQ () and tulu-2-13B-AWQ ().  Figure <ref> shows the heatmap for the four models.


Varying information types and transmission principles can elicit different results depending on whether the model is quantized. For the information flows involving “a fitness tracker” sharing  “owners heart rate” with the [device] manufacturer “if privacy policy permits it” or “if the owner is notified” base models and quantized models produced different results (darkred!90yellow!90darkgreen!60darkred!30). 
Similarly for , information flow when sharing “child's birthday” with the “third party service provider” or "device manufacturer” if “privacy policy permits it.”

Furthermore, there are cases where the base models within a specific capacity agree with their quantized counterparts, while disagreeing with models with other capacity. 
For example, in , the  (base and quantized) models view information flow involving “a smart watch” sharing “child's emergency contact” with “[device] manufacturer” if ”the owner has given a verifiable consent before the information is collected” as “somewhat acceptable” (darkred!90darkgreen!90darkgreen!30darkgreen!30).  
However,  models disagree: the base  model

deems the flow as “strongly acceptable,” while the quantized   model shows it as “strongly unacceptable.”

We also note instances where the base models and quantized models with the equal  capacity encode the same norm, for example,  in , “a fitness tracker” sharing “the time the owner is home” in the “emergency situation” (darkred!90darkred!90darkgreen!30darkgreen!30). The tulu-2-7B () and tulu-2-7B-AWQ () models deemed this information flow as “strongly unacceptable,” while tulu-2-13B () and  tulu-2-13B-AWQ () view it as “somewhat acceptable.” 
We see a similar agreement for  when sharing the“audio [and video] of the owner's child.” 


Significance of Results.
Table <ref> shows a pairwise Wilcoxon Signed-Rank tests of four model outputs that yielded p-values < 0.05) for all comparisons, rejecting the null hypothesis of no difference between the model responses. 
We conclude that quantization has statistically significantly impact on the encoded norms.



Alignment & Quantization (A&Q). To evaluate the impact of A&Q, we compare the responses of the A&Q models and those of the base models. 

Figure <ref> depicts the responses of tulu-2-7B (), tulu-2-13B (), tulu-2-dpo-7B-AWQ (), and tulu-2-dpo-13B-AWQ ().  

We can observe  instances of “agreement” or “disagreement” on norms between  and  models with both alignment and quantization (A&Q).
For example, in , there is an overall agreement “government intelligence agencies” as “somewhat unacceptable” or “strongly unacceptable,” with the exception of sharing “the times owner is home” “in an emergency situation.” 

For several information flows, there is a disagreement within the base models and within the A&Q models, for example, in , when it comes to “a fitness tracker”  sharing “owner's eating habits,” if the “[data] is anonymous” there is no inner agreement for all recipients to various degrees, except for “government intelligence agencies.” Similarly in , both  and  with A&Q models agree that sharing the “owner's child's call history” with “the third party provider” is mostly “somewhat unacceptable,” or “strongly  unacceptable,”  with the exception of  “a smart watch” sharing this information “to protect child's safety.” In this case,  the base models present a polar opposite view, whereas the A&Q models disagree on the level of acceptability (darkred!90darkred!90darkgreen!30darkred!30).


Significance of Results.
Table <ref> shows a pairwise Wilcoxon Signed-Rank tests of the four model outputs that  yielded p-values < 0.05) for all comparisons. With this result, we can reject the null hypothesis of no difference between model responses, indicating that there are statistically significant differences. 
We conclude that models with A&Q significantly impact the encoded norms.










§ DISCUSSION AND CONCLUSIONS


Our work builds on prior efforts aimed at the challenging task of evaluating the sociotechnical properties of LLM models. This task requires a deep understanding of both societal factors and the inner workings of the models. We discuss the limitations of our work and suggest future directions.










Encoded norms provenance.
While identifies encoded norms in LLMs, it does not trace their origin. The training datasets significantly impact a model's “view of the world,” and without dataset transparency, LLMs remain black boxes, making it difficult to understand their responses. Therefore, examining the dataset source is crucial to ensure LLMs are trained on valid and socially acceptable norms.
For instance, 82% of the training data for falcon-180B includes a massive scrape of the Web, 6% books, 5% conversations (e.g., from Reddit, StackOverflow, HackerNews), 5% code, and 2% from technical sources like arXiv, PubMed, and USPTO[<https://huggingface.co/tiiuae/falcon-180B#training-data>].
Making the contents available could help shed light on the biases in the models'  responses.

 
CI privacy (ground truth) norms alignment. Having reliably identified encoded norms in LLMs, we can use to detect deviations from socially acceptable ground truths, which may be based on regulations, laws, or survey studies <cit.>.  
Our CI-inspired approach can evaluate norms in LLMs across various settings and can be extended to compare with crowd-sourced ground truth on information flow acceptability, similar to <cit.>, to measure privacy leakage or correlation with human annotations as in <cit.> and <cit.>.
To ensure that the LLM norms align with the ground truths, we would need to fine-tune the models using alignment objectives (see Section <ref>). We consider this an area for future work.


Summary.
This paper introduces , the first open-source framework based on CI for evaluating contextual norms in LLMs. We propose a multi-prompt assessment methodology to extract encoded norms while addressing prompt sensitivity. Using , we evaluate norms in 10 LLMs, considering factors like model capacity, alignment, and quantization, and discuss their impact. Our work aims to provide a reliable evaluation guideline for future research.




§ ETHICS STATEMENT

The use of LLM has societal implications. To evaluate , we use latest LLMs that require a large amount of energy and resources to maintain. While our work has relatively little environmental impact because we are not training or fine-tuning the models, we acknowledge that through the use of tools like OpenAI in our research we contribute to the overall negative effect of these systems on the environment. 
Our work carries also a social implication: by using CI, brings additional layer of normative rigor in evaluating LLM-based system in understanding how they contribute to the purpose, values and functions in the contexts they operate.
Furthermore, LLMs can produce auxiliary information in the responses that could contain inappropriate information. This is also the reason we designed the clean-up module to filter the irrelevant information.  



§ OPEN SCIENCE

Datasets, scripts, binaries, and source code to reproduce our results will be publicly available upon publication to align with USENIX Security's open science policy. 
Additionally, we will submit artifacts for evaluation to ensure its availability, functionality, and reproducibility. 





§ ACKNOWLEDGEMENTS

We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), RGPIN-2022-04595, and  thank the OpenAI API Researcher Access Program for the credits to evaluate GPT-4 model.  Vasisht is supported by David R. Cheriton Scholarship, and Cybersecurity and Privacy Excellence Graduate Scholarship.



plainnat








§ APPENDIX











   








