# Towards Regulatable AI Systems:
Technical Gaps and Policy Opportunities

Xudong Shen
National University of Singapore
<EMAIL_ADDRESS>
Hannah Brown
National University of Singapore
<EMAIL_ADDRESS>
Jiashu Tao
National University of Singapore
<EMAIL_ADDRESS>
Martin Strobel
National University of Singapore
<EMAIL_ADDRESS>
Yao Tong
National University of Singapore
<EMAIL_ADDRESS>
Akshay Narayan
National University of Singapore
<EMAIL_ADDRESS>
Harold Soh
National University of Singapore
<EMAIL_ADDRESS>
Finale Doshi-Velez
Harvard University
<EMAIL_ADDRESS>

###### Abstract

There is increasing attention being given to how to regulate AI systems. As
governing bodies grapple with what values to encapsulate into regulation, we
consider the technical half of the question: To what extent can AI experts vet
an AI system for adherence to regulatory requirements? We investigate this
question through two public sector procurement checklists, identifying what we
can do now, what we should be able to do with technical innovation in AI, and
what requirements necessitate a more interdisciplinary approach.

##  1 Introduction

As AI systems become more advanced and integrated into our lives, there has
been a corresponding urgency to ensure they align with social values and
norms. Legal and regulatory authorities around the world are racing to produce
AI regulations [142, 44, 36]. However, the increasing size, generality,
opaqueness, and closed nature of present-day AI systems pose significant
challenges in achieving this alignment [150, 8]. Even when requirements can be
precisely articulated—already a difficult task—there is still the question of
whether and how it is possible to check whether an AI system adheres to those
requirements.

We consider the following questions: What innovations in AI systems are needed
for them to be effectively regulated? And in what areas will innovations in AI
methods alone be insufficient, and more interdisciplinary approaches required?

While there are many regulations involving AI, we consider public sector
procurement checklists because they are instrumental in shaping societal
norms, influence private actors to emulate similar practices, and are
relatively explicit. These checklists define the criteria and process for a
public sector agency to purchase products or services, including technical
criteria that an AI system must satisfy. The technical criteria about AI
systems contained in these lists are also present in other regulatory efforts.
Thus, improving our ability to vet an AI system against these checklists will
improve our ability to regulate AI systems more generally.

Specifically, we closely examine the technical criteria from two existing
procurement checklists: the World Economic Forum’s AI Procurement in a Box
(WEF) [144] and the Canadian Directive in Automated Decision-Making (CDADM)
[56]. The WEF checklist serves as a practical guidebook to unlock the public-
sector adoption of AI and has been piloted in various countries, including the
UK, Bahrain, the UAE, India, and Brazil [19]. The CDADM is one of the earliest
regulations specifically targeting AI systems to be put into operation. It
came into effect in April 2019, and requires full compliance by October 2023
for all new systems and by April 2024 for all existing systems. Thus, the
question of whether it is possible to build AI systems that meet such
checklist criteria is timely.

In the remainder of this document, we first group the technical criteria
contained in these two checklists into categories that will be familiar to AI
researchers and engineers: (pre-training) data checks, (post-hoc) system
monitoring, global explanation, local explanation, objective design, privacy,
and human + AI systems. For each category, we briefly summarize existing
technical approaches that could be used to construct AI systems that meet
those criteria. Next, we identify areas where relevant technical approaches
may exist, but additional technical innovation is needed to be able to vet
increasingly complex AI systems being used in increasingly varied contexts.
For example, the proliferation of large language models comes with a
significant difficulty in evaluating them, due to issues including open-
endedness and data leakage. While innovative approaches like Holistic
Evaluation of Language Models (HELM) [79] and Elo ratings [149] are proposed,
the evaluation of language models remains an open question and further
technical innovation is needed for effective regulation and oversight.
Finally, we briefly outline aspects of these criteria that may seem technical
but actually require interdisciplinary approaches to vet.

Throughout this exercise, we assume no concerns about expertise; that is, we
assume that there are sufficiently qualified AI and domain experts to review
whether the AI system meets the checklist criteria. Our concern is to identify
to what extent experts can currently vet AI systems against these regulatory
criteria. In doing so, we hope to highlight concrete areas where AI innovation
would improve our ability to create regulatable AI systems. There are many
subfields in AI. Moreover, AI systems are rapidly advancing, and the kinds of
contexts in which they are being used are rapidly growing. Thus, while our
list is certainly not comprehensive, we hope it serves as a starting point for
AI researchers interested in creating regulatable AI systems. Additionally,
this document informs both policy makers and AI engineers on issues where more
holistic, interdisciplinary efforts (rather than AI methods alone) are
necessary.

##  2 Inputs of the Model: (Pre-training) Data Checks

The characteristics of the training data have a large influence on the
behavior of an AI system. What checks must be done on these data before they
are used to train models? Motivations for the regulatory requirements in this
section include data consent, data privacy (discussed in more detail in
Section 7), and downstream impacts of data quality (e.g. on model performance,
generalization and bias). Examples of checklist criteria include:

  * CDADM 6.3.1: Before launching into production, developing processes so that the data and information used by the Automated Decision Systems are tested for unintended data biases and other factors that may unfairly impact the outcomes.

  * CDADM: 6.3.3: Validating that the data collected for, and used by, the Automated Decision System is relevant, accurate, up-to-date, and in accordance with the Policy on Service and Digital and the Privacy Act.

  * CDADM 6.3.4: Establishing measures to ensure that data used and generated by the automated decision system are traceable [fingerprinting], protected and accessed appropriately, and lawfully collected, used, retained, and disposed.

  * WEF: Assess whether relevant data will be available for the project […] Data is crucial for modern-day AI tools. You should determine, at a high level, data availability before starting your procurement process. This entails developing an understanding of what data might be required for the project.

  * WEF: Select data that fits criteria of fairness. For example, the data should be representative of the population that the AI solution will address, as well as being reasonably recent.

The technical questions underlying these criteria have to do with data
documentation procedures and checks that can expose potential risks in areas
such as fairness, generalization, and privacy.

###  2.1 What we know how to do

We have proxies for checking many properties in these criteria (data privacy,
label quality, feature selection, fairness, etc.) using exploratory data
analysis [135]. For example, we can inspect the annotation process and check
inter-annotator agreement to get an idea of label quality [12, 99]. We can
also measure (and correct for) imbalance in data if we are given group labels
that segment the dataset [76, 23]. We have techniques for identifying
influential points [13], outliers [14], and mislabelled points [20, 102] which
may cause models to exhibit poor performance or bias [46].

Further, there exist several standards for reporting dataset information [51,
64, 15, 86, 103], including on the data curation process, that are designed to
help expose potential biases and limitations for which the data may be used.
Sufficiently comprehensive data documentation facilitates investigation by
both experts and the public. In the realm of consent, non-consenting (opt-out)
data checks111For example, artists can opt-out their work with Spawning
(https://spawning.ai), which provides opt-out data checks as a service to AI
system developers. can give individuals control over how their data are used.

###  2.2 Directions requiring additional AI innovation

  * •

Metrics and Generalizability. More work is needed to connect the data metrics
with impact on outcomes. For example, we have reasonable tools to connect the
uncertainty or measurement error in a distance sensor to effects on motion
planning [43]. However, if a traffic image dataset has a certain annotator
disagreement score, what does that imply for an autonomous vehicle whose
vision system is trained on those data? The question of generalizability also
arises for data without explicit human annotation, such as internet-crawled
language and vision datasets [104, 117]. In this case, what data checks can we
perform to ensure that it will be appropriate for the domains in which the
model is deployed? Data checks might lose their validity if the data is used
outside its envisioned context.

A particularly important category are metrics that capture similarities of
different applications and thus capture scenarios in which a data set
collected for one purpose may be used for another. While this inquiry has
received considerable attention in domain adaptation research [35, 2], the
data-centric perspective remains relatively unexplored [3]. For example, a
dataset collected for autonomous vehicles in one city _might_ be suitable in
similar cities. But what statistics or meta-data would we need to be
confident? To ensure reliable utilization of datasets, additional metrics are
necessary to precisely determine the range of applications for which a dataset
can be safely used.

  * •

Data Quality Checks in the Context of Pretrained Models. Given the prevalence
of large pre-trained models [60] and (currently) limited transparency about
their training data [26], can we develop data checks that rely solely on
accessing the model [88], or do certain types of checks require disclosure of
specific information about the training data? Do checks for fine-tuning
data—e.g., the traffic images used to tune an autonomous vehicle’s vision
system on top of an existing image classifier—differ from checks for pre-
training data?

  * •

Unstructured Data. For structured data, it is relatively easy to report
statistics across features. For unstructured data like images or social media
messages, existing standards focus on reporting the statistics of the meta-
data [51, 64, 15, 86, 103]. However, is providing transparency about the meta-
data sufficient? For example, in the above scenario with the traffic images,
is it sufficient to provide information e.g. about where the images were
collected and what kinds of cameras were used? Or might it be important to
report certain information derived from the pixel values as well? Similarly,
if one had a collection of social media posts, would be it important to report
certain information derived from the actual content, in addition to meta-data
about the site and scraping procedure?

###  2.3 Areas that require interdisciplinary engagement

The specific metrics that would enable meaningful inference about the quality
of the data will depend on the application. Questions around bias and fairness
are also inherently multi-faceted and will depend on the use-case. Determining
the proper form of consent (opt-in vs. opt-out) is a legal decision.
Involvement from social scientists is necessary to assess how different data
collection processes include or exclude certain populations. Privacy
tensions—what data is retained, what statistics are made public, what kind of
access is granted to trusted auditors—must also be resolved within the broader
socio-technical context.

Furthermore, there is danger in living exclusively inside the data; cross-
talks inside and outside of the data are necessary to detect many normative
pitfalls. For example, bias can be introduced via the choices of labels (e.g.
are non-binary labels included when labeling gender?) and the labeling process
(e.g. whose perspective was being taken when an input was labeled as
acceptable or problematic content?). Healthcare algorithms that demonstrate
unbiased predictions of healthcare costs, but then use that prediction as a
proxy for illness severity, may introduce bias because unequal access to care
leads to lower healthcare spending by minority groups [94]. Detecting and
addressing such issues in data necessitates active dialogue between the data
realm and external perspectives. Section 6 delves deeper into the discussion
of label choice concerns.

##  3 Outputs of the Model: (Post-hoc) System Monitoring

Once a system is deployed, it is essential to monitor its operations. These
criteria have to do with monitoring for adverse outcomes and identifying
unintended consequences, making that information available for scrutiny, and
establishing contingencies if the system is behaving poorly. Metrics to
monitor the operations of a system also relate to methods for checking an AI’s
system’s performance after it has been trained. Examples of checklist criteria
include:

  * CDADM 6.3.2: Developing processes to monitor the outcomes of Automated Decision Systems to safeguard against unintentional outcomes and to verify compliance with institutional and program legislation, as well as this Directive, on a scheduled basis.

  * CDADM 6.3.6: Establishing contingency systems and/or processes as per Appendix C. (Which says: Ensure that contingency plans and/or backup systems are available should the Automated Decision System be unavailable.)

  * CDADM 6.5.1: Publishing information on the effectiveness and efficiency of the Automated Decision Systems in meeting program objectives on a website or service designated by the Treasury Board of Canada.

  * WEF: [T]here should be systematic and continuous risk monitoring during every stage of the AI solution’s life cycle, from design to post-implementation maintenance.

  * WEF: Testing the model on an ongoing basis is necessary to maintain its accuracy. An inaccurate model can result in erroneous decisions and affect users of public services.

  * WEF: Enable end-to-end auditability with a process log that gathers the data across the modelling, training, testing, verifying and implementation phases of the project life cycle. Such a log will allow for the variable accessibility and presentation of information with different users in mind to achieve interpretable and justifiable AI.

The technical questions associated with these criteria have to do with how to
monitor performance and identify various kinds of drift and unusual results
that warrant attention.

###  3.1 What we know how to do

Given a specific metric, it is relatively easy to put monitoring into place.
We can easily check to ensure that the outputs of an AI do not exceed
threshold values. Methods exist that establish distributions for ”normal
operation” and flag anomalous values during actual operation [49]. These
techniques can be employed to detect shifts in inputs and outputs, in model
confidences and calibrations [10], in derived quantities such as the top
features used to make a prediction (allowing a person to check if a shift is
sensible) and fairness metrics [55, 10, 119]. We can learn a trend in how a
particular quantity changes and see if that trend holds and whether any
external shock occurs. In RL settings, we can monitor differences between
expected and actual reward distributions. If the causal structure of the
environment is known, monitoring checks can specifically identify new
confounders and mediators. That said, all anomaly detection methods require
some specification of what kinds of behavior represent a change or anomaly.
They may not capture every unintended consequence, and given sets of
monitoring metrics may be gamed by an adversary.

More generally, we already have a set of norms around what kinds of tests
should be run prior to an AI system being deployed (e.g. [74, 124]). AI
developers should strive to test their systems with multiple independent,
external datasets to ensure that their results are replicable (and be
transparent if this kind of generalization has not been tested). These
datasets should include sufficient numbers of hard cases in their test sets,
and results should be presented stratified by difficulty. Similarly, one
should provide stratified results on performance of cases similar and
dissimilar to the training set. Performance measures should be reported with
respect to the real population proportions of each class, stratified by class,
or be independent of base rates so that they can be correctly applied to the
intended use-case and not the proportions present in the training set.

###  3.2 Directions requiring additional AI innovation

  * •

Monitoring Many Metrics. Monitoring multiple metrics increases the risk of
false positives and false negatives, which can overwhelm engineers. How can we
monitor many metrics efficiently while not incorrectly flagging too many cases
for review and not missing important deviations? Relatedly, once in operation,
what data should be gathered so that we can check additional metrics in the
future? For example, while we can monitor fairness for known minority groups,
what data should be logged during operation so that we can audit fairness when
an unknown demographic group (e.g., an intersection of some legally protected
attributes) contest for unfair outcomes [71]? The question of what logs to
retain only becomes more difficult when there are multiple AI systems
interacting at fast rates, such as the many AI components operating within an
autonomous vehicle. These questions remain despite advances in MLOps [74].

  * •

Certification of Use Cases. Across the very broad range of AI systems and
contexts, can we certify the settings in which an AI system is supposed to
work well? Can we assign a label to an AI model so that it is restricted to or
from being applied to specific use cases? Consider, for example, the need to
establish safeguards that prevent an open-access drug discovery model from
being utilized for de novo design of biochemical weapons [136]. Similarly,
image generative models should be restricted from generating pornographic
content. Relatedly, can we provide confidence about the post-hoc performance
of a deployed system on certified tasks while preventing a deployed system
from being misused?

In formal verification, one mathematically checks that the formal model of a
given system satisfies a desired property. Formal verification is widely in
safety-critical systems. As AI systems enter safety-critical settings—such as
autonomous driving or robot-assisted surgeries—it is essential that strong
safety guarantees can be maintained. Certifying neural networks for safety-
critical systems is an active research area [126, 122, 9, 70, 52, 146]. There
are also early proposals to define standards for levels of AI system
certification [148] (analogous to security standards [47]) that have yet to be
refined and adopted.

  * •

Correcting Models after Deployment. There exists some work on correcting
deployed models in a way that does not require re-training end-to-end (e.g.
unlearning [58, 131, 75, 32], fine-tuning [65], and in-context learning [141,
30]). But more work remains to be done, especially for AI systems with many
interacting parts.

  * •

Identifying Relevant Distribution Shift. There are many possible types of
shift: in input distributions, in the relationship between inputs and outputs,
in the rewards (objective)—and these shifts can take many forms and occur in
many ways. For example, the acceleration of newer cars may be different, as
well as what colors are popular. Can we distinguish between relevant and
irrelevant shifts (e.g., along the lines of [33])? If the shifts happen in
some uninterpretable embedding space, how can we explain them?

  * •

Monitoring Agents that are Learning Online. We can monitor for major adverse
effects. However, can we identify more subtle issues, such as initial signs of
catastrophic forgetting, cheating, and other harms that occur while the agent
continues to perform well on its reward metric? For instance, it would be
advantageous to detect early signs of reckless or inappropriate driving
behavior—such as reducing distances between the vehicle and pedestrians, or
increased use of residential streets where children may be playing—in
autonomous driving agents before any traffic accidents occur. Our
understanding of unintended consequences continues to grow [129, 24] but the
problem remains unsolved.

###  3.3 Areas that require interdisciplinary engagement

At a high level, there will always need to be some kind of decision made about
what needs to be monitored or prioritized in a given setting. There will also
need to be decisions made about what kinds of safety promises or guarantees
are needed e.g. how much shift is considered safe and acceptable, and how much
is not. It is crucial to translate the monitored metrics into meaningful
implications that enable people to make informed decisions within the broader
socio-technical system. For instance, in autonomous driving, comparing
monitored metrics against human performance can inform decisions regarding
human intervention. Finally, the task of contingency planning for back-ups
when models express unexpected or unwanted behaviors also requires an
understanding of the broader socio-technical system.

##  4 Inspecting the Model: Global Explanations for Model Validation

Global explanations describe a model as a whole and are often useful for
inspection or oversight. The goal is to expose information about the model
that would allow a domain expert to infer the existence of some kind of
unobserved confounder, something about the model that is non-causal, and other
limits on the scope of the model’s applicability. Criteria related to global
explanations include:

  * CDADM App. C: Plain language notice through all service delivery channels in use (Internet, in person, mail or telephone). In addition, publish documentation on relevant websites about the automated decision system, in plain language, describing: How the components work;

  * WEF: Public institutions cannot rely on black-box algorithms to justify decisions that affect individual and collective citizens’ rights, especially with the increased understanding about algorithmic bias and its discriminatory effects on access to public resources. There will be different considerations depending on the use case and application of AI that you are aiming to acquire, and you should plan to work with the supplier to explain the application for external scrutiny, ensuring your approach can be held to account. These considerations should link to the risk and impact assessment described in Guideline 2. Under certain scenarios, you could consider making it a requirement for providers to allow independent audit(s) of their solutions. This can help prevent or mitigate unintended outcomes.

  * WEF: Ensure that AI decision-making is as transparent as possible. – Encourage transparency of AI decision-making (i.e. the decisions and/or insights generated by AI). One way to do this is to encourage the use of explainable AI. You can also make it a requirement for the bidder to provide the required training and knowledge transfer to your team, even making your team part of the AI-implementation journey. Finally, you can ask for documentation that provides information about the algorithm (e.g. data used for training, whether the model is based on supervised, unsupervised or reinforcement learning, or any known biases).

Technical approaches associated with these criteria include the creation of
small, inherently interpretable models with high performance, sharing certain
parts or properties of a large model, and open-sourcing the model’s code.

###  4.1 What we know how to do

We can build inherently interpretable models (e.g. generalized additive
models, decision trees, rule-based models, etc.) for tabular and other simple,
relatively structured data [111]. We have some tools for interpreting neural
networks in terms of human-understandable components [106, 93, 95], such as
circuits [139] or even natural language [17]. When possible, these tools
provide a systematic approach to explain how tasks are performed in ML models
in a human understandable way. Finally, we can partially explain neural
networks and other complex models via methods such as distillation [130],
feature importance [80], or computing concept activation vectors [114].

###  4.2 Directions requiring additional AI innovation

  * •

Inherently Interpretable Models for More Data Types. While some initial work
exists for building inherently interpretable models for non-tabular data (e.g.
for images or audio) [31], this area is still nascent. Concept learning [73]
on top of the input may be a useful strategy.

  * •

Interactive “Openboxing” of Large Models. Can we build interactive,
hierarchical, and semantically-aligned views of large models such that these
models are (to some extent) inherently interpretable? For example, a traffic
image classifier that recognizes objects by multiplying object templates with
transformation matrices [147] would be more inherently explainable than
another model without this hierarchical structure. Further, can we allow users
to explore such explanations at different levels of fidelity for different
contexts? As noted above, methods to extract information from larger models
such as large language models exist (e.g., [114, 90]) but have limitations
with ways for people to effectively explore and understand larger models. More
work along the lines of [11, 128] is needed.

  * •

Checking Value Alignment. Whether it is criminal justice, benefits
allocations, or autonomous driving, AI systems are increasingly used in
situations that require value judgments. How do we elicit and encode societal
and individual values in diverse situations? What metrics can effectively
measure value alignment? How do we make this mapping transparent for others to
understand the value choices made (e.g., the drivers of other cars next to the
autonomous vehicle)? Advancing exisiting work e.g., [21, 42] is needed for our
increasing use cases.

###  4.3 Areas that require interdisciplinary engagement

There is a question of what to offer and to whom. For example, releasing the
code and environment may allow some people to directly answer their questions.
Providing an explanation broadens who can inspect the model, including users
and domain experts; however, what information to release, how it should be
extracted, and how often during the life cycle of the model that information
should be updated will depend on the use context. We will also need mechanisms
for people to request more information about a model as new concerns become
apparent. Finally, all information release must be balanced with concerns
about privacy and trade secrets.

##  5 Inspecting the Model: Local Explanations about Individual Decisions

These criteria have to do with providing information to a user about a
specific decision that is made, such as benefits denial. In some cases, it may
be sufficient to simply provide the information and logic that led to the
decision (a meaningful explanation). In other cases, it may be preferable to
provide actionable ways to change the decision (recourse) [138, 69]. In the
following, we use the term _local explanation_ to refer to explanations that
are meant to provide insight about a particular decision, rather than about
the model overall [89]. We use the term _recourse_ to refer a modification of
the input that results in the output changing to the desired value.

  * CDADM 6.2.3: Providing a meaningful explanation to affected individuals of how and why the decision was made as prescribed in Appendix C.

  * CDADM 6.4.1: Providing clients with any applicable recourse options that are available to them to challenge the administrative decision.

  * CDADM App. C: In addition to any applicable legal requirement, ensuring that a meaningful explanation is provided with any decision that resulted in the denial of a benefit, a service, or other regulatory action.

  * WEF: Explore mechanisms to enable interpretability of the algorithms internally and externally as a means of establishing accountability and contestability. – With AI solutions that make decisions affecting people’s rights and benefits, it is less important to know exactly how a machine-learning model has arrived at a result if we can show logical steps to achieving the outcome. In other words, the ability to know how and why a model performed in the way it did is a more appropriate means of evaluating transparency in the context of AI. For example, this might include what training data was used, which variables have contributed most to a result, and the types of audit and assurance the model went through in relation to systemic issues such as discrimination and fairness. This should be set out as documentation needed by your supplier. – It is also important to consider the potential tension between explainability and accuracy of AI when acquiring AI solutions. Classic statistical techniques such as decision-tree models are easier to explain but might have less predictive power, whereas more complex models,such as neural networks, have high predictive power but are considered to be black boxes.

Approaches for creating local explanations rely heavily on a notion of local
region, and thus some notion of distance. Some inputs are more easily
explained than others, and any explanation can introduce privacy risks.

###  5.1 What we know how to do

There are many techniques of providing local explanations for a model [38, 77,
108, 109, 81, 121]. Specifically, given a definition of distance, we can find
a counterfactual: the closest point such that the model’s output is a desired
class [57, 138]. This can be used to help an individual determine what
features set them apart compared to a nearby alternative, and also set the
foundation for recourse (if those features can be changed) [69].

###  5.2 Directions requiring additional AI innovation

  * •

Defining Distance Metrics. As noted above, local explanations rely heavily on
notions of nearby data. It can be difficult to adjudicate what correlations in
the data should be preserved and what should not. For example, if there are
correlations between the kind of sign and the geographic location in a traffic
image data set, should those correlations be retained in the distance metric?
What about for race and postal codes or sex and hormone levels? Some work
exists on using human input to define the appropriate distance metric for the
purposes of explanation and recourse [69], but more is needed.

  * •

Data without Interpretable Dimensions. The challenges associated with choosing
distance metrics are exacerbated when the individual dimensions of the data
are not interpretable. For example, suppose we have a medical imaging task in
which the AI system claims that certain cells represent a certain type of
cancer, or a face recognition task in which the AI system claims that the face
in a security video matches a face in a government database. What is a
meaningful explanation [87] in this case? Does it take the form of other
images in the dataset (which may create privacy issues)? Should it involve
first summarizing the input into interpretable concepts [72, 54]? Similar
issues arise with text [145] and timeseries data [6].

  * •

Provenance Adjudication. We may want to know if particular training datum was
used in a particular way to generate the given output. For example, we may be
curious if a traffic sign mis-classificatoin could be attributed to a specific
mislabel example, or we may need to resolve copyright issues from AI-generated
text and images. This is possible in small models, but in very nascent stages
for large models (e.g., LLMs [137] and diffusion-based image generation models
[37]).

  * •

Handling Out of Distribution Data. The idea behind recourse is that it gives a
person a path toward getting the outcome they desire. For example, if a loan
applicant is told that paying off their debts would make them eligible for the
loan, then they would expect to get the loan once the debts are paid. However,
if the applicant’s data is very far from the training data, then the AI-
produced recourse may indeed change the model’s output, but would not be
accepted by the loan officer in a real context.

  * •

Tradeoffs between Explainability and Privacy/Security. Releasing information
for auditing or recourse may allow bad actors access to private information
[120] or to game the system [100]. For example, explanations in the form of
training samples, like those of the traffic images, may allow actors to learn
not only how to trick the autonomous vehicle, but also learn about other
elements of those images (that are not road signs). Advancing existing work
e.g. [134] is necessary to understand the resulting dynamics.

###  5.3 Areas that require interdisciplinary engagement

The biggest question raised by these guidelines is what is the definition of a
“meaningful explanation” [127]. This definitions will depend on the socio-
technical context of the task—contesting a loan denial, a medical error, or a
benefits denial may require different kinds of explanations. Different kinds
of users may also require different explanations.

Relatedly, the purpose of the information provided for recourse will vary
across contexts. For one task, it may be enough to provide only one recourse,
while for others it may be necessary to provide multiple options. In other
contexts, the user might benefit from an interactive system to explore
different options. For example, they could themselves wish to navigate changes
and see if they would result in a favorable loan decision.

Finally, it may be that a recourse generated from a local explanation may not
be the appropriate way to assist a user unhappy with a decision. For example,
suppose someone is convinced that a voice-based covid test is in error about
their disease status. Rather than providing an explanation of the voice
features used to make the decision, the appropriate recourse may be to allow
that person to take a traditional covid test instead. We also note that
certain situations may require a justification (rationale for why a decision
is right with respect to laws, norms, and other aspects of the context) rather
than explanation (what features the AI used to generate the output).

##  6 Designing the Model: Objective Design

All AI systems require formulating goals in precise, mathematical terms.
Objective design converts general goals (e.g. drive safely) into precise
mathematical terms [16, 63]. This distillation process is fraught with
potential pitfalls; an incorrect conversation will result in the AI behaving
in unintended ways. For example, encoding safe driving as always ceding the
right of way may result in an autonomous vehicle that never makes a turn at a
busy intersection. Collaboration with stakeholders during the objective design
process can help ensure the true goals are addressed, rather than a proxy that
may not result in the desired behavior. Documentation of the objective design
process must be sufficiently transparent to ensure calibrated trust from
stakeholders. Examples of criteria include:

  * WEF: Focus on developing a clear problem statement, rather than on detailing the specifications of a solution. - AI technologies are developing rapidly, with new technologies and products constantly being introduced to the market. By focusing on describing the challenges and/ or opportunities that you want to address and drawing on the expertise of technology partners, you can better decipher what technology is most appropriate for the issue at hand. By focusing on the challenge and/or opportunity, you might also discover a higher-priority issue, or realize you were focusing on a symptom rather than the root cause.

The criteria above encourage public servants to identify their actual goals
and then allow the engineers to deliver. To be able to deliver, however, the
AI engineers must be able to convert the problem statement into precise terms.

###  6.1 What we know how to do

In some cases, it is possible to decompose a complex task into simpler
components. For example, in the context of an autonomous vehicle, we might
evaluate a perception system for its ability to identify and forecast the
trajectories of other objects in its environment, and the ability of a planner
to make safe decisions given this information. Algorithms for multi-objective
optimization can find a Pareto front of options corresponding to different
trade-offs between desiderata [115, 125]. There is also recent work in
inferring what objectives are truly desired given observed reward functions
[59].

###  6.2 Directions requiring additional AI innovation

  * •

Metrics for Metrics: Measuring Match to Goals. What are the measures that can
be used to determine whether some technical objective matches our policy
goals? Objective and reward design are relatively well-studied in some
domains, such as reinforcement learning [123, 59], but unsolved for the many
more situations—from autonomous vehicles to email text completion—in which we
see AI systems used today. Further, our goals may be multi-faceted; the
objective must not only be faithful to our goal but also transparent in how it
is faithful.

  * •

Properties of Popular Objective Functions. There are many objective functions
used for their computational convenience and statistical properties (squared
loss, log likelihood, etc.). Because they are so popular, their statistical
properties under various conditions are often well-understood [118]. For
example, we may know that L1 losses are more robust than L2; we may know that
decreased model capacity (e.g. fitting a line) can make a model more prone to
being swayed by influential points. However, how do these very technical
understandings of statistical properties relate to more complex goals,
including reward hacking and other short-cut risks? Better understanding of
these properties could enable better matching between popular losses and
broader policy goals.

  * •

Robustness to a Variety of Objectives. In some subfields of AI, there is
literature on creating agents that perform well across a range of reward
functions [91, 101]. This ensures resilience in the face of imperfections in
the objective. However, more work is needed to make this process efficient,
e.g., for large pre-trained models.

  * •

Computational Constraints for More Robust Objectives. Related to the above,
there are a variety of computational constraints and regularizers that often
make objectives more robust to imperfect specifications. These include
encouraging smoothness (e.g. Lipschitzness), sparsity, and robustness to
certain types of uncertainties (e.g. [42], and distributionally robust
optimization [105]. ). However, work remains to be done to more strongly
connect what these computational tools do in the context of aligning the
technical formulation with the true goal.

Furthermore, some constraints and regularizations are difficult to express
and/or operationalize in analytical forms; instead, they are incorporated
directly into the training procedure, such as adversarial training [132].
Relatedly, additional work is needed to effectively optimize objectives with
multiple criteria—whether those are constraints, regularizers, or competing
terms: Simply writing down an objective does not make it easy to optimize. As
additional terms are added to the objective, the question of how to weigh them
to achieve the desired behavior also becomes more complex.

  * •

Understanding Connections between Objectives and Learnt Model Behavior. Can we
efficiently explain how changes in a technical formulation of an objective
affect the model behavior? We understand this for certain simple models but
not sufficiently for more complex models: e.g., how will changing certain
weights in the reward change an autonomous vehicle’s driving? Conversely, can
we explain policies in terms of compatible reward functions? Can we
efficiently identify where two reward functions may result in different
policies in human-understandable terms? Some prior works tries to answer this
[48]; however, more analyses will facilitate a more fine-grained design of the
reward function to better align with intended objectives.

  * •

Inferring Goals from Observed Behavior. In some cases, we may have examples of
decisions or outputs that we know align with the true goal (e.g. safe driving
trajectories). However, the inverse problem of inferring rewards from behavior
is not identifiable. Advancing techniques[5] to help disambiguate important
elements of the reward function can help ensure that the learned policy aligns
with the desired objectives, leading to improved performance and
generalization.

###  6.3 Areas that require interdisciplinary engagement

Creating goals at a policy level requires considering factors such as
contextual relevance, attainability, and alignment with overarching desiderata
[66]. Ethical concerns associated with the power and impact of AI systems may
also be taken into account. Moreover, sometimes even at the policy level, the
objective remains unclear, making it more difficult to design proper
objectives for the AI systems much less validate and explain them. This issue
is evident in the application of AI in criminal justice, where a lack of clear
policy goals is common.

##  7 Designing the Model: Privacy

Bad actors may use transparency about the data, code, and model for
identifying private information about individuals. There are a number of
examples of regulatory criteria relating to privacy concerns, including:

  * CDADM 6.2.6: Releasing custom source code owned by the Government of Canada as per the requirements specified in section A.2.3.8 of the Directive on…

  * CDADM App. C: Plain language notice through all service delivery channels in use (Internet, in person, mail, or telephone). In addition, publish documentation on relevant websites about the automated decision system, in plain language, describing: A description of the training data, or a link to the anonymized training data if this data is publicly available.

  * WEF: There are many anonymization techniques to help safeguard data privacy, including data aggregation, masking, and synthetic data. Keep in mind, however, that you must manage anonymized data as carefully as the original data, since it may inadvertently expose important insights. RFPs should encourage innovative technological approaches, such as those mentioned above, that make less intrusive use of data or that achieve the same or similar outcomes with less sensitive datasets.

  * WEF: As important as data protection is, not all data is sensitive (e.g. open-government data is freely accessible online). All data, sensitive or not, must have its integrity safeguarded, but it is not necessary to keep non-sensitive data behind closed doors. It is important to assess the privacy needs of different datasets to determine the right level of protection. Normally, personally identifiable information (PII), such as financial and health data, is considered extremely sensitive. The RFP needs to reflect data governance requirements for both the procurement process and the project that are in accordance with the nature of the data.

However, the language in these regulations leaves a number of issues
unspecified, including a standardized, meaningful definition for privacy, and
assumes that we are currently able to properly assess the privacy of a dataset
and anonymize data, which are currently open research questions.

###  7.1 What we know how to do

Differential privacy is a widely-accepted theoretical notion of privacy [143,
39]. In settings where this notion of privacy is appropriate, we have
differentially private algorithms that can calculate statistical properties of
data [40], train machine learning models [1, 98], and generate synthetic data
[25]. Many other privacy notions exist [113, 83, 78]. Choosing which privacy
notion to use in a particular setting remains an open question.

###  7.2 Directions requiring additional AI innovation

  * •

Better Tradeoffs between (differential) Privacy and (predictive) Performance.
In general, differentially-private models have lower predictive performance
than models without privacy guarantees [7]. How can that gap be closed?
Related questions include: Can we ensure models are private even with many
queries and in conjunction with public data? What can we maximally expose
about a model and training data statistics in a way that is still private? Can
we precisely state what cannot be exposed, e.g. a long tail has been left out
[45]? (Note: if we can make this precise, then certain information could be
made public as it poses no privacy risk, and other information may be
available only to a trusted auditor.)

  * •

Creating and Assessing Privacy Definitions. How can we define privacy
appropriately and meaningfully for different types of data? (e.g.
trajectories, text [22], etc.). What do current definitions of privacy
actually achieve on these data?

  * •

Privacy via Minimal Data Collection. Can we collect only the input information
needed for each decision, which may involve collecting different inputs for
different people [133]? What privacy risks are mitigated by this approach? Are
new risks introduced because what inputs are measured is new information?

  * •

Private Generative Models. The main focus of existing work is on
classification. So, there are many open questions when it comes to the privacy
of generative models [28, 26, 27, 68]. For example: How can we prevent a
generative model from replicating training data? Is there a difference between
a private generative model and adding noise to data? Is there a benefit to a
private generative model vs. noised data? Are empirical methods to ensure
privacy e.g. via reinforcement learning with human feedback [150], sufficient?

  * •

Effective Machine Unlearning. In some cases, people may be allowed to elect to
have the influence of their data removed after the model has been trained.
Methods have been created to remove the influence of specific inputs from the
data, but these are still in progress [75, 32], especially for generative
models [116, 50, 62].

###  7.3 Areas that require interdisciplinary engagement

Current private models still allow third parties to infer private information
via access to additional, publicly available data. We need to develop new
notions of privacy for this setting [22]. Broader discussion is also needed
regarding what to do if privacy guarantees sacrifice predictive performance,
especially if the sacrifice is primarily to underrepresented groups [7]. More
generally, the appropriate definition of privacy, and how strict the privacy
guarantee must be (e.g., via hyperparameter settings), will depend on the
setting [41] and must be made transparent. For example, claiming a model is
differentially private when it has a very large epsilon may be misleading.

Finally, while this section has focused on privacy, we also note that there
are many security concerns must also be considered in a holistic manner. There
are clear limitations to what can be achieved with respect to adversarial
actors. If training data are available, a state actor or a large industry
actor could (re)create a model. Once a model or training technique is out, we
really cannot control its use. Unlimited public access to a model (via
queries) intrinsically allows an adversary to learn about the model and the
training data.

##  8 Interacting with the Model: Human + AI Systems

AI regulations frequently emphasize the involvement of humans in various
stages of the decision-making process. Often the intent is for the human
decision-maker to vet an AI recommendation, take responsibility for the final
decision, and intervene in case of emergency situations and system failures.
We also consider the case of learning from human input. Examples of related
criteria include:

  * CDADM App. C: Decisions cannot be made without having specific human intervention points during the decision-making process; and the final decision must be made by a human.

  * CDADM 6.3.6: Establishing contingency systems and/or processes as per Appendix C. (Which says: Ensure that contingency plans and/or backup systems are available should the Automated Decision System be unavailable.)

Technical approaches associated with these criteria include combining
information from multiple experts, as well as ways to ensure that humans are
fully engaged in the decisions.

###  8.1 What we know how to do

There has been significant work on learning from humans. We can apply methods
such as imitation learning [67, 85] and reinforcement learning [82] from human
feedback to orient the model based on expert control or learn human
intentions/preferences [97]. Active learning techniques can be used to
proactively ask for information to improve a model from humans [140, 107].
Finally, we also have methods for humans to take the initiative to correct an
agent (e.g., [110, 84, 92]). While methods in uncertainty quantification are
always being improved, for the purposes of flagging uncertain inputs for human
inspection [53], our current methods are reasonable.

###  8.2 Directions requiring additional AI innovation

  * •

HCI Methods for Avoiding Cognitive Biases. Humans have many cognitive biases
and limitations. If a system behaves most of the time, people may start to
over-rely on it. Confirmation bias can accompany backward reasoning (people
finding ways to justify a given decision) but can be mitigated if a person
performs forward reasoning first (looking at the evidence) [18]. Bias can also
come from imperfect information fusion, e.g., if a human inspects the input
data and then views an AI prediction based on the same input data, they may
falsely believe that the AI prediction is a new, independent piece of
information. For example, we may be concerned that a clinician forms an
opinion from patient data, and then sees an AI opinion based on the same data,
they may falsely treat the AI opinion a new, independent form of evidence.
Appropriate human+AI interaction can help mitigate these biases.

  * •

Shared Mental Models and Semantic Alignment. Shared mental models—between the
human and the AI system, between the AI system and the human—are essential for
effective human+AI interaction [4]. While there exists work in which agents
use or create models of humans (e.g., [29]) to facilitate interaction,
including modeling a person’s latent states such as cognitive workload and
emotions (e.g., [96]), it remains an open question as to how to develop and
validate these methods for increasing number of human+AI use cases.

One particularly important area is semantic alignment between the way humans
organize concepts and the way modern AI systems encode representations.
Grounding terms has a long history in AI [61] and innovation is needed for our
modern settings.

  * •

Humans-in-the-Loop in Time-Constrained Settings. How can we include humans in
the loop when decisions have to be made quickly e.g. industrial robots in
emergency scenarios involving human workers? It is crucial that automated
systems can fail gracefully and hand-over control to humans, even in time-
constrained settings [112].

  * •

Evaluation and Design of Realistic Human-in-the-Loop Systems. Most current
testing is for lay user and consumer applications, where risks and costs are
minimal. However, evaluation in other settings is more challenging:
Integrating a new interactive system into an existing workflow may require not
only significant software effort, but also training of users. In high-stakes
settings such as healthcare, criminal justice, and major financial decisions,
there is a risk of real harm to people. How can we evaluate and design for
these cases? Building more general knowledge about human-in-the-loop systems
and developing smarter experimental designs may help reduce these burdens. So
might validating methods for piloting methods in offline or de-risked ways
that may still inform the target application. Relatedly, standard procedures
are needed for evaluating and monitoring human-in-the-loop systems.

###  8.3 Areas that require interdisciplinary engagement

Shared human+AI decision-making is an interdisciplinary area involving social
science, psychology, cognitive science, etc., [34]. Fortunately, researchers
in HCI already have connections to these fields. Furthermore, the design
adoption of new tools into workplaces is well-studied in design, human factors
research, and management and operations science—and require interdisciplinary
teams with appropriate expertise. These interdisciplinary efforts will help
inform decisions about whether, how, and which humans to include in the loop,
as well as how a system that is expecting human input should respond to
inappropriate, slow, or absent input from the human.

##  9 Conclusion

In this document, we examined the technical criteria in two real regulatory
frameworks—the Canadian Directive on Automated Decision-Making and World
Economic Forum AI Procurement in a Box. We find that we only have some of the
tools needed to ascertain whether an AI system meets the stated requirements.
We list several concrete directions for AI innovation that, if addressed,
would improve our ability to create regulatable AI systems.

#### Acknowledgements.

The authors thank Andrew Ross, Siddharth Swaroop, Rishav Chourasia, Himabindu
Lakkaraju, and Brian Lim; all participants of NUS Responsible, Regulatable AI
Working Group 2022-2023 including Limsoon Wong, Angela Yao, Suparna
Ghanvatkar, and Davin Choo.

## References

  * Abadi et al. [2016] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pages 308–318, 2016.
  * Achille et al. [2019] A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. C. Fowlkes, S. Soatto, and P. Perona.  Task2vec: Task embedding for meta-learning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6430–6439, 2019.
  * Alvarez-Melis and Fusi [2020] D. Alvarez-Melis and N. Fusi.  Geometric dataset distances via optimal transport.  _Advances in Neural Information Processing Systems_ , 33:21428–21439, 2020.
  * Andrews et al. [2023] R. W. Andrews, J. M. Lilly, D. Srivastava, and K. M. Feigh.  The role of shared mental models in human-ai teams: a theoretical review.  _Theoretical Issues in Ergonomics Science_ , 24(2):129–175, 2023.  doi: 10.1080/1463922X.2022.2061080.
  * Arora and Doshi [2021] S. Arora and P. Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _Artificial Intelligence_ , 297:103500, 2021.
  * Ates et al. [2021] E. Ates, B. Aksar, V. J. Leung, and A. K. Coskun.  Counterfactual explanations for multivariate time series.  In _2021 International Conference on Applied Artificial Intelligence (ICAPAI)_ , pages 1–8. IEEE, 2021.
  * Bagdasaryan et al. [2019] E. Bagdasaryan, O. Poursaeed, and V. Shmatikov.  Differential privacy has disparate impact on model accuracy.  _Advances in neural information processing systems_ , 32, 2019.
  * Bai et al. [2022] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Baluta et al. [2021] T. Baluta, Z. L. Chua, K. S. Meel, and P. Saxena.  Scalable quantitative verification for deep neural networks.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pages 312–323. IEEE, 2021.
  * Barda et al. [2020] N. Barda, G. Yona, G. N. Rothblum, P. Greenland, M. Leibowitz, R. Balicer, E. Bachmat, and N. Dagan.  Addressing bias in prediction models by improving subpopulation calibration.  _Journal of the American Medical Informatics Association_ , 28(3):549–558, 11 2020.  ISSN 1527-974X.  doi: 10.1093/jamia/ocaa283.  URL https://doi.org/10.1093/jamia/ocaa283.
  * Bau et al. [2020] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba.  Understanding the role of individual units in a deep neural network.  _Proceedings of the National Academy of Sciences_ , 117(48):30071–30078, 2020.
  * Bayerl and Paul [2011] P. S. Bayerl and K. I. Paul.  What determines inter-coder agreement in manual annotations? a meta-analytic investigation.  _Computational Linguistics_ , 37(4):699–725, 2011\.
  * Belsley et al. [2005] D. A. Belsley, E. Kuh, and R. E. Welsch.  _Regression diagnostics: Identifying influential data and sources of collinearity_.  John Wiley & Sons, 2005.
  * Ben-Gal [2005] I. Ben-Gal.  Outlier detection.  _Data mining and knowledge discovery handbook_ , pages 131–146, 2005\.
  * Bender and Friedman [2018] E. M. Bender and B. Friedman.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604, 2018.
  * Bernardi et al. [2019] L. Bernardi, T. Mavridis, and P. Estevez.  150 successful machine learning models: 6 lessons learned at booking. com.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 1743–1751, 2019.
  * Bills et al. [2023] S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders.  Language models can explain neurons in language models, 2023.  URL https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
  * Bondi et al. [2022] E. Bondi, R. Koster, H. Sheahan, M. Chadwick, Y. Bachrach, T. Cemgil, U. Paquet, and K. D. Dvijotham.  Role of human-ai interaction in selective prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * Brazil Center for the Fourth Industrial Revolution [2022] Brazil Center for the Fourth Industrial Revolution.  Unpacking AI Procurement in a Box: Insights from Implementation.  Technical report, World Economic Forum, 2022.  URL https://www3.weforum.org/docs/WEF_Unpacking_AI_Procurement_in_a_Box_2022.pdf.
  * Brodley and Friedl [1999] C. E. Brodley and M. A. Friedl.  Identifying mislabeled training data.  _Journal of artificial intelligence research_ , 11:131–167, 1999.
  * Brown et al. [2021] D. S. Brown, J. Schneider, A. Dragan, and S. Niekum.  Value alignment verification.  In _International Conference on Machine Learning_ , pages 1105–1115. PMLR, 2021.
  * Brown et al. [2022] H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr.  What does it mean for a language model to preserve privacy?  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 2280–2292, 2022.
  * Buolamwini and Gebru [2018] J. Buolamwini and T. Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pages 77–91. PMLR, 2018.
  * Cabitza et al. [2017] F. Cabitza, R. Rasoini, and G. F. Gensini.  Unintended consequences of machine learning in medicine.  _Jama_ , 318(6):517–518, 2017.
  * Cai et al. [2023] K. Cai, X. Xiao, and G. Cormode.  Privlava: synthesizing relational data with foreign keys under differential privacy.  _arXiv preprint arXiv:2304.04545_ , 2023.
  * Carlini et al. [2021] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson, et al.  Extracting training data from large language models.  In _USENIX Security Symposium_ , volume 6, 2021.
  * Carlini et al. [2022] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ , 2022.
  * Carlini et al. [2023] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramer, B. Balle, D. Ippolito, and E. Wallace.  Extracting training data from diffusion models.  _arXiv preprint arXiv:2301.13188_ , 2023.
  * Carroll et al. [2019] M. Carroll, R. Shah, M. K. Ho, T. L. Griffiths, S. A. Seshia, P. Abbeel, and A. Dragan.  On the utility of learning about humans for human-ai coordination.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , Red Hook, NY, USA, 2019. Curran Associates Inc.
  * Chan et al. [2022] S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClelland, and F. Hill.  Data distributional properties drive emergent in-context learning in transformers.  _Advances in Neural Information Processing Systems_ , 35:18878–18891, 2022.
  * Chen et al. [2019] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ , 32, 2019.
  * Chourasia and Shah [2023] R. Chourasia and N. Shah.  Forget unlearning: Towards true data-deletion in machine learning, 2023\.
  * Chuang et al. [2020] C.-Y. Chuang, A. Torralba, and S. Jegelka.  Estimating generalization under distribution shifts via domain-invariant representations.  In _International Conference on Machine Learning_ , pages 1984–1994. PMLR, 2020.
  * Cila [2022] N. Cila.  Designing human-agent collaborations: Commitment, responsiveness, and support.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–18, 2022.
  * Cortes and Mohri [2011] C. Cortes and M. Mohri.  Domain adaptation in regression.  In _Algorithmic Learning Theory: 22nd International Conference, ALT 2011, Espoo, Finland, October 5-7, 2011. Proceedings 22_ , pages 308–323. Springer, 2011.
  * Cyberspace Administration of China [2023] Cyberspace Administration of China.  Generative AI Guideline (draft).  Technical report, Cyberspace Administration of China, 2023.  URL http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm.
  * Dai and Gifford [2023] Z. Dai and D. K. Gifford.  Training data attribution for diffusion models.  _arXiv preprint arXiv:2306.02174_ , 2023.
  * Doshi-Velez and Kim [2017] F. Doshi-Velez and B. Kim.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ , 2017.
  * Dwork [2006] C. Dwork.  Differential privacy.  In _Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33_ , pages 1–12. Springer, 2006.
  * Dwork et al. [2014] C. Dwork, A. Roth, et al.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 9(3–4):211–407, 2014.
  * Dwork et al. [2019] C. Dwork, N. Kohli, and D. Mulligan.  Differential privacy in practice: Expose your epsilons!  _Journal of Privacy and Confidentiality_ , 9(2), 2019.
  * Eckersley [2018] P. Eckersley.  Impossibility and uncertainty theorems in ai value alignment (or why your agi should not have a utility function).  _arXiv preprint arXiv:1901.00064_ , 2018.
  * Elbanhawi and Simic [2014] M. Elbanhawi and M. Simic.  Sampling-based robot motion planning: A review.  _Ieee access_ , 2:56–77, 2014.
  * European Union Commission [2022] European Union Commission.  Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts - General Approach.  _14954/22_ , 2022.
  * Feldman [2020] V. Feldman.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , pages 954–959, 2020.
  * Field et al. [2012] Z. Field, J. Miles, and A. Field.  Discovering statistics using r.  _Discovering Statistics Using R_ , pages 1–992, 2012.
  * for Standardization [2022] I. O. for Standardization.  Information security, cybersecurity and privacy protection — Evaluation criteria for IT security.  Standard, International Organization for Standardization, Geneva, CH, 2022\.
  * Gajcin et al. [2022] J. Gajcin, R. Nair, T. Pedapati, R. Marinescu, E. Daly, and I. Dusparic.  Contrastive explanations for comparing preferences of reinforcement learning.  In _AAAI Conference on Artificial Intelligence_ , 2022.
  * Gama et al. [2014] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia.  A survey on concept drift adaptation.  _ACM computing surveys (CSUR)_ , 46(4):1–37, 2014\.
  * Gandikota et al. [2023] R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau.  Erasing concepts from diffusion models.  _arXiv preprint arXiv:2303.07345_ , 2023.
  * Gebru et al. [2021] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021\.
  * Gehr et al. [2018] T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. T. Vechev.  Ai2: Safety and robustness certification of neural networks with abstract interpretation.  _2018 IEEE Symposium on Security and Privacy (SP)_ , pages 3–18, 2018\.
  * Geifman and El-Yaniv [2017] Y. Geifman and R. El-Yaniv.  Selective classification for deep neural networks.  _Advances in neural information processing systems_ , 30, 2017.
  * Ghorbani et al. [2019] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim.  Towards automatic concept-based explanations.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Ghosh et al. [2022] B. Ghosh, D. Basu, and K. S. Meel.  Algorithmic fairness verification with graphical models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 9539–9548, 2022.
  * Government of Canada [2019] Government of Canada.  Directive on Automated Decision-Making.  Technical report, Government of Canada, 2019.  URL https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592.
  * Guidotti [2022] R. Guidotti.  Counterfactual explanations and how to find them: literature review and benchmarking.  _Data Mining and Knowledge Discovery_ , pages 1–55, 2022.
  * Gupta et al. [2021] V. Gupta, C. Jung, S. Neel, A. Roth, S. Sharifi-Malvajerdi, and C. Waites.  Adaptive machine unlearning.  _Advances in Neural Information Processing Systems_ , 34:16319–16330, 2021.
  * Hadfield-Menell et al. [2017] D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan.  Inverse reward design.  _Advances in neural information processing systems_ , 30, 2017.
  * Han et al. [2021] X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang, et al.  Pre-trained models: Past, present and future.  _AI Open_ , 2:225–250, 2021.
  * Harnad [1990] S. Harnad.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_ , 42(1-3):335–346, 1990.
  * Heng and Soh [2023] A. Heng and H. Soh.  Selective amnesia: A continual learning approach to forgetting in deep generative models.  _arXiv preprint arXiv:2305.10120_ , 2023.
  * Hennig and Kutlukaya [2007] C. Hennig and M. Kutlukaya.  Some thoughts about the design of loss functions.  _REVSTAT-Statistical Journal_ , 5(1):19–39, 2007\.
  * Holland et al. [2020] S. Holland, A. Hosny, S. Newman, J. Joseph, and K. Chmielinski.  The dataset nutrition label.  _Data Protection and Privacy_ , 12(12):1, 2020\.
  * Hu et al. [2022] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.
  * Hulten [2018] G. Hulten.  _Building Intelligent Systems: A Guide to Machine Learning Engineering_.  Apress, 2018.
  * Hussein et al. [2017] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne.  Imitation learning: A survey of learning methods.  _ACM Computing Surveys (CSUR)_ , 50(2):1–35, 2017\.
  * Ippolito et al. [2022] D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini.  Preventing verbatim memorization in language models gives a false sense of privacy.  _arXiv preprint arXiv:2210.17546_ , 2022.
  * Karimi et al. [2021] A.-H. Karimi, B. Schölkopf, and I. Valera.  Algorithmic recourse: from counterfactual explanations to interventions.  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 353–362, 2021.
  * Katz et al. [2017] G. Katz, C. W. Barrett, D. L. Dill, K. D. Julian, and M. J. Kochenderfer.  Reluplex: An efficient smt solver for verifying deep neural networks.  In _International Conference on Computer Aided Verification_ , 2017\.
  * Kearns et al. [2018] M. Kearns, S. Neel, A. Roth, and Z. S. Wu.  Preventing fairness gerrymandering: Auditing and learning for subgroup fairness.  In J. Dy and A. Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2564–2572. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/kearns18a.html.
  * Kim et al. [2018] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677. PMLR, 2018.
  * Koh et al. [2020] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang.  Concept bottleneck models.  In _International Conference on Machine Learning_ , pages 5338–5348. PMLR, 2020.
  * Kreuzberger et al. [2023] D. Kreuzberger, N. Kühl, and S. Hirschl.  Machine learning operations (mlops): Overview, definition, and architecture.  _IEEE Access_ , 2023.
  * Krishna et al. [2023] S. Krishna, J. Ma, and H. Lakkaraju.  Towards bridging the gaps between the right to explanation and the right to be forgotten.  _arXiv preprint arXiv:2302.04288_ , 2023.
  * Kärkkäinen and Joo [2021] K. Kärkkäinen and J. Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1547–1557, 2021.  doi: 10.1109/WACV48630.2021.00159.
  * Lakkaraju et al. [2016] H. Lakkaraju, S. H. Bach, and J. Leskovec.  Interpretable decision sets: A joint framework for description and prediction.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1675–1684, 2016.
  * Li et al. [2006] N. Li, T. Li, and S. Venkatasubramanian.  t-closeness: Privacy beyond k-anonymity and l-diversity.  In _2007 IEEE 23rd international conference on data engineering_ , pages 106–115. IEEE, 2006.
  * Liang et al. [2022] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ , 2022.
  * Lundberg and Lee [2017a] S. M. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc., 2017a.  URL http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf.
  * Lundberg and Lee [2017b] S. M. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017b.
  * MacGlashan et al. [2017] J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman.  Interactive learning from policy-dependent human feedback.  In _International Conference on Machine Learning_ , pages 2285–2294. PMLR, 2017.
  * Machanavajjhala et al. [2007] A. Machanavajjhala, D. Kifer, J. Gehrke, and M. Venkitasubramaniam.  l-diversity: Privacy beyond k-anonymity.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , 1(1):3–es, 2007.
  * Madras et al. [2018] D. Madras, T. Pitassi, and R. Zemel.  Predict responsibly: improving fairness and accuracy by learning to defer.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Mandlekar et al. [2020] A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese.  Human-in-the-loop imitation learning using remote teleoperation.  _arXiv preprint arXiv:2012.06733_ , 2020.
  * McMillan-Major et al. [2023] A. McMillan-Major, E. M. Bender, and B. Friedman.  Data statements: From technical concept to community practice.  _ACM Journal on Responsible Computing_ , 2023.
  * Mertes et al. [2022] S. Mertes, T. Huber, K. Weitz, A. Heimerl, and E. André.  Ganterfactual—counterfactual explanations for medical non-experts using generative adversarial learning.  _Frontiers in artificial intelligence_ , 5, 2022.
  * Mitchell et al. [2019] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru.  Model cards for model reporting.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 220–229, 2019.
  * Mittelstadt et al. [2019] B. Mittelstadt, C. Russell, and S. Wachter.  Explaining explanations in ai.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 279–288, 2019.
  * Molnar [2020] C. Molnar.  _Interpretable machine learning_.  Lulu. com, 2020.
  * Moos et al. [2022] J. Moos, K. Hansel, H. Abdulsamad, S. Stark, D. Clever, and J. Peters.  Robust reinforcement learning: A review of foundations and recent advances.  _Machine Learning and Knowledge Extraction_ , 4(1):276–315, 2022.
  * Mozannar and Sontag [2020] H. Mozannar and D. Sontag.  Consistent estimators for learning to defer to an expert.  In _International Conference on Machine Learning_ , pages 7076–7087. PMLR, 2020.
  * Nanda et al. [2023] N. Nanda, L. Chan, T. Liberum, J. Smith, and J. Steinhardt.  Progress measures for grokking via mechanistic interpretability.  _arXiv preprint arXiv:2301.05217_ , 2023.
  * Obermeyer et al. [2019] Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ , 366(6464):447–453, 2019.
  * Olsson et al. [2022] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_ , 2022.
  * Ong et al. [2019] D. C. Ong, J. Zaki, and N. D. Goodman.  Computational models of emotion inference in theory of mind: A review and roadmap.  _Topics in cognitive science_ , 11(2):338–357, 2019.
  * Palan et al. [2019] M. Palan, G. Shevchuk, N. Charles Landolfi, and D. Sadigh.  Learning reward functions by integrating human demonstrations and preferences.  In _Robotics: Science and Systems_ , 2019.
  * Papernot et al. [2016] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar.  Semi-supervised knowledge transfer for deep learning from private training data.  _arXiv preprint arXiv:1610.05755_ , 2016.
  * Paun et al. [2022] S. Paun, R. Artstein, and M. Poesio.  Statistical methods for annotation analysis.  _Synthesis Lectures on Human Language Technologies_ , 15(1):1–217, 2022.
  * Pawelczyk et al. [2023] M. Pawelczyk, H. Lakkaraju, and S. Neel.  On the privacy risks of algorithmic recourse.  In _International Conference on Artificial Intelligence and Statistics_ , pages 9680–9696. PMLR, 2023.
  * Pinto et al. [2017] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta.  Robust adversarial reinforcement learning.  In _International Conference on Machine Learning_ , pages 2817–2826. PMLR, 2017.
  * Pleiss et al. [2020] G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger.  Identifying mislabeled data using the area under the margin ranking.  _Advances in Neural Information Processing Systems_ , 33:17044–17056, 2020.
  * Pushkarna et al. [2022] M. Pushkarna, A. Zaldivar, and O. Kjartansson.  Data cards: Purposeful and transparent dataset documentation for responsible ai.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1776–1826, 2022.
  * Raffel et al. [2019] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv e-prints_ , 2019.
  * Rahimian and Mehrotra [2019] H. Rahimian and S. Mehrotra.  Distributionally robust optimization: A review.  _arXiv preprint arXiv:1908.05659_ , 2019.
  * Räukur et al. [2022] T. Räukur, A. Ho, S. Casper, and D. Hadfield-Menell.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks.  _arXiv preprint arXiv:2207.13243_ , 2022.
  * Ren et al. [2021] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang.  A survey of deep active learning.  _ACM computing surveys (CSUR)_ , 54(9):1–40, 2021\.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144, 2016.
  * Ribeiro et al. [2018] M. T. Ribeiro, S. Singh, and C. Guestrin.  Anchors: High-precision model-agnostic explanations.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * Rudin [2019] C. Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature machine intelligence_ , 1(5):206–215, 2019.
  * Russell et al. [2016] H. E. Russell, L. K. Harbott, I. Nisky, S. Pan, A. M. Okamura, and J. C. Gerdes.  Motor learning affects car-to-driver handover in automated vehicles.  _Science Robotics_ , 1(1):eaah5682, 2016.
  * Samarati and Sweeney [1998] P. Samarati and L. Sweeney.  Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression.  Technical report, technical report, SRI International, 1998.
  * Samek and Müller [2019] W. Samek and K.-R. Müller.  Towards explainable artificial intelligence.  _Explainable AI: interpreting, explaining and visualizing deep learning_ , pages 5–22, 2019.
  * Sawaragi et al. [1985] Y. Sawaragi, H. NAKAYAMA, and T. TANINO.  _Theory of multiobjective optimization_.  Elsevier, 1985.
  * Schramowski et al. [2022] P. Schramowski, M. Brack, B. Deiseroth, and K. Kersting.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  _arXiv preprint arXiv:2211.05105_ , 2022.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Shalev-Shwartz and Ben-David [2014] S. Shalev-Shwartz and S. Ben-David.  _Understanding machine learning: From theory to algorithms_.  Cambridge university press, 2014.
  * Shen et al. [2023] X. Shen, Y. Wong, and M. Kankanhalli.  Fair representation: Guaranteeing approximate multiple group fairness for unknown tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(1):525–538, 2023.  doi: 10.1109/TPAMI.2022.3148905.
  * Shokri et al. [2021] R. Shokri, M. Strobel, and Y. Zick.  On the privacy risks of model explanations.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 231–241, 2021.
  * Simonyan et al. [2013] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Singh et al. [2019] G. Singh, T. Gehr, M. Püschel, and M. Vechev.  An abstract domain for certifying neural networks.  _Proceedings of the ACM on Programming Languages_ , 3(POPL):1–30, 2019.
  * Singh et al. [2009] S. Singh, R. L. Lewis, and A. G. Barto.  Where do rewards come from.  In _Proceedings of the annual conference of the cognitive science society_ , pages 2601–2606. Cognitive Science Society, 2009.
  * Smith et al. [2022] A. Smith, R. Black, J. Davenport, J. Olszewska, J. Rößler, and J. Wright.  _Artificial Intelligence and Software Testing_.  BCS, The Chartered Institute for IT, 2022.
  * Soh and Demiris [2011] H. Soh and Y. Demiris.  Evolving policies for multi-reward partially observable markov decision processes (mr-pomdps).  In _Proceedings of the 13th annual conference on Genetic and evolutionary computation_ , pages 713–720, 2011.
  * Sommer et al. [2020] D. M. Sommer, L. Song, S. Wagh, and P. Mittal.  Towards probabilistic verification of machine unlearning.  _arXiv preprint arXiv:2003.04247_ , 2020.
  * Sosa [1997] D. Sosa.  Meaningful explanation.  _Philosophical Issues_ , 8:351–356, 1997.
  * Strobelt et al. [2017] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush.  Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks.  _IEEE transactions on visualization and computer graphics_ , 24(1):667–676, 2017.
  * Suresh and Guttag [2019] H. Suresh and J. V. Guttag.  A framework for understanding unintended consequences of machine learning.  _arXiv preprint arXiv:1901.10002_ , 2(8), 2019.
  * Tan et al. [2018] S. Tan, R. Caruana, G. Hooker, and Y. Lou.  Distill-and-compare: Auditing black-box models using transparent model distillation.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 303–310, 2018.
  * Tarun et al. [2023] A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli.  Fast yet effective machine unlearning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2023\.
  * Tramèr et al. [2018] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel.  Ensemble adversarial training: Attacks and defenses.  In _6th International Conference on Learning Representations, ICLR 2018_ , 2018.
  * Tran and Fioretto [2023] C. Tran and F. Fioretto.  Data minimization at inference time, 2023.
  * Tsirtsis and Gomez Rodriguez [2020] S. Tsirtsis and M. Gomez Rodriguez.  Decisions, counterfactual explanations and strategic behavior.  _Advances in Neural Information Processing Systems_ , 33:16749–16760, 2020.
  * Tukey et al. [1977] J. W. Tukey et al.  _Exploratory data analysis_ , volume 2.  Reading, MA, 1977.
  * Urbina et al. [2022] F. Urbina, F. Lentzos, C. Invernizzi, and S. Ekins.  Dual use of artificial-intelligence-powered drug discovery.  _Nature Machine Intelligence_ , 4(3):189–191, 2022.
  * Vyas et al. [2023] N. Vyas, S. Kakade, and B. Barak.  Provable copyright protection for generative models.  _arXiv preprint arXiv:2302.10870_ , 2023.
  * Wachter et al. [2017] S. Wachter, B. Mittelstadt, and C. Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  _Harv. JL & Tech._, 31:841, 2017.
  * Wang et al. [2023] K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  In _International Conference on Learning Representation_ , 2023.
  * Ware et al. [2001] M. Ware, E. Frank, G. Holmes, M. Hall, and I. H. Witten.  Interactive machine learning: letting users build classifiers.  _International Journal of Human-Computer Studies_ , 55(3):281–292, 2001.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , 2022.
  * White House Office of Science and Technology Policy [2022] White House Office of Science and Technology Policy.  Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People.  _White House_ , 2022.  URL https://www.whitehouse.gov/ostp/ai-bill-of-rights.
  * Wood et al. [2018] A. Wood, M. Altman, A. Bembenek, M. Bun, M. Gaboardi, J. Honaker, K. Nissim, D. R. O’Brien, T. Steinke, and S. Vadhan.  Differential privacy: A primer for a non-technical audience.  _Vand. J. Ent. & Tech. L._, 21:209, 2018.
  * World Economic Forum [2020] World Economic Forum.  AI Procurement in a Box.  Technical report, World Economic Forum, 2020.  URL https://www.weforum.org/reports/ai-procurement-in-a-box/.
  * Wu et al. [2021] T. Wu, M. T. Ribeiro, J. Heer, and D. S. Weld.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, 2021.
  * Xiao et al. [2022] Y. Xiao, I. Beschastnikh, Y. Lin, R. S. Hundal, X. Xie, D. S. Rosenblum, and J. S. Dong.  Self-checking deep neural networks for anomalies and adversaries in deployment.  _IEEE Transactions on Dependable and Secure Computing_ , 2022.
  * Xu et al. [2021] Z. Xu, X. Shen, Y. Wong, and M. S. Kankanhalli.  Unsupervised motion representation learning with capsule autoencoders.  _Advances in Neural Information Processing Systems_ , 34:3205–3217, 2021.
  * Yap [2021] R. H. Yap.  Towards certifying trustworthy machine learning systems.  In _Trustworthy AI-Integrating Learning, Optimization and Reasoning: First International Workshop, TAILOR 2020, Virtual Event, September 4–5, 2020, Revised Selected Papers 1_ , pages 77–82. Springer, 2021\.
  * Zheng et al. [2023] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.